---
reliability: 4.2 # 5 − music gen .2 − wizper transcription .2 − 7-stage chain .4; the rest (trim/grade/concat/mux/export) is deterministic ffmpeg; subtitles + credits OPTIONAL; agent can't watch the clips — trim quality depends on user-confirmed timestamps
---

# Phone clips to cinematic short — agent playbook

> Paste this whole file (with the BRIEF filled in) into Claude cowork, chat, or Code. The agent walks through it with you and stops for your approval at every step that matters.

## What you'll get

A polished short film built from the raw phone clips you already shot. This playbook is mostly **non-AI work** — ffmpeg color-grading, aspect cropping, music-bedding — proving that the MCP is a production-grade video editor, not just a generator.

- **Trim plan** — each clip cut to its strongest moment (the agent proposes timestamps based on your story brief; you approve).
- **Color-graded master** — LUT applied via `ffmpeg-overlay` so the iPhone clips and Android clips look like they came from the same camera.
- **Music bed** — either generated to match the mood, or your own audio mixed in cleanly.
- **Optional burned subtitles** — if you provided a narration, captions burned in at the lower-third.
- **9:16 vertical cut** — for Reels / TikTok / Stories.
- **16:9 horizontal cut** — for YouTube / family TV.
- **Credits screen** — a clean closing card with your name + date.

**Time**: ~20 minutes wall clock. **Your attention**: about 5 minutes across 6 approvals. **Cost**: $0.50 – $1.80 end to end (mostly ffmpeg, very cheap).

## Tell the agent about the footage

```yaml
clip_urls:              # list of mp4 URLs — upload your phone clips via /upload first, then paste each URL on its own line under this key
intended_story:         # 1 sentence — what is this short ABOUT? e.g. "my daughter's first day of kindergarten" · "Lisbon weekend with the boys" · "the wedding rehearsal dinner"
mood:                   # joyful · contemplative · triumphant · nostalgic · bittersweet · celebratory · quiet
music_style:            # one of: "ambient piano" · "indie folk" · "epic cinematic" · "lo-fi hip-hop" · "user-supplied: <url>" (paste audio URL)
narration_url:          # OPTIONAL — audio of you talking over the clips. Burned-in subtitles will be generated.
aspect_ratio_primary:   # 16:9 (default) or 9:16
length_target_seconds:  # 30-90 (default 60)
short_slug:             # kebab-case, e.g. lisbon-weekend
your_name:              # for the credits screen
```

## Quality controls (new — auto-applied)

This playbook leverages the closed quality loop. The agent applies these by default; you don't have to ask:

- **Cost upfront.** Before any multi-step batch fires, the agent calls `submit_plan` with `confirm:false` and shows you the proposed steps + total cost. Nothing runs until you approve.
- **Auto-retry on weak shots.** The server default is `quality_threshold: 0.7`. For generation-heavy steps the playbook will pass `quality_threshold: 0.85` so a low-confidence first try gets one auto-retry before you see it.
- **Critic verdict on the result.** When a multi-scene step finishes, the agent reads the `critique` field on the response (verdict: ship / iterate, plus issues / missing items) and surfaces it for your review.
- **One-click recovery.** If a step fails, `get_plan` returns a `recovery` hint pointing at `replan({plan_id})` — the agent classifies the failure and proposes a recovery plan you can approve.
- **End-of-project scorecard.** The final wrap-up calls `get_scorecard({project_id})` with succeeded count, mean quality score, critic verdict, total cost, and wall time.

You can override any of these per step: pass `quality_gate: false` on cost-sensitive tries, or `submit_plan({plan_id, confirm:true})` to skip the cost-preview pause when you trust the chain.

## How the agent should run this (interaction contract)

1. **CONFIRM (one message, ≤1 question):** restate in 1 line ("<N> clips → ~60s graded short, <mood> LUT + <music_style> bed, 16:9 + 9:16 cuts, ~$0.50–1.80, ~20 min") + real choices: LUT family (warm-amber / faded-film / teal-orange / desaturated) · generated vs user-supplied music · subtitles only if narration exists. One question max — usually "which mood/LUT?".
2. **PREVIEW CHECKPOINT:** the TRIM PLAN table ($0) + the FIRST graded clip (~$0.02) before grading the rest — the agent can't watch your footage, so the trim timestamps and the LUT choice are the two things worth eyes-on before the chain runs.
3. **NARRATE:** per stage: "grading 6 clips (~3 min)…", "music bed generating (job …, ~2 min)"; poll `get_create_media` every 10–15s; ffmpeg steps are fast — only music gen can stall, so report it specifically; never silent >2 min.
4. **FAIL GRACEFULLY:** `music` weak/fails → retry once → `sfx` ambient bed → ask for a user-supplied URL (≤2 retries, deterministic steps don't need retries); `wizper` fails → `nemotron-asr` → ship without subtitles; `ffmpeg-concat` drops the clips array via MCP → direct SDK with `clips` in the body. A short with no subtitles still ships; a stalled chain doesn't.
5. **DELIVER:** both cuts + one honest line ("trim points were proposed from metadata, not viewing — clip 3's cut may land mid-action; music is generated, mood-matched not melody-specific") + ONE next step ("AirDrop the 16:9 to family, or re-run one clip's trim window").

## How this works

Six gates. Most of the work is deterministic ffmpeg — fast, cheap, and reproducible. The AI shows up only where it has to (music gen if you didn't supply audio; subtitle generation if narration is present). The point is: your clips, finished.

## The steps

### Step 0 — Confirm the brief

The agent reads back the BRIEF and lists each `clip_urls` entry with its duration + thumbnail so you can spot upload errors.

**STOP**: "Brief looks right? All clips load? (approve / edit [field] [new value])"

### Step 1 — Propose trim plan (~1 min, ~$0.05)

The agent reads each clip's metadata (duration, dimensions, codec) and proposes a trim point per clip — typically the middle 60% of each, since phone clips usually have shaky starts and ends. It writes the plan as a table:

```
clip_1.mp4 (12s)  →  trim to 0:02–0:08  (6s, "establishing shot")
clip_2.mp4 (24s)  →  trim to 0:05–0:12  (7s, "the moment")
...
```

The agent can't actually watch your clips, so it asks you to confirm or edit trim points. The default plan still gives you a viable assembly even without edits.

> **For the agent**: present the trim plan as an editable table. Total trimmed duration should approximate `length_target_seconds` ± 20%. If it doesn't, propose which clips to trim shorter or drop entirely.

**STOP**: "Trim plan looks right? (approve / edit clip [#] to [start–end] / drop clip [#])"

### Step 2 — Cut and color-grade (~3 min, ~$0.10)

The agent calls `ffmpeg-trim` on each clip per the approved plan, then `ffmpeg-overlay` with a LUT matched to `mood`:

- joyful / celebratory → warm-amber LUT
- contemplative / nostalgic → faded-film LUT
- triumphant / cinematic → teal-and-orange LUT
- quiet / bittersweet → desaturated-soft LUT

Result: every clip now feels like the same camera shot the whole thing. This is the single highest-impact transformation in the playbook.

> **For the agent**: pure ffmpeg work — no AI inference, no quality_threshold needed. Surface the first graded clip before processing the rest so the user can pick a different LUT if the mood doesn't match.

**STOP**: "Color grade approved? (approve / try [LUT name])"

### Step 3 — Music bed (~3 min, ~$0.40)

If `music_style` starts with `"user-supplied:"`, the agent skips generation and uses your audio URL directly.

Otherwise the agent calls `create_media({ action: "music", prompt: <style>, lyrics_prompt: "[Instrumental]", duration: length_target_seconds })` — generating an instrumental music bed in the chosen style at -15dB ready for mix.

> **For the agent**: pass `quality_threshold: 0.75` on music gen (music has wide acceptable variance). Length should be ≥ `length_target_seconds` so it doesn't loop or cut. **Critical:** pass `lyrics_prompt: "[Instrumental]"` explicitly — without it, the `music` cap will sing your style prompt verbatim. Phone-to-cinema is always instrumental.

**STOP**: "Music bed approved? (approve / regenerate / try [different style])"

### Step 4 — Optional subtitle pass (~2 min, ~$0.20)

Only fires if `narration_url` is set. The agent transcribes `narration_url` via `wizper` (Whisper-v3) → timestamped SRT. Then `ffmpeg-burn-subtitles` overlays them at the lower-third in a clean sans-serif, palette-matched to the mood.

Skip if no narration.

> **For the agent**: pass `style: "lower-third clean sans"` to the burn step. Keep font size moderate — these are personal videos, not broadcast captions.

**STOP** (only if narration present): "Subtitles approved? (approve / regenerate / skip)"

### Step 5 — Stitch + audio mix (no gate, ~3 min, ~$0.10)

The agent calls `ffmpeg-concat` on the graded clips in order (transition: cut, or crossfade-300 if exactly 2 clips), then `ffmpeg-mux` to lay audio under everything with `audio_fill: "loop"` — the loop fill guarantees the bed covers the FULL short with no silent tail if the music is shorter than the video. If narration is present, narration rides on top at 1.0 with music ducked to 0.55; if not, music sits at 0.5 through the whole short.

This is all-deterministic ffmpeg. No AI in this step. Reliably fast.

> **For the agent**: the standard MCP envelope drops array params on ffmpeg-concat with 3+ clips — fall back to direct SDK with the `clips` array in the params body. Use `ffmpeg-mux` (video_url + audio_url + `audio_fill:"loop"`), NOT a separate mix call.

### Step 6 — Dual aspect-ratio export + credits (~3 min, ~$0.10)

The agent calls `ffmpeg-export` twice — once at the source aspect, once at the inverted aspect using `crop: "center-crop"` for the 16:9 ↔ 9:16 swap (or `crop: "letterbox"` if you'd rather preserve full framing with black bars). Then `hyperframes-lower-third` composes the final credits frame: `your_name` + the date + `intended_story` as subtitle, palette-matched, fading in over 1.5s on the last frame of each export.

Final files: primary aspect master + opposite aspect short.

**STOP**: "Both cuts approved? (approve / redo [which] crop / redo credits)"

### Final step — scorecard wrap-up

Call `get_scorecard({project_id: ...})` — paste output to your channel.

```
Scorecard for cjob_xxx
N/N steps shipped — critic: SHIP
Title: {short_slug}
Steps: N/N succeeded (100%)
Quality: mean X.XX · M/N passed · K retries
Cost: $X.XX actual (est $Y.YY) · Ts wall time
Viewer: <short URL>
```

## When you're done

```
✅ Primary cut ({aspect_ratio_primary}): <URL>
✅ Short cut (opposite aspect): <URL>
✅ Music bed: <URL>
✅ Subtitles SRT (if generated): <URL>

Total spent: $X.XX
Total wall-clock: MM:SS
```

## What to do next

Send the 16:9 cut to family / friends via AirDrop or text — it plays full-bleed on any TV. Post the 9:16 cut to Reels / TikTok / your Story to share with the wider circle.

The same trim-grade-bed-stitch pipeline works for every "I have a bunch of phone clips" moment — birthday parties, vacations, weddings, kids' first day of anything. Save the LUT preference in your brief once and the second run of this playbook (next vacation) finishes even faster.

For creators / brand content: this is your B-roll finisher. Shoot loose on your phone, run this playbook to ship-quality, post. The cost-per-finished-video is under $2 vs. the $50-200 a freelance video editor charges for the same turnaround.

---

## Notes for the agent (only read if a step fails)

**ffmpeg-concat with 3+ clips**: fall back to direct SDK as described in the Quality Controls section. Same direct-SDK pattern applies to `ffmpeg-mux` and `ffmpeg-export`.

**Aspect-ratio cropping**: `ffmpeg-export`'s `crop` param is a 2-value enum on the live tool — `center-crop` (default for inverted-aspect; cuts to fit the target frame) and `letterbox` (preserves full framing with black bars). Face-bias / top-bias smart cropping ships with Director v2; until then, choose `center-crop` for landscapes and people-centered shots, `letterbox` when preserving framing matters more than filling the screen.

**Music gen fallback chain**: `music` → `sfx` → fall through to user-supplied requirement (ask user to paste a URL).