--- tier: hero --- # Paste a Link · Get a Project — agent playbook ⭐ flagship > Paste this whole file (with the BRIEF filled in) into Claude cowork, chat, or Code. **This is the video-understanding showcase.** A user pastes a YouTube / TikTok / Vimeo URL; Storyboard watches it, decomposes it into scenes, plans a remixable project, and generates the user's own version — in their own style, with their own cast — in **under 5 minutes**. ## What it unlocks (the wow) The single most-asked feature in the AI-video category: **"can it understand this video and make me my own version?"** Until today, the answer was "paste me your shot list and I'll do my best." Now, with the `marlin-video` cap registered (2026-06-03), the answer is: **paste the URL.** Storyboard handles understanding, planning, generation, critique, and finishing — all chained, all observable. This is the playbook that turns video understanding from a research curio into a one-paragraph onboarding flow. ## Real-use scenarios | Persona | Today | With this playbook | |---|---|---| | Brand marketer | "I saw a great TikTok yesterday but I can't remember exactly what made it work. Can you study it and give me an ad in our brand style?" — week of agency work | One URL paste · 5 minutes · their brand colors and voice | | Indie filmmaker | "I want to learn from this Wes Anderson short. Can you decompose its shot grammar?" — hours of manual analysis | Auto-extracted shot list + camera motions + palette + remix prompts per scene | | Webtoon creator | "This anime trailer feels like the show I'm making. How do I adapt that vibe as my next chapter?" — weeks of mood-boarding | Scene-by-scene narrative beats, palette match, cast-locked regeneration | | Educator | "Here's a great explainer video — rebuild it with my domain examples." | Structure extracted, narration scaffolded, examples swapped in | | Director | "Give me 6 moodboard variants of this reference reel in my style." | Each scene re-rendered as a still in 6 styles for the moodboard | ## What you'll get - **Scene-by-scene JSON** extracted from the source video — `{ title, palette, mood, scenes: [{ start, end, description, camera_motion, subjects, dominant_color, remix_prompt }] }` - **A populated project** ready to re-render — scenes are pre-filled with extracted prompts - **6 regenerated keyframes** in your chosen style (krea-2-large default) - **6 i2v animations** (seedance-i2v default) — each scene re-animated - **Per-scene drift critique** — marlin-video pass on every regenerated clip, drift-score 0-10 - **Auto-retry** on scenes scoring above 7 (configurable threshold) - **1 final stitched cinema reel** (1920×1080) with optional music + voice - **Side-by-side comparison viewer** showing original frame vs your remix per scene **Time:** 4–7 minutes wall clock. **Your attention:** ~3 minutes (one style pick + one approval at the critique gate). **Cost:** ~$2.50–$4.50 end-to-end. ## Tell the agent about the remix ```yaml piece_title: # what you'll call your remix source_url: # the YouTube/TikTok/Vimeo URL — the REFERENCE # (or @last_video / a canvas card ref) style_target: # your style — keywords: "painterly noir", "Studio # Ghibli watercolour", "1980s anime", "Vermeer painterly" brand_kit_id: # OPTIONAL — auto-injects palette + brand keywords cast_anchor_url: # OPTIONAL — your face / your character as identity ref; # if set, every regenerated scene face-swaps to this auto_critique_threshold: 7 # 0-10; scenes with drift_score >= this auto-regen max_scenes: # default 6; cap at marlin-video's natural break count music_brief: # OPTIONAL — music brief to score the remix narration: # OPTIONAL — gemini-tts script about your remix output_aspect: # cinema 1920×1080 (default) or vertical 1080×1920 piece_slug: # kebab-case for filenames ``` ## Quality controls (auto-applied) - **Cost upfront.** Use `submit_plan` with `confirm:false` to preview steps + total before any inference: 1 × marlin-video parse (~$0.05) + 6 × krea-2-large keyframes ($0.38) + 6 × seedance-i2v 10s ($2.16) + 1 × marlin-video critique pass per scene ($0.30) + optional music ($0.30) = ~$3.20 baseline. Approve, then run. - **Style picker after parse.** marlin-video extracts dominant_palette from the source. The agent shows you the source palette + your style_target + a "Use source palette / use my style_target / blend both" toggle BEFORE regenerating any keyframe. Saves ~$3 in re-renders if the source palette already matches. - **Per-scene drift gate.** After each scene re-animates, run a marlin-video critique. Above threshold → auto-regen ONCE (cost: 1 extra seedance-i2v call per drift). Twice-drifted scenes pause for human approval. - **Auto-chunk long sources.** marlin-video caps at ~240 frames @ 2fps ≈ 2 minutes. Sources longer than that auto-chunk via ffmpeg-trim, parsed per chunk, results concatenated. - **Moderation pre-publish.** Final reel runs through moderation_check before any `case-studies` wire-up. Severity=high blocks; severity=med warns. ## The 6-stage pipeline (canonical) ``` Stage 1 — UNDERSTAND (the marlin-video parse) 1 × create_media(action=generate, model=marlin-video, source_url=USER_URL, prompt=SCENE_TAG_SCHEMA) → JSON: { title, palette, mood, scenes: [1..N] } Cost: ~$0.05 | Wall: ~30s Stage 2 — STYLE PICKER (your attention, ~30s) Show: source palette + your style_target + the option to blend Output: final_style_brief used by Stage 3 Stage 3 — KEYFRAMES (parallel) N × create_media(action=generate, model=krea-2-large, prompt=[final_style_brief + cast_spec + scene.remix_prompt]) → N × keyframe images Cost: ~$0.38 (for N=6) | Wall: ~30s (parallel) Stage 4 — ANIMATE (parallel) N × create_media(action=animate, model=seedance-i2v, source_url=, resolution="auto", duration=) → N × 10s videos (prefer ltx-q-i2v for higher fidelity; grok-imagine-video for premium) Cost: ~$2.16 (for N=6) | Wall: ~90s (parallel) Stage 5 — CRITIQUE + AUTO-RETRY (the drift gate) N × create_media(action=generate, model=marlin-video, source_url=, prompt=DRIFT_GATE_SCHEMA) → N × { drift_score, motion_budget, cast_lock_violations, anatomy_issues } For each scene with drift_score >= threshold: re-fire Stage 4 once. Cost: ~$0.30 + (retries × $0.36) | Wall: ~30s Stage 6 — FINISHING (MANDATORY real finishing pass) Optional: cast_anchor_url set → face-swap-video per scene a. ffmpeg-concat 6 scenes (transition:"cut") → silent reel (~60s) b. music → create_media(action:"music", prompt:music_brief, lyrics_prompt:"[Instrumental]", duration:) c. optional gemini-tts narration d. ffmpeg-mux audio under reel with audio_fill:"loop" (no silent tail); narration 1.0, music ducked to 0.55 when both present e. ffmpeg-export size:output_aspect (landscape_16_9 / portrait_9_16) Final: stitched MP4 at 1920×1080 (or 1080×1920) Cost: ~$0.30-0.60 | Wall: ~60s ``` **Total:** ~$3.20-4.50, ~4-7 minutes. ## Cross-surface parity | Surface | How to invoke | |---|---| | **Webapp chat** | Paste URL in chat. Preprocessor detects, asks "remix this in your style?" → yes → playbook runs | | **Webapp slash** | `/remix ` | | **MCP** | `mcp__storyboard__generate_project({ playbook: "paste-link-get-project", source_url, style_target })` (Phase 2 — for now, manual pipeline) | | **CLI** | `livepeer playbook paste-link-get-project --source --style ""` | | **Right-click webapp** | On any imported video card → "Remix this in my style" | ## The agent orchestration shape (5 caps working together) ``` ┌─────────────┐ USER URL ─────────────▶│ marlin-video │ ──▶ JSON plan └─────────────┘ │ ▼ project_create │ ┌──────────┼──────────┐ ▼ ▼ ▼ krea-2-large × N (parallel) │ ▼ seedance-i2v × N (parallel) │ ▼ marlin-video × N (drift gate) │ ┌────────────┴──────────────┐ ▼ ▼ drift >= 7: re-fire drift < 7: pass seedance-i2v once │ │ └────────────┬──────────────┘ ▼ (optional) face-swap-video │ ▼ ffmpeg-concat │ ┌────┼────┐ ▼ ▼ music gemini-tts │ │ └────┬────┘ ▼ FINAL REEL │ ▼ marlin-video (moderation) │ ▼ SHIP ``` 5 distinct caps in one orchestrated flow: **marlin-video (understand+critique+moderate) + krea-2-large (keyframes) + seedance-i2v (animate) + music + gemini-tts**. The "wow" is the chain itself — each cap is the right tool for its job, and the agent stitches them with no human in the loop. ## Honest limits - **Source must be public.** marlin-video accepts URLs but needs to fetch the file. YouTube videos need to be downloaded first (ytdl-style) or hosted as direct mp4 — the playbook auto-chains `ffmpeg-fetch` (planned tool cap) for YT/TikTok; for now, the user provides a direct mp4 URL or uses a re-uploaded clip. - **Identity preservation across regenerated scenes is per-scene, not cross-scene.** If you want cast lock across all 6, set `cast_anchor_url` — face-swap-video runs as Stage 6.5, swapping every scene to the same face. For higher-fidelity multi-scene identity, train a LoRA first. - **2-minute source limit per parse.** Longer sources get auto-chunked but the regen scope expands linearly with chunk count. - **Critique gate is heuristic.** marlin-video's drift-score is a single LLM judgment, not ground-truth. Calibrate the threshold on your style — for cinematic the default 7 works; for cartoon styles you may need 5. - **The "remix" depends on style_target.** A vague "anime" gives vague remixes. The playbook works best when style_target is concrete (one art-movement + one palette + one mood). Borrow phrasing from the source palette extracted in Stage 1. ## Open questions / Phase 2 - Streaming-snapshot helper so recipe 3 (`lv2v_quality_monitor`) becomes live-callable - Dedicated `analyze_video` MCP tool that wraps marlin-video with a clean schema-typed return (instead of routing through `create_media` with `model_override`) - `ffmpeg-fetch` cap to auto-download YT/TikTok URLs in-pipeline (legality / TOS gate) - Side-by-side comparison viewer route at `/v/remix/` (the proof-of-remix surface) - Direct `marlin-video` invocation from the agent's `/analyze` slash command (today image-only, video coming) ## Execution mode (autopilot / director / auto) This is a multi-scene project, so pass a `mode` to `generate_project` to control the **pre-finish gate** (after every scene renders, before delivery): `autopilot` (certain brief → ship in one call, no pause), `director` (always pause for a human pick/steer before delivery), or `auto` (recommended — ship UNLESS the auto-critic is unsure: score < `confidence_threshold` (default 0.8), verdict `iterate`, or a flagged shot). Setting `mode` runs the project async (returns a `job_id`); resume a director/auto pause with `review_checkpoint` + `resume_from_checkpoint` (decision `continue` to deliver / `abort` to discard). Omit `mode` for legacy straight-through behavior.