⭐ Storyboard MCP · Flagship · Video understanding

Paste a Link · Get a ProjectThe flagship that turns any video URL into a fully remixable creative project — in under 5 minutes

Until today, the answer to "can it understand this video and make me my own version?" was "paste me your shot list and I'll do my best." With the marlin-video capability registered on the Livepeer BYOC orchestrator (2026-06-03), the answer is now: paste the URL. The agent watches the video, decomposes it into scene-by-scene structure, plans a remixable project, regenerates every scene in your style, runs a drift-gate on the output, optionally face-swaps a cast onto every shot, and ships a finished cinema reel — all chained, all observable.

runtime

4–7 minutes wall

caps orchestrated

5 in one chain

cost

~$3.20–$4.50 e2e

human touchpoints

2 (style + critique)

style of source video

any (up to 2 min)

Six cinematic film-strip frames with marlin-video JSON data overlays — the video-understanding flagship

The proof · marlin-video sees the scenes, the agent plans the remix krea-2-large hero · the live demo reel ships in Phase 2

Five real-use scenarios — who needs this, what it unlocks

Video understanding is not a research curio. It's the missing layer between "this reference exists" and "make me my own version of it." These are the personas who ask for this every week.

Brand marketer

"I saw a great TikTok yesterday but I can't remember exactly what made it work. Can you study it and give me an ad in our brand style?"

Today: one URL paste · 5 minutes · their brand colors and voice. Previously: a week of agency back-and-forth.

Indie filmmaker

"I want to learn from this Wes Anderson short. Can you decompose its shot grammar so I can study it?"

Today: auto-extracted shot list + camera motions + palette + per-scene remix prompts. Previously: hours of manual scene-by-scene transcription.

Webtoon creator

"This anime trailer feels like the show I'm making. How do I adapt that vibe as my next chapter?"

Today: scene-by-scene narrative beats + palette match + cast-locked regeneration. Previously: weeks of mood-boarding.

Educator

"Here's a great explainer video — rebuild it with my domain examples."

Today: structure extracted, narration scaffolded, examples swapped in. Previously: hire a video producer.

Director / creative agency

"Give me 6 moodboard variants of this reference reel in our house style."

Today: each source scene re-rendered as a still in 6 different styles for the moodboard, all in one chain. Previously: a freelance illustrator week.

The orchestration — five capabilities working together

The "wow" is the chain itself. marlin-video is the new capability, but the magic is how it cooperates with four other production caps in a single agentic flow. Each step uses the right tool for its job; the agent stitches them with no human in the loop except at two designated approval gates.

Stage	Capability	What it does	Cost
Stage 1 · Understand	marlin-video	Watches the source video; returns JSON with N scenes, each scene tagged with start/end, description, camera motion, dominant color, subjects, and a remix-ready prompt	~$0.05
Stage 2 · Style picker	(your attention)	Show source palette + your style_target. Choose: source / yours / blend. The cheapest control point in the pipeline	$0
Stage 3 · Generate keyframes	krea-2-large × 6	Brand-locked moodboard-driven foundation image gen. Each scene's remix_prompt is regenerated with the chosen style applied verbatim	~$0.38
Stage 4 · Animate	seedance-i2v × 6	Each keyframe animates to a 10s clip. Parallel fanout. B9 motion-budget enforced via the marlin-extracted camera_motion field	~$2.16
Stage 5 · Critique gate	marlin-video × 6	Same cap, different prompt. Rate each regenerated clip 0-10 on drift / cast-lock / motion-budget / anatomy. Auto-retry scenes scoring ≥7 (one re-fire allowed)	~$0.30
Stage 6 · Finish	ffmpeg-concat + minimax-music + gemini-tts	Stitch 6 clips. Optional music bed (palette-matched). Optional narration. Final cinema 1920×1080	~$0.30-0.60
Stage 7 · Moderate	marlin-video	One last pass on the final reel — flag any safety issues before publish	~$0.05

What `marlin-video` returns — actual JSON shape from Stage 1

The agent parses this JSON and uses it as the project plan. Every field has a downstream consumer: palette feeds the krea-2-large prompt, camera_motion feeds the seedance-i2v prompt, remix_prompt IS the regen brief. The cap is small (2B params) but the output is structured enough to drive 6 downstream caps cleanly.

// Example response from marlin-video for a 60s reference clip { "title": "Empty studio at dawn — a single dancer's morning practice", "overall_mood": "meditative, luminous, controlled", "dominant_palette": ["warm amber dawn", "dove-grey leotard", "faded oak parquet"], "scenes": [ { "id": 1, "start_sec": 0.0, "end_sec": 10.0, "description": "A dancer enters an empty Beaux-Arts studio at first light, walking toward the barre", "camera_motion": "push-in", "subjects": ["ballerina"], "dominant_color": "warm amber", "remix_prompt": "a single dancer walks into an empty studio at dawn, full-body, painterly cinematic" }, { "id": 2, "start_sec": 10.0, "end_sec": 20.0, "description": "Tendu at the barre — one leg extends forward, returns", "camera_motion": "static", "subjects": ["ballerina", "barre"], "dominant_color": "dove grey", "remix_prompt": "dancer at barre executing a tendu, mid shot, classical pose" }, // ... 4 more scenes ] }

Pipeline trace — end-to-end with parallelism + drift gate

Stage 1 — UNDERSTAND 1 × create_media action=generate, model=marlin-video, source_url=<USER URL> prompt: SCENE_TAG_SCHEMA (see playbook) → JSON returned in ~30s
Stage 2 — STYLE PICKER (human gate · ~30s) Show: source palette · your style_target · "Use source / Use mine / Blend" Output: final_style_brief consumed by Stage 3 prompts
Stage 3 — KEYFRAMES (parallel) 6 × create_media action=generate, model=krea-2-large Each: [final_style_brief] + [cast_spec if any] + [scene.remix_prompt] → 6 keyframes in ~30s
Stage 4 — ANIMATE (parallel) 6 × create_media action=animate, model=seedance-i2v, duration=10s Each prompt includes scene.camera_motion from Stage 1 → 6 clips in ~90s
Stage 5 — CRITIQUE + AUTO-RETRY (drift gate) 6 × create_media action=generate, model=marlin-video Each evaluates the regenerated clip against the source scene Returns: { drift_score, motion_budget, cast_lock_violations, anatomy_issues } Any scene with drift_score ≥ 7 → re-fire Stage 4 ONCE (then human review)
Stage 6 — FINISH Optional face-swap-video per scene (if cast_anchor_url set) ffmpeg-concat 6 clips → 1920×1080 cinema Optional minimax-music palette-matched bed (0.55 vol) Optional gemini-tts narration (1.0 vol) → Final reel ~60s wall
Stage 7 — MODERATE 1 × create_media model=marlin-video on the final reel Block on severity=high; warn on med; log to cost report
SHIP → viewer URL with side-by-side comparison toggle (source vs remix per scene)

Cost — honest end-to-end

Stage 1 + 5 + 7 parses

~$0.40

8 × marlin-video calls @ ~$0.05 ea

Keyframes

~$0.38

6 × krea-2-large @ $0.063

Animations

~$2.16

6 × seedance-i2v 10s @ $0.36

Music + VO

~$0.55

minimax-music + gemini-tts (optional)

Baseline cost: ~$3.49. With 1 auto-retry on a drifted scene: ~$3.85. With cast-anchor face-swap-video on all 6: +$1.44 = ~$4.93. Cheapest creative-agency-replacement workflow in the catalogue, per-second of finished cinema.

Why this is the flagship for video understanding — and not just a cap registration

What this showcase proves

Multi-cap orchestration is the wedge. Five distinct caps cooperate in one chain — understand + plan + generate + animate + critique + finish + moderate — with no human-in-the-loop except for two designated approval gates. Other tools call one model at a time; storyboard composes.
marlin-video is used in three different roles in the same pipeline. Stage 1: scene-tag (understand). Stage 5: critique (drift gate). Stage 7: moderate (publish gate). Same cap, three jobs — that's what "general purpose video-VLM" buys you.
The cheapest auto-replacement of agency video work. $3.85–$4.93 per finished remix vs $25-80K per agency campaign. The math compounds when a brand wants 100 personalized variants.
Real-use is one URL paste. The user doesn't write a shot list, doesn't pick scenes, doesn't even type a prompt — they paste a URL they already love and pick their style. Onboarding friction collapses.
The drift gate is the trust gate. Stage 5 is what makes the auto-pipeline credible. Without an automatic quality check, multi-step pipelines erode confidence at every step. With marlin-video as the gate, the chain is self-correcting.
The pattern generalizes. Same orchestration, swap the source: a podcast clip (with future video-from-audio), a screen recording (tutorial remix), a phone clip (cinematize). The understand-plan-generate-critique loop is the chassis.

Open invariants — what the playbook enforces

Quality gates (auto-applied)

Cost upfront. Before any inference: estimated total shown, user approves.
Style picker after parse. Source palette + your style_target + blend option, BEFORE regenerating keyframes. Catches mismatch early.
Per-scene drift gate. marlin-video critique on every regenerated clip. drift_score ≥ 7 → auto-regen once, then human review.
Auto-chunk long sources. >2 min input → ffmpeg-trim, parse per chunk, concat results.
Moderation pre-publish. Final reel → severity=high blocks publish; severity=med warns + logs.
Side-by-side comparison. The output viewer shows source vs remix per scene. No hiding, no marketing tricks — the proof is the toggle.

Cross-surface parity — webapp · MCP · CLI · slash

Surface	Entry point	Status
Webapp chat	Paste any video URL into chat. Preprocessor detects → "remix this in your style?" → playbook runs.	Phase 1 — needs URL detect in `lib/agents/preprocessor.ts`
Webapp slash	`/remix <url>`	Phase 1 — new entry in `lib/skills/commands.ts`
Webapp right-click	On any imported video card → "Remix this in my style"	Phase 2 — context menu entry
MCP	`mcp__storyboard__generate_project({ playbook: "paste-link-get-project", source_url, style_target })`	Phase 2 — registered playbook in `lib/mcp-server/playbooks.ts`
CLI	`livepeer playbook paste-link-get-project --source <url> --style "<keywords>"`	Phase 2 — playbook runner in livepeer CLI

Reproduce — the playbook is open

Pipeline lives in the paste-link-get-project playbook · capability spec is in the video-understanding skill. Three new caps from the 2026-06-03 expansion (marlin-video + face-swap-image + face-swap-video) are documented in CLAUDE.md. marlin-video is live on BYOC at sdk.daydream.monster/capabilities — verify any time.

Open the playbook → Raw markdown video-understanding skill All showcases