---
name: video-understanding
description: "Use marlin-video to understand a video — scenes, motion, subjects as JSON"
agent_rule: "User wants a video broken into scenes/subjects/timestamps — call create_media (marlin-video) and plan against the JSON."
task: [analyze]
domain: [video]
persona: [agent-only]
scope: episode
category: capability
type: capability
trigger_keywords:
  - watch this video
  - analyze this video
  - break this into scenes
  - what's happening in this video
  - tag this video
  - scene-by-scene
  - paste a link
  - remix this
  - reverse engineer
  - storyboard from video
  - critique this video
  - rate this video
  - check this for drift
when_to_use: >
  Any time the agent needs to TURN VIDEO INTO STRUCTURE — scene boundaries,
  subjects, camera motion, color, timestamps. Five canonical recipes below.
when_NOT_to_use: >
  - For audio understanding (speech, music): use gemini-text or chatterbox-tts
  - For generating new video: use seedance-i2v / kling-v3-i2v / veo-i2v
  - For simple thumbnail extraction: use ffmpeg-trim instead (cheaper)
  - For live LV2V scoring (NOT yet implemented): a 5s tick of marlin-video
    against the latest stream output is technically possible but the SDK
    has no `stream_sample` action yet — see Open Questions.
---

# video-understanding — capability skill

> **Reliability ceiling:** marlin is a 2B model sampling at 2fps — it is **noisy on fast footage** (sports, quick cuts): counts are ESTIMATES, not ground truth, and timestamps drift ±1–2s. Present its numbers as approximate ("~6 shots"), never as exact counts; for load-bearing reads, retry once with a tighter prompt, then fall back to `nemotron-omni` or `gemini-text` on extracted keyframes.

## The cap

| Field | Value |
|---|---|
| Cap name | `marlin-video` |
| Backend | `fal-ai/marlin` (open-source 2B video-VLM) |
| Cost | **$0.015 per 1K tokens** (~$0.01-0.05 per 30s clip) |
| Max input | mp4 / mov / webm / m4v / gif · ~240 frames @ 2fps ≈ 2 minutes |
| Output | structured text (JSON, prompt-driven) |
| Capacity | 4 concurrent |

The model accepts a video URL + a free-form text prompt that instructs it what to extract. Returns text only — typed as `text` in CAPABILITY_KIND because its output is a JSON document, not a renderable media URL.

## Calling pattern

```typescript
mcp__storyboard__create_media({
  action: "generate",
  model_override: "marlin-video",
  prompt: "<the question you want answered about the video, often a JSON schema instruction>",
  source_url: "<https://video-url.mp4>",
});
```

The capability returns a `result_text` field (not `url`). Parse JSON out of it.

## The 5 canonical recipes

### 1. `scene_tag_reel` — Reverse-engineer a finished video into a project

**Goal:** paste a YouTube / TikTok / Vimeo URL, get a remixable project ready to re-render in user's style in <5 minutes.

```typescript
const result = await create_media({
  action: "generate", model_override: "marlin-video",
  source_url: USER_PROVIDED_URL,
  prompt: `Watch this short video and break it into discrete scenes. Return JSON:
{
  "title": "<one-line title>",
  "overall_mood": "<one short phrase>",
  "dominant_palette": ["<color1>", "<color2>", "<color3>"],
  "scenes": [{
    "id": <int>, "start_sec": <float>, "end_sec": <float>,
    "description": "<one sentence>", "camera_motion": "<static|push-in|pull-out|pan|tilt|orbit|handheld|tracking>",
    "subjects": ["<primary>", "<secondary?>"],
    "dominant_color": "<color>",
    "remix_prompt": "<1-sentence prompt that could regenerate this scene in a new style>"
  }]
}
Return ONLY the JSON.`,
});
const plan = JSON.parse(result.result_text);
// Hand to project_create with scenes mapped from plan.scenes
```

This is the **flagship recipe**. See playbook: `paste-link-get-project.md`.

### 2. `auto_critique_video` — Post-generation drift gate

**Goal:** every time a `/film` or `/story` scene renders, call marlin-video to grade it 0-10 on prompt-adherence + cast-lock + motion budget. Auto-trigger `director_re_render` on scenes below threshold.

```typescript
await create_media({
  action: "generate", model_override: "marlin-video",
  source_url: SCENE_VIDEO_URL,
  prompt: `Analyze this generated video. Return JSON:
{
  "drift_score": <0-10, where 10 = severe character/style drift>,
  "motion_budget": "<still|low|medium|high|chaotic>",
  "scene_continuity_issues": [{"timestamp": <float>, "issue": "<string>"}],
  "cast_lock_violations": "<empty-array OR list of mid-clip face/identity changes>",
  "anatomy_issues": "<empty OR list of e.g. 'three legs at 5.2s', 'fused hands at 8.0s'>"
}
Return ONLY the JSON.`,
});
```

Surfaces as a red-dot indicator on the scene card when `drift_score >= 7`. The same prompt can be re-pasted by `director_re_render` to give the regen a concrete "fix these" target.

### 3. `lv2v_quality_monitor` — Real-time stream scoring (deferred)

**Goal:** during an active LV2V stream, tick every 8s, grab the last 5s of output, score for prompt-adherence + stability against the current scope_prompt. When score < 4 for 3 consecutive ticks → suggest `scope_control({ reset_cache: true })`.

```typescript
// every 8s:
const clip = await trickle_capture({ stream_id, last_n_seconds: 5 });
const result = await create_media({
  action: "generate", model_override: "marlin-video",
  source_url: clip.url,
  prompt: `Rate this 5s clip on prompt-adherence (0-10) and visual stability (0-10)
given the target prompt: "${current_scope_prompt}".
Return JSON: { "prompt_score": <0-10>, "stability_score": <0-10>, "suggestion": "<one short fix>" }`,
});
```

**Status:** designed, not yet implemented. Needs `trickle_capture` helper to snapshot the last N seconds of an active stream as an MP4. Tracked as a Phase-2 follow-up.

### 4. `storyboard_from_video` — Onboarding flow (paste-link wedge)

**Goal:** user pastes a URL into chat as their FIRST action. Preprocessor detects the URL → runs `scene_tag_reel` recipe → hands JSON to `project_create` → user has a fully-populated project in 30 seconds.

```typescript
// preprocessor in lib/agents/preprocessor.ts:
if (URL_REGEX.test(userMessage)) {
  const result = await create_media({
    action: "generate", model_override: "marlin-video",
    source_url: extracted_url,
    prompt: SCENE_TAG_PROMPT,  // recipe 1
  });
  const plan = JSON.parse(result.result_text);
  await project_create({
    title: plan.title,
    brief: plan.overall_mood,
    visual_style: plan.dominant_palette.join(", "),
    scenes: plan.scenes.map(s => ({ prompt: s.remix_prompt, ... })),
  });
}
```

Pairs with the existing `/analyze` flow which is image-only today.

### 5. `moderation_check` — Pre-publish safety pass

**Goal:** before any `moodboard_publish` / `publora_publish_post` / case-studies cards-array commit, run a marlin-video pass against the final reel to flag content that may violate platform policies. Cheap ($0.01-0.05) and runs in parallel with the publish prep.

```typescript
await create_media({
  action: "generate", model_override: "marlin-video",
  source_url: REEL_URL,
  prompt: `Flag any content that may violate platform safety policies.
Return JSON:
{ "flagged": <bool>, "categories": ["<nudity|weapons|violence|recognizable_real_people_especially_minors|trademarked_logos|copyrighted_characters>"],
  "severity": "<low|med|high>", "notes": "<one paragraph>" }`,
});
// Block publish on severity=high; warn on med; log to cost report.
```

## The agent orchestration pattern (the "wow")

```
USER:   "make this in my brand style: https://tiktok.com/@.../video/123"
  ↓
[1] preprocessor detects URL → marlin-video (scene_tag_reel)
        → returns JSON { title, scenes:[1..6], palette, mood }
  ↓
[2] project_create with the extracted plan
  ↓
[3] In parallel:
      krea-2-large × 6 (regenerate keyframes in user's style)
      seedance-i2v × 6 (animate each scene)
      minimax-music (palette-matched music bed)
  ↓
[4] AUTO-CRITIQUE: marlin-video × 6 (drift gate)
        → re-render any scene with drift_score >= 7
  ↓
[5] ffmpeg-concat → final reel
  ↓
[6] moderation_check (pre-publish gate)
  ↓
SHIP: viewer URL + side-by-side comparison toggle (original vs remix)
```

**Total wall-clock:** 4-7 minutes. **Total cost:** ~$2-4. **What it unlocks:** every user video becomes a personalized creative starter — the most-asked-for feature in the AI video category.

## Cross-surface parity

| Surface | Entry point | Today | After full wire-through |
|---|---|---|---|
| MCP | `create_media({ action: "generate", model_override: "marlin-video", ... })` | ✅ live (after lib/sdk/capabilities.ts deploy) | + dedicated `analyze_video` MCP tool (Phase 2) |
| Webapp | "paste any video URL into chat → auto-detected → /remix flow" | ✅ via preprocessor URL detect | Right-click video card → "Tag scenes" / "Critique this" / "Remix" |
| Slash | `/remix <url>` and `/critique <card-ref>` | not yet (Phase 2 — needs slash command in lib/skills/commands.ts) | Both live |
| CLI | `livepeer video tag <url>` / `livepeer video critique <card>` | not yet | Both live |

## Honest deferrals

- **Max input ~2 minutes (240 frames @ 2fps).** For longer videos, chunk via `ffmpeg-trim` first, run scene_tag_reel per chunk, concatenate results.
- **Output is text, not media.** The `marlin-video` cap returns a JSON document in `result_text`. Downstream tools need to know to parse it — the action enum is still `generate` for compatibility but the result handling diverges.
- **Live LV2V scoring (recipe 3) needs a streaming snapshot helper.** Designed but not implemented; tracked as follow-up.
- **Image-only or single-frame video analysis is overkill.** For a still, use `gemini-text` with `image_url` instead — much cheaper and faster.