---
title: "Live Stream Commentary — paste live URL, get rolling understanding"
tier: hero
format: live-commentary
theme: realtime-understand
persona: accessibility, monitor, educator, journalist, broadcaster
duration: unbounded — emits as long as stream runs
budget_usd: $0.50–$2 per hour of watching (depending on tick rate)
caps: ["gemini-text", "nemotron-asr", "wizper", "marlin-video"]
skills: ["live-stream-commentary", "media-to-script"]
showcases: []
status: "design — Canvas v1 implementation pending"
reliability: 2.0 # design-only — Canvas v1 not built, no runnable E2E surface today; basis: 5 − live .8 − per-tick vision/marlin .8 − asr .2 − 6-stage tick loop .2 − ingest fragility (HLS/yt-dlp) .2 − fuzzy change-detection .2 + .4 each component cap individually proven; floor-adjacent until v1 ships
---

# Live Stream Commentary — paste live URL, rolling understanding as it happens

The live variant of `media-to-script.md`. Paste a **live stream URL** (YouTube Live, Twitch, a public HLS endpoint), pick a tick rate (default every 2 seconds), and the agent emits a rolling commentary of what's happening — emitting only when things change, not on every tick.

Designed for accessibility (audio description of live events), monitoring (newsroom desk watching multiple feeds), classroom (live lecture transcript + scene description), or just curiosity ("what's going on in this stream right now?").

## What you'll get

A **streaming commentary feed** that updates only when the scene materially changes. Default output format:

```
[00:00:00] Aerial drone shot of a coastal city at dusk. Orange sky, dark water,
           distant cargo ship on the horizon.
[00:00:14] Cut to street level. Two reporters in front of a marketplace,
           handheld camera, microphones visible. Crowd noise.
[00:00:42] Same shot. Continues unchanged.   ← suppressed; not re-emitted
[00:01:08] Cut to studio. News anchor at a desk, "BREAKING" banner along bottom,
           map of the region in the background.
…
```

Each emission has a timestamp + a paragraph describing the scene. Re-emissions are suppressed unless the change-detection threshold is exceeded.

## How the agent should run this (interaction contract)

1. **CONFIRM (one message, ≤1 question):** be honest first — "the live rolling loop is design-status; no Canvas v1 / MCP v2 surface exists yet." Offer the 3 things that DO run today: a one-shot read (grab a frame + recent audio → `gemini-text` vision + `wizper`, ~$0.01) · the finite-media sibling `media-to-script.md` on a recorded segment · a manual N-tick loop the agent drives by hand (state tick rate + `max_cost_usd`, e.g. "2s ticks ≈ $1.80/hr"). One question: which of the three.
2. **PREVIEW CHECKPOINT:** ONE sample tick (~$0.001–0.01) — a single frame description (+ optional ASR snippet) in the chosen emission format — BEFORE committing to a long watch. Tune format/threshold on that sample.
3. **NARRATE:** if hand-driving a loop, emit the rolling commentary itself as the progress signal; also post a meta-line every ~2 min ("tick 60/1800, 9 emissions, ~$0.06 spent") and respect `max_cost_usd` / `max_emissions` hard stops.
4. **FAIL GRACEFULLY:** HLS/yt-dlp pull fails (member-only / DRM / geo) → ask for a public .m3u8 or a recorded clip → fall back to `media-to-script`; `nemotron-asr` unavailable → `wizper` batch chunks; marlin rollup noisy → `nemotron-omni` or plain `gemini-text` per-frame. ≤2 retries per tick, then skip the tick — a gappy commentary beats a dead session.
5. **DELIVER:** the timestamped emission log + one honest note ("N ticks watched, M emissions; change-detection is fuzzy — duplicates possible; this was an agent-driven loop, not the streaming product") + ONE next step ("run media-to-script on the recording, or set up a longer watch with a cost cap").

## Architectural reality (read this first)

**MCP transport is request-response, not streaming.** A pure MCP call returns one answer; it can't "watch a live stream and emit updates" by itself. There are two architectures, both valid:

### Path A — Canvas v1 (recommended MVP)

```
Live URL (YouTube/Twitch/HLS)
    ↓ (hls.js in browser, or yt-dlp → HLS reflector)
HLS stream
    ↓ (lib/stream/frame-extractor.ts at 0.5 fps — already on the stack)
Frame + recent audio chunk
    ↓ (per tick)
gemini-text(image) → "scene description"
    ↓
Change-detection (cosine similarity vs last emit, threshold 0.7)
    ↓ (emit only if changed)
Rolling commentary card on canvas
```

Pros: ~1 day of work. Reuses existing `frame-extractor.ts`. No new server-side infrastructure. Works in any modern browser.

Cons: only works in the storyboard browser tab. CLI / Claude Desktop / external MCP clients can't subscribe.

### Path B — MCP v2 (longer-term, all-surface)

```
User calls submit_live_understanding({url, interval_sec, change_threshold})
    ↓ returns subscription_id immediately
Server-side worker:
    pulls HLS via ffmpeg
    extracts frame + N-sec audio every interval_sec
    runs gemini-text (vision) per tick
    deduplicates against last emission
    persists emissions to a per-subscription log
Client polls get_live_understanding(subscription_id, since: <last_ts>)
    OR subscribes via SSE/WebSocket
```

Pros: works on every MCP client (Claude Desktop, CLI, third-party agents, Storyboard browser, etc.). Persistent across page refresh.

Cons: ~3-5 days of work. Requires new storyboard MCP server infrastructure: background worker, HLS pull container, subscription state store, SSE endpoint or websocket. Operational ongoing — workers need to be reaped when subscription ends.

**Recommended order:** ship Canvas v1 first to learn the UX, then promote to MCP v2.

## YAML brief schema

```yaml
project:
  name: "BBC News live commentary for L2 English learners"
  source:
    live_url: "https://www.youtube.com/watch?v=…"  # YouTube live, Twitch, or direct HLS .m3u8
    auth_headers: {}                                # optional, for member-only streams

  # Tick & emission
  tick_interval_sec: 2          # 2 (default) | 5 | 10 | 30
  change_threshold: 0.70         # 0.0 (emit every tick) ↔ 1.0 (emit only on major scene change)
  emission_format: "audio-description"   # audio-description | journalistic | learner-l2 | one-sentence
  target_language: "English"
  reading_level: "general-adult"

  # Modality
  include_audio: true            # extract recent audio every N ticks too (via nemotron-asr; wizper fallback)
                                  # cheaper to enable on news/dialogue streams; skip for music/sports

  # Session lifecycle
  max_duration_minutes: 60       # auto-stop after this even if stream continues
  max_emissions: 200             # auto-stop after this many emissions
  max_cost_usd: 5.0              # hard cost cap

  project_slug: "bbc-news-live"
```

## Pipeline (Canvas v1)

```
Stage 1 — INGEST
  hls.js (or yt-dlp → local HLS reflector) → continuous video element in browser

Stage 2 — TICK (every tick_interval_sec)
  Frame: canvas.captureStream() → grab one frame as JPEG
  Audio (optional): MediaRecorder buffer last 2-3s → wav blob

Stage 3 — UNDERSTAND
  Per tick:
    gemini-text(image: <frame>, prompt: <scene-description template>) → 1-2 sentence description
  Every 5 ticks (if include_audio):
    nemotron-asr(audio: <wav>) → transcript snippet (streaming, punctuated, 40 langs — prefer for live;
      wizper is the batch fallback) → merged into next emission

Stage 4 — CHANGE-DETECT
  Hash new description (first 50 chars) and embed it (gemini-embed if available, else cosine on raw).
  Compare to last EMITTED description.
  If similarity > change_threshold, do NOT emit.
  If similarity < change_threshold, emit with timestamp.

Stage 5 — EMIT
  Append to rolling commentary card on canvas.
  Optional: feed to gemini-tts for live audio rendering (accessibility use case).

Stage 6 — LIFECYCLE
  Stop when:
    user clicks Stop, OR
    max_duration_minutes reached, OR
    max_emissions reached, OR
    max_cost_usd reached, OR
    HLS stream goes dead.
```

## Emission formats

| Format | Style | Use case |
|---|---|---|
| **audio-description** (default) | Spatial, present-tense, factual. "A man in a blue jacket walks left-to-right across the screen." | Blind / low-vision viewers |
| **journalistic** | Active voice, news register. "Cut to studio. Anchor introduces a breaking story." | Newsroom desk monitoring |
| **learner-l2** | Slow vocabulary, present simple tense. "Two people are talking. They look happy." | Language learning |
| **one-sentence** | Single sentence per emission, max 12 words. | Phone notifications, alerting |
| **dramatic** | Stylized; reads like a film synopsis | Entertainment commentary |

## Cost guide

For a 1-hour watch:

| Mode | Vision calls | Audio calls | Total |
|---|---|---|---|
| **Lean** (10s ticks, no audio) | 360 × $0.001 | 0 | ~$0.36 |
| **Default** (2s ticks, no audio) | 1800 × $0.001 | 0 | ~$1.80 |
| **Rich** (2s ticks + audio every 10s) | 1800 × $0.001 + 360 × $0.003 (wizper chunks) | | ~$2.88 |
| **Sport / Music** (10s ticks, no audio, music-heavy → less emission anyway) | 360 × $0.001 | 0 | ~$0.36 |

Real cost is lower than gross because **change-detection suppresses re-emissions** — you pay for the VISION call but you don't pay for redundant LLM rewrite calls (vision already produced the description).

Hard cost cap (`max_cost_usd`) is enforced server-side once MCP v2 ships; Canvas v1 enforces client-side via emission count + tick count.

## Honest limits

- **Public-only HLS.** YouTube live works via yt-dlp / hls.js if the stream is public. Member-only, age-gated, geo-blocked, or DRM-protected streams break. Twitch generally works. Direct HLS .m3u8 endpoints work universally.
- **Audio modality is optional.** Sports / music streams don't benefit much from wizper; news / dialogue streams do. Default off.
- **Latency floor ~2s.** The first emission lands ~2s after you start (one tick) plus ~1s of vision call latency. Real-time-real-time isn't achievable on this stack today.
- **Change-detection is fuzzy.** A cosine similarity on text descriptions isn't perfect; sometimes the agent emits twice for "the same" scene if the wording drifts. Tune `change_threshold` per stream.
- **Long sessions are bounded by cost, not by infrastructure.** A 24-hour watch at default tick rate is ~$43. Set `max_duration_minutes` aggressively.
- **Canvas v1 is browser-only.** If you close the tab, the session ends. MCP v2 (when shipped) fixes this.
- **YouTube ToS.** Downloading copyrighted YouTube content has license implications. Live commentary for accessibility / journalism / education is yours to judge; commercial republication is not free.

## Cross-surface (current + planned)

| Surface | Status |
|---|---|
| **Webapp (Canvas v1)** | Planned. New "Watch Stream" card type. ~1 day to build. |
| **CLI** | Awaits MCP v2. Will be `livepeer watch-stream <url> --tick 2s --threshold 0.7 --output rolling.log`. |
| **MCP (Claude Desktop / external agents)** | Awaits MCP v2: `submit_live_understanding` + `get_live_understanding` tools. |
| **Chat preprocessor** | Will detect live URLs (twitch.tv/*, youtube.com/watch?v=*&live=*, *.m3u8) → offer "Watch this stream?" |

## Real-use scenarios

| Persona | What they need | What this playbook gives |
|---|---|---|
| **Blind / low-vision viewer** | Real-time audio description of a live news broadcast | `audio-description` emissions piped through gemini-tts → spoken in their ear via screen reader |
| **Newsroom desk** | Live monitoring of 3 competitor broadcasts at once | 3 stream cards on the canvas, each emitting journalistic-style commentary; flag on keyword match |
| **L2 language teacher** | A live English broadcast simplified to A2 level for the class | `learner-l2` emissions in slow English; learners listen + read in parallel |
| **Sports producer** | "What's happening in this feed?" pre-air check on a long-tail stream | `dramatic` emissions; flag on score changes |
| **Parent / educator** | A live cooking demo for kids, narrated as a 5-year-old story | `learner-l2` + reading_level `5yo` → naive but accurate scene-by-scene |
| **Conference organizer** | Live captions + scene context for a hybrid event | both `include_audio: true` (wizper transcript) + visual scene description |

## Chaining

| Chain | Result |
|---|---|
| live-stream-commentary → talking-avatar | Rolling commentary read by a virtual presenter |
| live-stream-commentary → ffmpeg-burn-subtitles | Overlay emissions as live subtitles on a recorded copy of the stream |
| live-stream-commentary → translate (gemini-text) per emission | Each emission also translated to N other languages, fanned out |

## Implementation status (2026-06-04)

This playbook ships as **docs only** in PR #370. The Canvas v1 implementation is real engineering work and lands in a follow-up PR:

- [ ] Canvas card type "Watch Stream" — accepts URL, tick rate, threshold
- [ ] hls.js integration for HLS pull
- [ ] yt-dlp reflector for YouTube live URLs (server-side; the browser can't yt-dlp directly)
- [ ] Per-tick gemini-text vision call
- [ ] Cosine-similarity change detection (or text-hash diff for v1)
- [ ] Rolling emission renderer with timestamps
- [ ] Cost/emission/duration caps
- [ ] /watch slash command

Once Canvas v1 stabilizes (1-2 weeks of real use), the same patterns get hoisted to MCP v2 with a background worker + SSE.

## Cross-reference

- `media-to-script.md` — the finite-media sibling. Same understanding caps, single-pass output.
- `paste-link-get-project.md` — pre-recorded video → remixable storyboard project (different goal).
- `live-cast-member.md` — live LV2V transform on a stream (different layer: video out, not text out).
- `live-director.md` — Scope-based live composition.

## Trace

Designed 2026-06-04. Caps: `gemini-text` (already registered, vision mode), `wizper` (already registered, audio path), `marlin-video` (already registered, used optionally for longer rollup ticks). No new BYOC registrations needed. Canvas v1 implementation pending — request via the storyboard issue tracker if you'd like priority.