--- title: "Media to Script — paste image / video / audio, get a human-readable narrative" tier: hero format: narrative theme: understand persona: educator, accessibility, journalist, archivist, parent, museum, podcaster duration: matches source (single-pass, finite media) budget_usd: $0.01–$0.50 per source, plus optional voice render caps: ["gemini-text", "marlin-video", "wizper", "gemini-tts", "chatterbox-tts"] skills: ["media-to-script"] showcases: [] reliability: 4.0 # path-dependent: image ≈4.7 (5 − vision misreads .3); video ≈3.9 (5 − marlin .8 − audio-blind merge .3); audio ≈4.8 (5 − asr .2); optional tts −.2; ≤5 stages, single-pass; no shipped showcase yet --- # Media to Script — paste media, get a script anyone can read Paste an **image URL, video URL, or audio URL** (or upload the file). The agent looks at / listens to it, then writes a **human-readable narrative script** at the reading level you choose — for an 8-year-old, for a general adult audience, for a museum audio guide, for a podcast pitch, for an audio-described accessibility version. Single-pass. Finite media. One output: a Markdown script you can paste into a doc, a teleprompter, an Anki card, a screen reader, a TTS engine, or directly into the talking-avatar playbook to make a video of it. ## What you'll get A Markdown file with: - **Title** — a short reader-friendly title for the piece - **Synopsis** — one sentence; what this is - **Narrative script** — the actual reading, paragraphed for breath, length matches the source duration (~150 spoken words per minute) - **Optional sections** — characters, settings, key moments (for video); subjects, palette, mood (for image); speakers, topics, quotes (for audio) - **Reading-level note** — confirms which audience the script targets - **Pronunciation hints** — proper nouns, acronyms, foreign words Optional second pass: pipe the script through `gemini-tts` (or the talking-avatar playbook) to ship a voice rendering alongside. ## How the agent should run this (interaction contract) 1. **CONFIRM (one message, ≤1 question):** restate in 1 line ("90s museum-guide script from your image, warm-curious tone, ~$0.01, under a minute") + real choices: reading level (the one question worth asking) · tone · script-only vs +voice render. If the brief is clear, state assumptions and run. 2. **PREVIEW CHECKPOINT:** show the raw Stage-2 understanding (~$0.005–0.05) before the narrative rewrite — misidentified subjects, misread text, or a missed speaker get caught here for pennies, not in the polished script (or a $0.05/min voice render). 3. **NARRATE:** image/audio understanding lands in <30s; marlin on a ≤2-min clip takes ~30–90s — say which is running. If a voice render fires, post the job line and poll `get_create_media` every 10–15s; never silent >2 min. 4. **FAIL GRACEFULLY:** marlin noisy/garbled → re-prompt tighter → `nemotron-omni` as alternate video reasoner; `wizper` fail → `nemotron-asr`; video >2 min → chunk at scene boundaries, never one giant call; `gemini-tts` fail → `chatterbox-tts`. ≤2 retries per stage; a script without the optional voice beats nothing. 5. **DELIVER:** `script.md` + `understanding.json` + one honest line ("vision can misread small text/obscure faces; marlin ignored the audio track — transcript merged via wizper" or "not merged") + ONE next step ("voice it with gemini-tts, or chain into talking-avatar"). ## When to use this - **Audio description for the blind / low-vision** — turn a 2-min video into a clean spoken description - **Museum / gallery audio guide** — turn a painting image into a 90-second listening track - **Kids' explainer** — paint a documentary clip into a 5-year-old-friendly bedtime story - **Journalism pre-write** — turn a press conference video into a draft article - **Archivist note** — describe a historical photo in usable cataloguing language - **Podcast notes** — transcribe + re-narrate an audio interview as a readable summary - **Reading practice (L2)** — rewrite a video's content at intermediate reading level in target language - **Talking-avatar feeder** — pipe the script straight into `talking-avatar.md` to make a video of the script ## YAML brief schema ```yaml project: name: "Painting study — Vermeer's Milkmaid" source: # Pick ONE image_url: "https://example.com/painting.jpg" video_url: "https://www.youtube.com/watch?v=…" # or any video URL; yt-dlp handles fetch audio_url: "https://example.com/podcast.mp3" file: "manuscript.pdf" # uploaded file # Output controls target_language: "English" # any gemini-text supported language reading_level: "general-adult" # 5yo | 10yo | teen | general-adult | academic | museum-guide target_length: "90 seconds" # "30 seconds" | "1 minute" | "5 minutes" | "as long as needed" tone: "warm-curious" # warm-curious | reverent | clinical | playful | journalistic output_format: "script" # script (default) | bulleted-notes | dialogue (multi-speaker) | poem # Optional: chain into a voice render also_synthesize_audio: false # true → run gemini-tts on the script too also_make_talking_avatar: false # true → chain into talking-avatar.md project_slug: "vermeer-milkmaid" ``` ## Pipeline ``` Stage 1 — DETECT MEDIA TYPE Inspect source URL / file extension → {image | video | audio} Stage 2 — UNDERSTAND (cap depends on media type) IMAGE → gemini-text with image input (vision) Returns: {subjects, setting, palette, mood, composition, narrative} VIDEO → marlin-video (fal-ai/marlin) Returns: scene boundaries with timestamps, camera motion, subjects per scene, dominant colors, dialogue (if any) NOTE: marlin-video takes the clip via the `video_url` param (raw SDK /inference), NOT create_media's source_url. Keep clips ≤2 min per call. AUDIO → wizper (fal-ai/wizper) Returns: timestamped transcript with speaker tags (Optional pre-step: separate music + voice via mirelo-sfx-v2v) Stage 3 — REWRITE AS NARRATIVE gemini-text takes the understanding output + the brief's reading_level, tone, target_length, output_format. Writes the narrative script. Prompt template: "Rewrite the following media understanding as a {{output_format}} for a {{reading_level}} audience. Tone: {{tone}}. Target length: {{target_length}}. Language: {{target_language}}. Rules: - Lead with the most evocative detail. - Use concrete nouns. Avoid jargon. De-acronymize anything 3+ letters. - Match natural reading rhythm; vary sentence length. - If this is an audio description for the blind, prefer spatial words ('top-left', 'in the foreground', 'behind her') over visual-only adjectives. - For kids' levels: shorter sentences, sensory language, no abstract concepts. Media understanding: {{stage_2_output}} Brief: {{brief_metadata}}" Stage 4 (optional) — VOICE THE SCRIPT If also_synthesize_audio: pipe script → gemini-tts. If also_make_talking_avatar: chain into talking-avatar.md. Stage 5 — DELIVER Output: - script.md (the artifact) - understanding.json (raw Stage-2 output, kept for re-runs in different tones) - Optional: audio.mp3 (if Stage 4 ran) - Optional: talking-avatar.mp4 (if chained) ``` ## Cost guide | Media type | Cap | Cost per source | |---|---|---| | Single image | gemini-text (vision) | ~$0.005 | | Video, ≤ 2 min | marlin-video | ~$0.015 / 1K output tokens (~$0.05 for a 2-min clip) | | Audio, ≤ 10 min | wizper | ~$0.10 | | Narrative rewrite | gemini-text | ~$0.005 | | **Subtotal (script only)** | | **$0.01–$0.20** | | Voice rendering (optional) | gemini-tts | ~$0.05/min of output | | Talking-avatar (optional) | full talking-avatar.md cost | ~$0.30/20s | For most one-shot uses: **under 50 cents**. For "watching a 1-hour talk and writing a 5-min audio guide": ~$1.50. ## Reading-level table | Level | Sentence length | Vocabulary | Concept abstraction | |---|---|---|---| | **5yo** | 5–8 words | concrete nouns, sensory verbs, no jargon | only what can be touched/seen/heard | | **10yo** | 8–12 words | concrete + occasional abstract | "fairness", "respect" OK; no economics terms | | **teen** | 12–18 words | mixed; some technical | one new concept per paragraph | | **general-adult** (default) | 15–25 words, varied | educated lay reader | reasonable jargon if defined | | **academic** | 20–35 words | discipline-specific | formal register, citation-friendly | | **museum-guide** | 12–18 words | evocative; sensory | one historical anchor per piece | ## Multi-modal outputs The default is one script. You can also request: - **Bullet notes** — for slide decks; key facts only, no narrative flow - **Dialogue (multi-speaker)** — two-host podcast format; agent identifies natural Q/A breaks in the source - **Poem** — for short evocative pieces; haiku for images, sonnet for short videos - **Bilingual side-by-side** — same script in source + target language, paragraphs aligned ## Cross-surface | Surface | How to invoke | |---|---| | **MCP (agent)** | `submit_creative_job({ playbook: "media-to-script", source, reading_level, tone, target_length })` | | **CLI** | `livepeer playbook media-to-script --src --level general-adult --tone warm-curious --length "90 seconds"` | | **Webapp** | paste any media URL in chat → preprocessor offers "Describe this for me?" → runs the playbook | | **Webapp (right-click)** | right-click any media card → "Generate script" | ## Chaining with other playbooks | Chain | Result | |---|---| | media-to-script → talking-avatar | Paste a famous painting → get a museum-guide audio + talking-avatar reading it | | media-to-script → video-translate-multilingual | Paste a video → get a script → translate the script → render an N-language audio guide | | media-to-script → suno-music-video | Paste a poem → get a musical reading → music-video with lyric typewriter | | paste-link-get-project → media-to-script | Decompose a video, then re-narrate the decomposition for an audience | ## Anti-patterns - **Don't paste an entire feature film** — chunk at chapter / scene boundaries first. marlin-video's effective sweet spot is ≤2 min per call. - **Don't ask for "academic" level on a sensory image** — paintings deserve evocative language, not vocabulary like "chromatic saturation." - **Don't skip Stage 2** — gemini-text doesn't see images; it reads text. The vision/transcription pass is mandatory. - **Don't claim the script is original** — it's a rewriting of the source. If publishing, attribute the source. - **For copyrighted material**: this playbook generates a derivative description. Fair-use for criticism / commentary / accessibility is yours to judge. For published audio guides / commercial reuse, get a license. ## Honest limits - **Vision models are imperfect.** Gemini's image understanding is strong but not perfect — it may misread small text, misidentify obscure people, hallucinate details in low-resolution images. - **marlin-video doesn't watch sound.** Audio in a video is ignored. For dialogue / music context, run `wizper` on the audio track separately and merge into the narrative. - **Long audio gets pricey.** wizper at 10 min costs ~$0.10; at 2 hours, ~$1.20. For long-form, consider chunking + summarizing per chunk before the rewrite step. - **"All ages" is not a real audience.** Pick a level (5yo / 10yo / teen / adult). "All ages" means nobody. - **No image-to-poem in v1 unless you ask.** Default is narrative script. Set `output_format: poem` explicitly. ## Live-stream variant For real-time / live URLs (YouTube Live, Twitch, etc.), use the sibling playbook **`live-stream-commentary.md`** instead. Same understanding caps, different lifecycle — continuous frame sampling, change-detection, rolling emissions instead of one-shot output. ## Cross-reference - `paste-link-get-project.md` — also takes a video URL, but the output is a remixable storyboard project (regenerate every scene in your style). Different goal: media-to-script outputs PROSE, paste-link outputs a PROJECT. - `talking-avatar.md` — natural downstream chain: script → speaking avatar. - `voice-memo-to-video.md` — close cousin but specific to voice memos with B-roll. ## Reproduce — sample MCP call ```typescript // 1. Detect type → run appropriate understanding cap const understanding = await create_media({ action: "generate", model_override: "gemini-text", // for image; use marlin-video for video; wizper for audio prompt: `Look at this and describe everything important. Include subjects, setting, palette, mood, composition. Be thorough but factual.`, image_url: source.image_url, }); // 2. Rewrite as narrative const script = await create_media({ action: "generate", model_override: "gemini-text", prompt: NARRATIVE_REWRITE_PROMPT .replace("{{understanding}}", understanding.text) .replace("{{reading_level}}", brief.reading_level) .replace("{{tone}}", brief.tone) .replace("{{target_length}}", brief.target_length) .replace("{{target_language}}", brief.target_language), }); // 3. Save as markdown fs.writeFileSync(`public/chapters/${brief.project_slug}/script.md`, script.text); // 4 (optional) — pipe into talking-avatar if (brief.also_make_talking_avatar) { await runPlaybook("talking-avatar", { source: { text: script.text }, target_language: brief.target_language }); } ``` ## Trace Documented 2026-06-04. Uses three already-registered BYOC caps (gemini-text, marlin-video, wizper) plus optional gemini-tts. No new capability registrations needed; ships as a docs-first playbook with chaining patterns to existing playbooks.