--- reliability: 4.0 # animated mode: 5 − image gen .3 − i2v .6 − tts .2 − music .2 − 7-stage path .2 + .5 proven Lighthouse Bird both-mode example; webtoon-only mode runs ~4.5 (no i2v/tts/music — keyframes + deterministic captions/stitch) --- # Story-to-screen — agent playbook ⭐ flagship > Paste this whole file (with your story pasted into the BRIEF) into Claude Code, Claude Cowork, Codex, or Gemini Code. **Paste a story → get a webtoon, an animated video reel, or both.** The agent breaks the story into N visual panels, locks the cast + style, generates the panels with character-anchored consistency, and finishes the chosen output mode. Works for short stories, essays, fables, vignettes, children's books, mood pieces. ## What you'll get Depending on `output_mode` you pick: - **Webtoon mode** — 6–12 vertical 9:16 panels, narration + dialogue burned in as caption boxes (bottom of each panel), assembled as a vertical-scroll page. Reads like a Webtoon / LINE Manga / Manhwa scroll. ~$1.20–$2.40 per story. - **Animated video mode** — same 6–12 panels but each animated for 10 seconds via real `seedance-i2v` (or `ltx-q-i2v` for higher fidelity), stitched into a single reel with a warm narrator voice-over and a unified instrumental score that runs the **full** length (muxed with `audio_fill:"loop"`), vertical-fit via blurred-source padding so there's no aspect distortion. ~$8–$14 per story for 6 panels at 10s each. - **Both modes** — share the same N keyframes, double the finishing, ~$9–$16 total. Either way, the output is **a faithful visual telling of the user's story** with consistent characters across every panel and a consistent painterly style. No character drift, no costume changes, no narrative jumps. **Time**: ~10–18 minutes wall clock depending on mode. **Your attention**: ~5 minutes (one approval gate on the cast + scene breakdown before fire). **Cost**: $1.20–$12 depending on mode + scene count. --- ## How to use this in any agent framework The playbook is just text — works the same in Claude Code, Claude Cowork, Codex, Gemini Code, ChatGPT-with-MCP, etc. Steps: 1. Paste this whole file into the agent. 2. Paste YOUR story into the `story_text` field of the YAML brief below. 3. Pick your `output_mode` and other parameters. 4. The agent reads your story, breaks it into scenes, surfaces the cast + scene plan for your approval. 5. You say "go" (or redirect specific scenes). 6. The agent calls the storyboard MCP (`mcp__storyboard__*`) for everything else — keyframes via `generate_project` (pass `aspect_ratio`) or `create_media`, optional animation via `create_media` action=animate (`model_override: "seedance-i2v"` or `"ltx-q-i2v"`, `resolution: "auto"`), voice-over via `create_media` action=generate model=gemini-tts, music via `create_media` action=music **with `lyrics_prompt: "[Instrumental]"` (or real lyrics in [Verse]/[Chorus] markup if a vocal track is intended) — never omit it; minimax-music sings whatever is in `lyrics_prompt` verbatim**. 7. Stitches (`ffmpeg-concat`), lays the score under the full reel (`create_media model_override:"ffmpeg-mux"`, `audio_fill:"loop"`), exports to aspect (`ffmpeg-export`), and ships a showcase HTML page. The agent framework you use just needs storyboard MCP access. Everything else (story parsing, scene breakdown, cast extraction) happens in the agent's own reasoning — no specialized tool required. ## Tell the agent about the story ```yaml story_text: | # Paste your story here. Any length up to ~2000 words. # Public-domain text or your own writing only — the agent will refuse # to render copyrighted text verbatim and will ask you to paraphrase. output_mode: # webtoon | animated-video | both # webtoon = cheapest, fastest, scroll-page output # animated-video = pricier, ~60–90s reel with narration + music # both = share keyframes, ship both pages panel_count: # auto (default — agent picks 6–12 based on story length) # OR explicit number 4–16 style_brief: # Visual style for all panels. Examples: # - "Studio-Ghibli-inspired soft hand-painted watercolor, # warm earth palette, golden coastal light" # - "Pixel-art 16-bit JRPG, vibrant chiptune-era palette" # - "Pixar 3D animated, cinematic lighting, glossy character surfaces" # - "Inked manga black-and-white, screentone shading, dramatic angles" # - "Mucha art-nouveau, decorative borders, flat color blocks" cast: # Auto-extracted by the agent from your story. Override if needed. # Each character ~30 words of physical spec: # - name: Mira # age: ~9 years old # physical: curly red braids, freckled cheeks # wardrobe: blue wool sweater under yellow raincoat # expression: alert wide blue eyes, gentle expression # The agent embeds this verbatim in every panel prompt for consistency. aspect: # 9:16 (default — works for both webtoon scroll and TikTok-shaped video) # 16:9 (cinematic landscape — for traditional video output) # 4:5 (Instagram-feed friendly) narration_voice_brief: # For animated-video mode. Examples: # - "warm female storyteller, slow contemplative pace, soft and tender — # like reading a bedtime story" # - "deep grandfather voice, weathered, slight smile in the tone" music_brief: # For animated-video mode. Use the music-brief template from # local-food-short / viral-dance-short: BPM + genre + structure + anti-guide. # For story-to-screen, more atmospheric: # - "Solo piano + soft strings, intimate Studio Ghibli-tier score, # builds gently from contemplative open to warm major-key release # in the closing 10 seconds. No vocals. Tender." story_slug: # kebab-case for output. e.g. the-lighthouse-bird / the-baker-and-the-fox. ``` --- ## How the agent should run this (interaction contract) 1. **CONFIRM (one message, ≤1 question):** restate in 1 line ("your story → N panels as , ~$1.20–2.40 webtoon / ~$8–14 animated, ~10–18 min") + real choices: webtoon vs animated-video vs both · panel count (auto picks 6–12) · style brief. Phase A's SINGLE approval gate is the only question round — never 5 separate questions. 2. **PREVIEW CHECKPOINT:** the Phase-A bundle (cast bible + beat sheet + color script + continuity matrix, $0) and then the keyframe contact sheet (~$1.20) BEFORE any i2v fires — a $0.20 panel re-fire beats a wasted $1+ animation per panel. 3. **NARRATE:** animate in waves of ≤4–5 (a 10s clip runs ~2–5 min); poll `get_create_media` every 10–15s, post a 1-liner per wave with job ids + ETA, never silent >2 min. 4. **FAIL GRACEFULLY:** `seedance-i2v` hang/"fetch failed" → retry once → `ltx-q-i2v` (resolution:"auto", `num_frames:241` for 10s) → Ken-Burns slow zoom (already the sanctioned path for 3+-character panels); gpt-image 422 → soften wording → flux-dev. 1-line what/why/fallback each swap; ≤2 retries per panel; a partial reel with a note beats nothing. 5. **DELIVER:** scroll/reel + one honest line ("cast held N/N panels; i2v can morph a face mid-clip — motions kept gentle, one-motion-only rule applied") + ONE next step (the other output mode reuses these keyframes for ~half price). ## Quality controls (auto-applied) The constraint set that keeps the output coherent across N panels. Organised by **phase** — what the agent does BEFORE any expensive call, what runs DURING generation, what runs AFTER. Most quality wins are upstream; cheaper to fix a bad cast spec than to re-fire 6 keyframes. ### Phase A — Pre-render (story → plan, no MCP calls yet) These run in the agent's own reasoning before a single image is generated. They are the difference between "consistent across 6 panels" and "best-quality possible". Bundle them all into ONE approval gate so the user sees the full plan and can redirect anything before money is spent. #### A1. Cast bible — multi-axis spec, not a one-liner The single most common cause of character drift is an underspecified cast description. The agent does NOT just pull one descriptive sentence from the prose. It generates a full **5-axis character bible** for each named figure plus any side-character that appears in 2+ panels: ```yaml characters: - name: Mira identity_anchor: "9-year-old red-braided lighthouse-keeper's granddaughter" # ← 5–8 words. Goes FIRST in every prompt mention of this character. # The model anchors stronger on the first ~10 tokens of an entity description. face: head_shape: "round, slightly heart-shaped" hair: "two thick rust-red braids tied with rough twine, framing freckled cheeks" eyes: "wide alert clear blue eyes" skin: "pale freckled, sun-pinked cheeks" mouth: "small, soft, contemplative" wardrobe: top: "thick blue wool fisherman's sweater" outer: "yellow oilcloth fisherman's raincoat, hood down" bottom: "dark grey wool trousers tucked into rubber boots" footwear: "dark rubber wellington boots, scuffed" signature: "small leather satchel slung across body" # optional change_policy: "SAME CLOTHING across all panels (no costume changes)" age_build: "~9 years old, slight, small for age" expression_default: "attentive, gentle, watchful" silhouette: "small upright figure, braids visible from any angle" relationship_note: "the protagonist; the eye through which the story is told" ``` Each axis exists because each is a separate failure mode the model can hit: - **identity_anchor** — first-token bias; opens every mention - **face axes** — locks the look across radically different angles - **wardrobe + change_policy** — the model loves to swap outfits; the explicit lock prevents it - **expression_default** — prevents random face mood drift - **silhouette** — gives the model the read-at-a-glance shape that survives heavy stylization - **relationship_note** — informs framing (foreground / background / two-shot) For each side character or named animal: shorter version (3 axes is fine), but still explicit. The wounded gull in the example has: `species: herring gull`, `markings: white body, grey wings, yellow beak with red dot`, `condition: left wing drooping, dishevelled`. The cast bible is shown to the user at the single approval gate. The user can override any axis ("change Mira to ~7 years old", "swap raincoat to a knitted scarf"). Then the agent freezes it. #### A2. Beat sheet — story → N panels with explicit dramatic roles The agent does NOT just pick N evenly-spaced moments. It writes a beat sheet that labels each panel's job in the dramatic arc. For a 6-panel adaptation: | Panel | Beat role | What it must do | |-------|-----------|----------------| | 1 | **SETUP** | Establish world, time, mood. Wide / environmental. Plant the protagonist. | | 2 | **INCITING** | The thing that starts the story. Often a discovery or arrival. Close on the trigger. | | 3 | **CARE / EARLY ACT** | The first response. Intimate, two-character if the story has them. | | 4 | **DEEPENING / RISING** | The bond / threat / problem deepens. Medium framing. | | 5 | **TURN** | The pivot toward resolution. Often outdoor / change of place / change of light. | | 6 | **PAYOFF** | The emotional release. The image the reader will remember. | For 4 panels: collapse 3+4. For 8 panels: split SETUP and RISING. For 12 panels: each beat gets 2 panels (a "wide" + a "detail"). Never more than 16 — beyond that, the per-panel attention budget for the user is too low. Beat-role tagging matters because it answers "what is this panel FOR?" — which constrains the camera, the framing, the lighting, and the emotional dial without any extra prompt engineering. #### A3. Shot grammar — forced variety across N panels AI image models love to repeat compositions. Without enforcement, N panels often come back as N variants of "medium two-shot, eye-level, soft light" — cinematic flatness. The agent enforces shot variety: For 6 panels: at least **1 wide / 2 medium / 2 close / 1 detail-or-environmental**. For 4 panels: 1 wide / 1 close / 2 medium with at least one establishing shot first. For 8+: maintain ~1/3 close, 1/3 medium, 1/3 wide ratio. | Shot | Use for | Camera prompt | |------|---------|---------------| | **Wide / establishing** | SETUP, TURN, PAYOFF | "wide establishing shot, full environment, characters small in frame, low angle" | | **Medium** | CARE, DEEPENING | "medium shot, waist-up, eye level" | | **Close** | INCITING, emotional beats | "close-up portrait, head-and-shoulders, soft natural light from frame-left" | | **Detail / environmental** | atmospheric beats | "detail shot, hands / object / texture, no faces in frame" | Mixing shot types is how a 6-panel story reads cinematic instead of like a yearbook. #### A4. Color script — palette evolves across the arc A locked single-style palette ≠ a static palette. The agent designs a 6-step **color script** that evolves with the emotional arc. For story-to-screen this usually traces COOL → WARM as the story moves from tension to resolution. For the lighthouse-bird example: | Panel | Dominant palette | Mood | Time of day | |-------|-----------------|------|-------------| | 1 | Cool: storm-blue + lantern-amber pinprick | foreboding | night | | 2 | Cool muted: low-tide grey + seaweed olive | uncertain discovery | overcast morning | | 3 | Warm interior: lantern-room amber + wool-cream | sheltered intimacy | midday interior | | 4 | Warm domestic: hearth amber + sweater-cream | bonding | indoor afternoon | | 5 | Cool-going-warm: pre-dawn purple-pink | anticipation | sunrise | | 6 | Full warm: golden sunrise + sky-pale-blue | release | full sunrise | The palette per panel is injected into the prompt's `[style brief]` clause as a one-line append: "muted storm-blue with single amber pinprick from the lantern". The style sentence stays constant; the palette accent shifts. This is what makes a 6-panel sequence feel like a film instead of 6 separate paintings. #### A5. Continuity state matrix — what's true in which panel A small table the agent maintains so prop / wardrobe / character-state continuity holds across panels. For the lighthouse-bird: | State flag | P1 | P2 | P3 | P4 | P5 | P6 | |-----------|----|----|----|----|----|----| | Mira in raincoat | ✓ | ✓ | – | – | ✓ | ✓ | | Mira indoors | ✓ | – | ✓ | ✓ | – | – | | Tobin present | ✓ | – | ✓ | ✓ | ✓ | ✓ | | Gull present | – | ✓ | ✓ | ✓ | ✓ | ✓ | | Gull wing visible-injury | – | ✓ | ✓ | – | – | – | | Gull splint | – | – | – | ✓ | ✓ | – | | Gull free / flying | – | – | – | – | – | ✓ | | Stormy weather | ✓ | – | – | – | – | – | Each `✓` becomes a continuity flag injected into that panel's prompt: "gull's left wing splinted with small wooden splint and white linen". Each `–` becomes a negative clause if forgetting it would cause drift: "no splint on the gull's wing (it has flown free)". #### A6. Single approval gate — show ALL of A1-A5 at once, get ONE "go" Cast bible + beat sheet + shot grammar + color script + continuity matrix are bundled into one approval gate. The user sees the full plan at a glance and can redirect anything. The agent does NOT ask 5 separate questions — that fragments attention. ONE gate, then fire. > **Why this matters more than render-time controls.** A bad cast spec produces 6 bad keyframes that take 30 seconds and $1.20 to re-fire. A bad beat sheet produces a story that doesn't read — and you don't realize it until you've already shipped. Pre-render gates catch story-level errors before they become render-level errors. Spend the 90 seconds of agent reasoning here; it pays for itself 10×. ### Phase B — Render-time controls (during generation) These run during the actual MCP calls and finishing pipeline: #### B1. Story-stage IP filter If the user pastes recognizably-copyrighted text (excerpts from named books / songs / scripts), the agent refuses and asks them to paraphrase. The example in this playbook uses an original story written for the playbook itself. #### B2. Character constant block, repeated verbatim Every panel prompt embeds the full character bible (A1) verbatim. Not "Mira (see above)" — the full multi-axis spec literally re-pasted. The identity_anchor goes FIRST in the character mention. AI image models do not have memory between calls; they need the full description every time, in the same order. #### B3. Style constant + palette evolution The same `style_brief` sentence appears verbatim in every panel prompt. The color script (A4) appends a one-line palette accent per panel. Key-light direction stays consistent unless the panel is explicitly outdoor / different time of day. #### B4. Deterministic prompt assembly template Every panel prompt is assembled in the **same field order**, every time. The model rewards predictability: ``` [STYLE BRIEF SENTENCE] ← A3-locked [PALETTE ACCENT for this panel] ← from A4 color script [CHARACTER 1: identity_anchor + face + wardrobe block] ← from A1, full verbatim [CHARACTER 2: identity_anchor + face + wardrobe block] ← only if in this panel per A5 [SETTING: location + time-of-day + key light direction] [ACTION: what's happening, what's on each face] [CAMERA: shot type + angle, per A3] [CONTINUITY FLAGS: positive + negative state, per A5] [NEGATIVE: no readable text, no banners, no signs] ``` The deterministic order is itself a quality control — it removes prompt-craft variance between scenes. #### B5. Mid-render contact-sheet review (cheap gate) After Wave 1 (keyframes) lands, the agent surfaces all N panels for the user to scan. Any off-style / off-character panel can be re-fired for $0.20 BEFORE the expensive animation wave fires. #### B6. Scene-by-scene narration matching For animated-video mode, the narrator script is broken so each ~5–13 seconds of voice-over aligns with one scene. The agent times the script naturally — no rushed reading, no awkward pauses at scene boundaries. #### B7. Webtoon caption discipline Caption boxes burned in via Pillow → ffmpeg overlay. Position: bottom of each panel for narration, top for dialogue. Latin serif (Cormorant Garamond) for narration boxes, sans (Inter) for dialogue. Heavy drop-shadow + semi-transparent dark band behind for legibility on any background. Captions are short enough to read in 2 seconds at thumbnail scale. #### B8. NO readable text in panels Keyframe prompts explicitly forbid signs / banners / titles / typography rendered IN the image — AI text rendering is the hardest known failure mode. All text appears at the Pillow / ffmpeg overlay stage, never in the gpt-image render. #### B9. Real animation per panel — i2v with gentle-drift prompts (default), Ken-Burns only as a fallback **Animated-video mode means actual frame-by-frame animation, not a slideshow of zooming stills.** The default for every panel is a real image-to-video pass through `seedance-i2v` (cinematic, up to 15s) or `ltx-q-i2v` (higher fidelity, LTX 2.3 Quality tier). Always pass `resolution: "auto"` — a literal `"1080p"` 422s on the ltx schema. Ken-Burns is a fallback for the rare case where a panel must be a perfect still — never the default. The art of i2v under cast-lock constraints is the **motion-budget per panel**. Every i2v prompt must explicitly cap what's allowed to move: - **Allowed**: subtle camera push-in / push-out, gentle hair sway, clothing breeze, eye blinks, slight head-tilt, breathing-shoulder, water lapping, distant clouds drifting, candle / lantern flicker, soft wind through grass, gull wing folding-unfolding once at most. - **Banned per prompt**: large body motion, big head turns, walking, running, camera pan (zoom-in is fine, panning across the scene swaps faces), aggressive expression changes (smile → frown), costume changes, props appearing/disappearing. Concrete prompt skeleton, repeated verbatim across panels except for the per-panel verb: > "Very gentle motion. Slow subtle camera push-in across the scene. [PANEL-SPECIFIC drift verb: 'soft wind through Mira's braids only / gull's eye blinks once / lantern flame flickers softly / waves rise and fall in the background only / her hand lifts the gull skyward smoothly']. Characters' faces and bodies stay still. No walking, no head turns, no costume changes. Painterly cinematic quality, Studio Ghibli motion language. 10 seconds." The hardest failure mode is i2v drift on a named character's face. The mitigations: - **Cast-lock prompt clause** in the i2v prompt itself (re-paste the character anchor: "Mira: 9-year-old with rust-red braids and yellow raincoat, face holds steady"). i2v models read text prompts too. - **One-motion-only rule**: pick the SINGLE element that moves per panel; lock everything else. AI video models drift when asked to animate multiple things; they hold form when asked for one. - **Aspect-native source**: panel was generated at the same aspect as the i2v output (e.g. 9:16 portrait keyframe → 9:16 portrait i2v). Never i2v a 4:3 source into a 9:16 target — the model has to re-frame, which doubles drift risk and stretches geometry. Use Ken-Burns only when: a panel has 3+ distinct named characters in frame (i2v will swap two of them) OR the panel is the climactic payoff that must NOT drift (rare). In those cases, a very slow zoom over the locked still gives the same "moving" sensation without the face-drift risk. ### Phase C — Post-render controls (before ship) #### C1. Auto-drift sanity check After Wave 1 keyframes land, the agent itself reviews the contact sheet for character drift BEFORE showing the user. Quick heuristic: every panel containing a named character should match the identity_anchor at-a-glance (red braids, yellow raincoat, freckles). Flag any panel where the character "feels like a different person" and queue a re-fire candidate. #### C2. Climax verification The agent verifies the final panel actually IS the payoff (matches the closing narration line / final beat in A2). If panel 6 shows two characters indoors when the narration is "the gull flew into the morning sun", something is misaligned — re-fire panel 6. #### C3. Finishing pass — score covers the FULL reel (animated mode) The narrator track is concat'd scene-by-scene (`ffmpeg-concat`). The instrumental score (one `create_media action:"music"` call with `lyrics_prompt:"[Instrumental]"`) is laid under the entire reel via `create_media({ model_override: "ffmpeg-mux", audio_url: score, video_url: reel, audio_fill: "loop" })` — `audio_fill:"loop"` repeats a short bed across the full duration so the last third is never silent. Final `ffmpeg-export` to the chosen `aspect`. A reel whose music cuts out partway is a draft, not a deliverable. #### C4. Final reel review gate (animated mode) Before declaring done, the agent surfaces the final reel to the user for a "ship it / one more pass" gate. One last cheap stop before the artifact is considered final. > **Why this playbook works for any source story.** A short story already comes with everything we need: a small cast, a set of locations, a beat-by-beat arc, an emotional payoff. The agent's job is just to LISTEN to that structure (extract a multi-axis cast bible, identify visual beats with dramatic roles, design a color script, build a continuity matrix) and then ENFORCE it across N generations (verbatim character block, identical style sentence with per-panel palette accent, deterministic prompt assembly). The hard problem (writing a good story) is on the user. The mechanical problem (rendering it consistently AND cinematically) is on the playbook — and the pre-render gates in Phase A are where most of the quality is won. ## Output structure ``` public/chapters// panel-01.png ... panel-NN.png ← N keyframes (9:16 vertical) webtoon-scroll.png ← vertical-stacked panels with caption boxes panel-01.mp4 ... panel-NN.mp4 ← per-scene animations (animated-video mode) voiceover.mp3 ← narrator voice-over (animated-video mode) music.mp3 ← unified score (animated-video mode) -reel.mp4 ← final animated video reel public/chapters/-webtoon-example.html ← webtoon showcase public/chapters/-video-example.html ← animated video showcase ``` The example for this playbook is **The Lighthouse Bird** — an original short story written for the playbook. Cast: Mira (a young girl with curly red braids) and her grandfather Tobin. Six panels covering the arc of finding, healing, and releasing a wounded seagull. Ships in BOTH modes: webtoon scroll AND animated reel with a warm narrator voice-over. ## Execution mode (autopilot / director / auto) This is a multi-scene project, so pass a `mode` to `generate_project` to control the **pre-finish gate** (after every scene renders, before delivery): `autopilot` (certain brief → ship in one call, no pause), `director` (always pause for a human pick/steer before delivery), or `auto` (recommended — ship UNLESS the auto-critic is unsure: score < `confidence_threshold` (default 0.8), verdict `iterate`, or a flagged shot). Setting `mode` runs the project async (returns a `job_id`); resume a director/auto pause with `review_checkpoint` + `resume_from_checkpoint` (decision `continue` to deliver / `abort` to discard). Omit `mode` for legacy straight-through behavior.