--- title: "Talking Avatar — document in, talking-avatar video + audio out" tier: hero format: avatar-video theme: read-aloud persona: educator, marketer, communicator, accessibility, creator duration: matches source document, typically 30s–5min budget_usd: $0.50–$3 per minute of output caps: ["gemini-text", "krea-2-large", "gpt-image", "gemini-tts", "chatterbox-tts", "inworld-tts", "lipsync", "ffmpeg-audio-mix"] skills: ["talking-avatar"] showcases: - title: "Steve Jobs · Stanford Commencement excerpt (English)" href: "/chapters/talking-avatar-jobs-example.html" - title: "朱自清 · 《匆匆》 (Chinese)" href: "/chapters/talking-avatar-zhuziqing-example.html" reliability: 3.2 # 5 − portrait image gen .3 − tts .2 − i2v motion .6 − lipsync video .6 − 3 stages over 5 (.6) + .5 two verified E2E showcases (2026-06-04); chunk+concat multiplies the video stages on long reads — per-chunk risk is independent --- # Talking Avatar — document in, talking-avatar video + audio out Paste a document URL or a block of text. Pick a target language. Get back **two artifacts**: a talking-avatar video where a generated character reads the document in the language you chose, and a clean audio-only track of the same reading. The agent doesn't just read the doc verbatim — it first reviews the text, optimizes it for speaking (break punctuation, emphasis cues, pacing marks), picks an appropriate voice + character archetype for the topic, and only then renders. ## What you'll get | Artifact | Format | Use | |---|---|---| | **Talking-avatar video** | 1024×1024 MP4 (or 9:16 / 16:9), 25 fps, AAC audio | embed on landing pages, social posts, internal explainers, audio-described accessibility versions | | **Audio-only track** | MP3, 128 kbps stereo | podcast, audiobook chapter, e-learning narration, lower-bandwidth share | Both artifacts are produced in one pipeline. No separate audio re-render needed; the audio in the video is bit-identical to the standalone MP3. ## What "any language" actually means The avatar can speak any language `gemini-tts` (the default TTS) supports — currently 30+ including: - **English** (en-US, en-GB, en-AU) · **Spanish** (es-ES, es-419) · **French** (fr-FR) · **German** (de-DE) · **Italian** (it-IT) · **Portuguese** (pt-BR, pt-PT) - **Mandarin Chinese** (cmn-CN) · **Japanese** (ja-JP) · **Korean** (ko-KR) · **Cantonese** (yue-HK) - **Arabic** (ar) · **Hindi** (hi-IN) · **Russian** (ru-RU) · **Polish** (pl-PL) · **Dutch** (nl-NL) · **Turkish** (tr-TR) If the source doc is in a different language than the target output, the agent translates it with `gemini-text` first (register-locked — see "Speak-out optimization" below). ## YAML brief schema ```yaml project: name: "Stanford commencement read-aloud" source: # Pick ONE text: | When I was 17, I read a quote that went something like… url: "https://example.com/some-article" file: "manifesto.md" # local file source_language: "English" # for reference; agent auto-detects target_language: "English" # what the avatar speaks # set to anything else for live translation # Character / voice (optional — agent picks if omitted) voice_archetype: "warm tech-keynote speaker" # free-form description voice_gender: "male" # male | female | neutral voice_age: "50" # the voice character's apparent age voice_pace: "measured" # measured | conversational | energetic | reverent multi_speaker: false # if true, the agent identifies multiple roles # from the source and assigns voices per speaker # Avatar appearance (optional — agent picks if omitted) avatar_archetype: "salt-and-pepper hair, dark mock-turtleneck, Rembrandt lighting" avatar_setting: "plain warm-grey backdrop, indoor portrait, single subject" # Output prefs output_aspect: "1:1" # 1:1 (default) | 9:16 (TikTok / Reels) | 16:9 (cinema) audio_only_also: true # produce the MP3 alongside the MP4 project_slug: "talking-avatar-jobs" ``` ## How the agent should run this (interaction contract) 1. **CONFIRM (one message, ≤1 question):** restate in 1 line ("talking avatar reading your ~N-word doc in , MP4 + MP3, ~$0.83 per 5s chunk ≈ $X total, ~10–20 min") + real choices: gemini-tts (default, 30+ languages) vs chatterbox voice-clone (needs reference audio) vs inworld (emotion controls) · aspect 1:1 / 9:16 / 16:9. The one question worth asking: voice gender/archetype if the doc doesn't imply one — it MUST match the portrait. 2. **PREVIEW CHECKPOINT:** the speak-out-optimized script + the portrait + a one-sentence TTS sample (~$0.05 total) BEFORE any kling motion + lipsync chunks fire — those dominate the bill (~$0.57/chunk × N chunks). 3. **NARRATE:** per chunk ("chunk 3/12: kling motion ~2 min, lipsync ~3 min, job …"); poll `get_create_media` every 10–15s, never silent >2 min — a 1-min read is ~12 chunks, say the total up front. 4. **FAIL GRACEFULLY:** the `talking-head` BYOC cap has a known param-shape bug — NEVER route to it; the working path IS animate-then-lipsync (`kling-v3-i2v` → `lipsync`). Lipsync face-detect fails on cartoon portraits → regenerate a photoreal-human portrait; kling hang → retry once → `ltx-q-i2v` (resolution:"auto", `num_frames:121` for 5s) as the motion source. ≤2 retries per chunk; if video dies, ship the audio-only MP3 — it's a first-class deliverable. 5. **DELIVER:** MP4 + MP3 + one honest line ("lipsync is ±50ms, not phonetician-grade; the avatar is static below the neck — `heygen-twin` is the fidelity upgrade") + ONE next step (same chassis, another `target_language`). ## The 8-stage pipeline (Stage 6 — ANIMATE — is MANDATORY) ⚠ **A static portrait + lipsync produces a paper-puppet effect.** The avatar's lips move but the rest of the face is frozen — visibly artificial within 1 second of watching. Stage 6 (kling-v3-i2v motion video) is **required**, not optional. It adds subtle blinks, breathing, weight shifts that make the avatar read as alive. Also: **voice must match avatar gender.** A female-coded TTS voice (Despina, Achernar) on a male portrait, or vice versa, fails immediately. The agent picks voice + portrait together in Stage 3, before either is generated. ``` Stage 1 — REVIEW THE DOC (LLM) gemini-text reads the source text. Returns a structured analysis: {topic, intent, tone, length_in_words, suggested_voice_archetype, suggested_pace, paragraph_breaks, emphasized_phrases, proper_nouns_to_lock} → Cost: ~$0.002 per 500 words Stage 2 — OPTIMIZE FOR SPEAKING (LLM) gemini-text rewrites the doc into "speak-out" form: - Add comma pauses where the reader needs to breathe - Insert ellipses where the speaker should slow down - Mark proper nouns / acronyms that need careful pronunciation - If target_language differs from source_language, translate with register lock (preserve sentence count to match natural pacing) - Strip non-readable artifacts (markdown, table syntax, URLs that would be read out as "h-t-t-p-s-colon-slash-slash…") → Cost: ~$0.005 per 500 words Stage 3 — PICK CHARACTER + VOICE Default: gemini-tts with a voice that matches archetype. Single-speaker: pick ONE voice (Charon, Despina, Gacrux, etc.) Multi-speaker: identify dialogue blocks, assign voices per role. Voice options: - gemini-tts — 30+ languages, fastest, $0.001/sec - inworld-tts — character voices with emotion controls - chatterbox-tts — voice-clone if you supply reference audio - grok-tts — xAI's TTS (mature, emotion-aware) → Cost depends on cap; gemini-tts is the default. Stage 4 — GENERATE AVATAR IMAGE krea-2-large (default) or gpt-image renders the portrait from voice_archetype + avatar_archetype. 1024×1024 square portrait, mouth-closed, camera-direct, soft lighting. → Cost: ~$0.063 (krea-2-large) or $0.04 (gpt-image) Stage 5 — SYNTHESIZE THE SPOKEN AUDIO create_media(action:"tts", model_override:"gemini-tts", prompt:, voice:) Duration: ~0.4–0.5 seconds per word in English; ~0.15–0.2 sec per character in Mandarin. → Cost: ~$0.05 per minute of output Stage 6 — ANIMATE THE PORTRAIT (MANDATORY) create_media(action:"animate", model_override:"kling-v3-i2v", source_url:, resolution:"auto", prompt:"subtle natural motion: gentle head sway, occasional blink, slight shoulder shift, locked face identity") → 5-second motion-source.mp4 — adds blinks, breath, weight shift → Cost: ~$0.53 per 5s of motion source Stage 7 — LIP-SYNC + PACKAGE 7a · Match audio duration to motion-source duration (5s default). Chunk longer audio at sentence boundaries (see "Longer reads" below). 7b · Push motion-source.mp4 + audio chunk to a publicly fetchable URL via create_media(action:"publish", source_url:) → permanent CDN URL. 7c · create_media(action:"lipsync", model_override:"lipsync", source_url:, audio_url:) — regenerates lip movement on the MOTION video (NOT a static portrait). Output: talking-avatar.mp4 with real motion + accurate lips. → Cost: ~$0.04 per 5s of output Stage 8 — DELIVER TWO ARTIFACTS → talking-avatar.mp4 (5s default; chunk + concat for longer) → audio-only.mp3: keep the Stage-5 gemini-tts MP3 (bit-identical to the video's audio track). It's the standalone podcast / audiobook deliverable. ``` Total typical cost per 5-second clip: **~$0.83** (gpt-image $0.04 + gemini-tts $0.07 + kling-v3-i2v $0.53 + sync-lipsync $0.04 + local $0.00). For longer reads, multiply the motion + lipsync line items by the number of 5-second chunks. ## Longer reads — chunk + concat pattern For 1 minute of talking avatar: - gemini-tts the full text (~1 min audio) - ffmpeg-split the audio at sentence boundaries into ~5s chunks (12 chunks) - For each chunk: kling-v3-i2v motion-source + lipsync (~$0.57/chunk = ~$6.84 total) - ffmpeg-concat the 12 talking-avatar clips with hard cuts (or 0.2s crossfades) - Extract one combined audio track For 1 minute total: **~$7**. For 30 seconds: **~$3.50**. ## Speak-out optimization — what makes the difference Random text read by TTS sounds like a robot reading. The Stage-2 optimization is what makes the difference between "text-to-speech" and "actually engaging narration." What the agent does: - **Insert breath pauses** — every 15-20 syllables, add a comma or ellipsis where a human reader would breathe - **Lengthen end-of-thought** — em-dash before the final clause of a paragraph; the TTS lengthens the pause - **Mark proper nouns** — wrap "Steve Jobs" in `` or repeat with phonetic hints - **De-acronymize** — "iPhone" stays; "API" becomes "A-P-I"; "MIT" becomes "M-I-T" - **Strip non-readable** — markdown headers (`##`), table syntax (`|`), code blocks, raw URLs - **Match cadence to register** — keynote speech: long meaningful pauses; technical explainer: faster + clipped; bedtime story: very slow + soft The LLM prompt template: ``` Rewrite the following text for read-aloud TTS in {{target_language}}. Register: {{register}} # warm-keynote | technical-explainer | bedtime-story | sales-energetic | poetic-reverent Audience: {{audience}} Preserve these proper nouns exactly: {{proper_noun_lock}} Pace: {{pace}} # measured | conversational | energetic | reverent Rules: - Add commas / ellipses where the reader needs to breathe (every ~15 syllables). - Use em-dashes before final clauses for emphasis. - Don't paraphrase the meaning. Re-punctuate only. - Strip markdown, code, table syntax. - De-acronymize anything 3+ letters long. Source text: {{source_text}} ``` Pass the result of this LLM call to gemini-tts. Same text, much better cadence. ## Multi-speaker mode If the source doc is a dialogue, transcript, or interview, set `multi_speaker: true`. The agent then: 1. **Stage 1.5** — identifies speaker roles in the source (Speaker A, Speaker B, NARRATOR). 2. **Stage 3** — assigns a different voice to each role. 3. **Stage 5** — renders each speaker's lines separately, then concatenates with `ffmpeg-concat` (sequential, no overlap). 4. **Stage 6** — generates ONE avatar per role (or one avatar with multiple sub-clips, depending on `avatar_archetype` config). For dialogue showcases, the simplest path is N portraits cut together as a back-and-forth. The showcase pair ships single-speaker; multi-speaker is documented but more variable. ## Source video / portrait requirements - Portrait must be a **real human-looking face**. Cartoon / animated portraits fail lipsync's face-detection. - Front-facing or 3/4 view; no occlusion. - Mouth closed in the source portrait (lipsync regenerates lips). - 1024×1024 or larger; smaller portraits get upscaled. - Single face per portrait. Lipsync v3 (`fal-ai/sync-lipsync/v3`) accepts video_url + audio_url. We build the video_url by looping the still portrait for the audio duration. ## Cost (worked example — Jobs ~80 sec output) | Stage | Cap | Cost | |---|---|---| | Review + optimize | gemini-text × 2 calls | ~$0.01 | | Avatar image | gpt-image (1024×1024) | ~$0.04 | | Spoken audio (80 sec) | gemini-tts | ~$0.07 | | Static MP4 build | ffmpeg (local) | $0.00 | | Lipsync (80 sec) | lipsync (sync-lipsync/v3) | ~$0.65 | | **Total** | | **~$0.77** | For longer docs the lipsync cost dominates linearly. A 5-minute read-aloud is ~$2.50. ## QC checklist - [ ] **Voice match** — does the voice fit the doc's register? (Sales energy on a poetic essay is jarring.) - [ ] **Pacing** — does the reader breathe at natural break points? Listen with eyes closed. - [ ] **Proper noun pronunciation** — names, places, technical terms rendered correctly? - [ ] **Lip-sync quality** — open the MP4, mute audio, watch mouth. Movements should track audible syllables ±50ms. - [ ] **Audio-only track** — does it stand on its own without the video? (Bitrate, clipping, dropouts.) - [ ] **Duration parity** — audio length should be within 5% of expected (300 words ≈ 90-110 sec in English). - [ ] **Translation register** (if target ≠ source) — read the translated text aloud; does it sound like something the source author would have actually said? ## Cross-surface parity | Surface | How to invoke | |---|---| | **MCP (agent)** | run the 8 stages as `create_media` calls (review/optimize → image → tts → animate → lipsync); paste this file into Claude and it orchestrates them | | **CLI** | `livepeer playbook talking-avatar --doc --lang Mandarin --voice "warm-female-narrator"` | | **Webapp** | paste text in chat → preprocessor detects long-form prose → shows "Make a talking-avatar?" suggestion → runs the playbook | | **Webapp (file drop)** | drop a `.md` / `.txt` / `.pdf` on the canvas → context menu → "Talking avatar" | ## Real-use scenarios | Persona | What they need | What this playbook gives | |---|---|---| | **Educator** | A 90-second read-aloud of a difficult passage in 3 languages | Same chassis, swap `target_language` 3×. Audio-only MP3s also play in Anki / podcast clients. | | **Founder** | A talking-avatar pitch for an investor deck | Single-speaker, warm tech archetype, ~2 min. Audio-only goes in the email body; video on the deck cover slide. | | **Accessibility comms** | Audio-described version of a written announcement | `audio_only_also: true` — ship MP3 as the screen-reader-friendly artifact. | | **L&D / training** | Microlearning module on a policy update | Optimize the policy text for speak-out, pick a measured female voice, render. Replaces the $200/module voice-actor recording. | | **Indie author** | Audiobook preview chapter from a manuscript | 5-minute reading. The MP3 is the deliverable; the MP4 is the social-share teaser. | | **Localization team** | Internal announcement in 8 languages | Run the playbook 8× with different `target_language`. All voices remain "warm keynote" archetype. ~$20 for 8 languages × 60s. | ## Honest limits - **Lip-sync isn't lip-perfect.** Sync-lipsync v3 is excellent for ±50ms accuracy but won't fool a phonetician. For client-final video, run `heygen-twin` (which trains a digital twin on 15s of real footage) — much higher fidelity, requires a training step. - **Avatar is static below the neck.** Gesture is implied via subtle blinks + head sway from lipsync, but the avatar doesn't gesticulate with hands. For full-body gesture, you'd train a digital twin (heygen-twin) or composite with `omnihuman/v1.5` (which needs a video input + has its own quality tradeoffs). - **Emotion is in the voice, not the face.** The avatar's facial expression stays close to the portrait's baseline expression. If you want big emotional range, generate multiple portraits (one per emotion) and concatenate. - **Single-speaker is easier than multi.** Multi-speaker mode works but the back-and-forth cuts feel a little stiff. For genuinely conversational dialogue, this isn't yet the right tool — train a heygen-twin per speaker. - **Long docs need chunking.** Lipsync v3 has a soft duration limit (~5-10 min per call). For longer reads, chunk the audio, lipsync each chunk, concatenate. - **Music / SFX not included.** The audio track is voice-only. Add background bed via `ffmpeg-audio-mix` if needed. - **Cartoon avatars fail lipsync.** Real human faces only. For stylized avatars, use a real-face base and apply style transfer afterward — adds steps + cost. ## Two reference showcases 1. **`/chapters/talking-avatar-jobs-example.html`** — Steve Jobs Stanford Commencement excerpt (~350 words). Generic warm-tech-keynote archetype avatar. English. Shows the speak-out optimization, both artifacts, full pipeline trace. 2. **`/chapters/talking-avatar-zhuziqing-example.html`** — 朱自清《匆匆》("Hurry") excerpt (~350 chars). Generic 1920s Republic-of-China scholar archetype. Mandarin. Same pipeline, swapped language + voice + character. Both ship the talking-avatar MP4 + the audio-only MP3 + the speak-out-optimized text + cost breakdown. ## Reproduce — sample MCP call ```typescript // Or just paste this whole file into Claude with your doc and ask // the agent to follow the 8-stage pipeline. // Stage 1+2 — review + optimize text const optimized = await create_media({ action: "generate", model_override: "gemini-text", prompt: SPEAK_OUT_PROMPT.replace("{{source_text}}", sourceText) .replace("{{target_language}}", "English") .replace("{{register}}", "warm-keynote"), }); // Stage 4 — avatar image const portrait = await create_media({ action: "generate", model_override: "krea-2-large", prompt: `${voice_archetype} portrait, mouth closed, soft Rembrandt lighting…`, }); // Stage 5 — TTS (this MP3 is also your second deliverable) const audio = await create_media({ action: "tts", model_override: "gemini-tts", prompt: optimized.text, voice: "Charon", // pick by archetype; must match portrait gender }); // Stage 6 — ANIMATE the portrait (MANDATORY — never lipsync a static still) const motion = await create_media({ action: "animate", model_override: "kling-v3-i2v", source_url: portrait.url, resolution: "auto", prompt: "subtle natural motion: gentle head sway, occasional blink, locked face identity", }); // Stage 7 — lipsync the MOTION video to the audio const talking = await create_media({ action: "lipsync", model_override: "lipsync", source_url: motion.url, // the animated clip, NOT a static portrait audio_url: audio.url, }); // Stage 8 — audio.url (the Stage-5 MP3) is your second artifact, bit-identical to the video audio ``` ## Verified end-to-end (2026-06-04) - ~350-word English source (Steve Jobs Stanford excerpt) → 81-second avatar video + MP3 - ~350-char Chinese source (朱自清《匆匆》excerpt) → 84-second avatar video + MP3 - Total cost ~$0.77 per showcase - Lipsync wall-clock ~3 min per call ## Trace Built 2026-06-04 on `feat/talking-avatar-playbook`. Capabilities used: gpt-image (portrait), gemini-tts (voice), lipsync (sync-lipsync/v3, fal-ai). Talking-head BYOC cap is currently broken on a separate param-shape issue (see `docs/bugs/2026-06-03-talking-head-omnihuman-v15.md`); this playbook deliberately uses `lipsync` instead, which is registered and working.