--- title: "Video Translate — paste a video URL, get N languages back" tier: hero format: multilingual theme: dub persona: marketing, creator, education duration: same as source (typically 30s–10min) budget_usd: ~$0.80/min per target language caps: ["wizper", "nemotron-asr", "gemini-text", "gemini-tts", "veed-lipsync-v2", "lipsync", "ffmpeg-mux"] skills: ["video-translation"] showcases: - title: "Camille (French) → English + Spanish + Chinese" href: "/chapters/camille-multilingual-example.html" - title: "David (English) → French + Chinese" href: "/chapters/david-multilingual-example.html" reliability: 4.3 # 5 − ASR .2 − tts .2 − lipsync video .6 − 7-stage path .2 + .5 two proven E2E showcases (Camille / David); translate + ffmpeg finishing deterministic; per-language passes are independent (one language failing doesn't sink the rest) --- # Video Translate — paste a URL, get N languages back **Paste a video URL** (YouTube, Vimeo, an mp4 you uploaded, a raw GitHub link). **Pick target languages.** Get back N MP4s where the same speaker speaks each language — lip-synced, with the original pacing preserved. The pipeline is four real caps, all live on BYOC: **transcribe → translate → speak → lip-sync**. One pass per language. The lip-sync step defaults to **`veed-lipsync-v2`** (Veed Lip Sync 2.0 — swap audio on a video, match lips/emotion/timing, up to 4K/10 min); `lipsync` (sync-lipsync/v3) stays as the cheaper fallback. ## YAML brief schema ```yaml project: name: "Camille talk — multilingual launch" source: # Pick ONE youtube_url: "https://www.youtube.com/watch?v=abc123" # OR an mp4 URL directly (must return 200 OK to curl -I) video_url: "https://raw.githubusercontent.com/livepeer/storyboard/main/public/chapters/multilingual-demo/camille-source-fr.mp4" source_language: "French" # for reference; wizper auto-detects targets: ["English", "Spanish", "Chinese"] voice_clone: true # clone the speaker's own voice via chatterbox-tts (≥10s clean audio) # false → use a gemini-tts voice matched to the speaker register: "conversational" # conversational | formal | technical | poetic — locks translation tone ``` ## The pipeline (5 minutes start to ship for a 30s source) Run **Stages 3–6 in parallel across targets** — each language is independent. ``` Stage 1 — FETCH (if YouTube) yt-dlp --format "mp4[height<=720]" -o source.mp4 # Optional clip trim: create_media(action:"animate", model_override:"ffmpeg-trim", source_url: source.mp4, params:{ ss: 5, to: 35 }) Stage 2 — HOST (so fal can fetch) Push to public/chapters//clip.mp4 (raw GitHub URL is live in <1s), or create_media(action:"publish", source_url: ) → permanent CDN URL. YouTube URLs block fal's UA → always re-host. Stage 3 — TRANSCRIBE (per source, once) wizper (Whisper-v3) → timestamped transcript + word timings. Non-English / streaming → nemotron-asr (40 locales, native punctuation). create_media(action:"animate", model_override:"wizper", source_url:) → returns transcript text + segment timestamps. Stage 4 — TRANSLATE WITH REGISTER LOCK (per target) gemini-text rewrites the transcript into the target language, register-locked, matching the SOURCE sentence count + rhythm so the audio stays in sync. create_media(action:"generate", model_override:"gemini-text", prompt: TRANSLATE_PROMPT(target, register, proper_nouns)) Lock proper nouns explicitly so "Mémé" doesn't become "Granny". Stage 5 — SPEAK (per target) voice_clone:true → voice_create from ≥10s of the source audio, then chatterbox-tts with that voice_id → the speaker's OWN voice in the new language. voice_clone:false → gemini-tts with a voice matched to the speaker's gender/age. create_media(action:"tts", model_override:"chatterbox-tts" | "gemini-tts", prompt: , audio_url:) Stage 6 — LIP-SYNC + PACKAGE (per target) veed-lipsync-v2 (Veed Lip Sync 2.0) swaps the audio on the source video and matches lips + emotion + style + timing — purpose-built for dubbing, up to 4K and 10 min: create_media(action:"lipsync", source_url:, audio_url:) → action="lipsync" on a VIDEO source AUTO-ROUTES to veed-lipsync-v2 (no model_override needed). Pass model_override:"lipsync" (sync-lipsync/v3) for the cheaper budget path, or model_override:"veed-lipsync-v2" to force it. → one MP4 per target. Audio is already exact-length (TTS of the translated line), so lips track the full clip — no silent tail. Stage 7 — OPTIONAL FINISH ffmpeg-grid: stitch 4 outputs as a 2x2 comparison reel. ffmpeg-burn-subtitles: bilingual captions (target + source-as-subs). ``` ## Critical: register-locked translation beats raw machine translation The whole quality lever is **Stage 4**. A literal translation drifts in tone and length, blowing the lip-sync. The `gemini-text` prompt MUST: 1. **Match sentence count + rhythm** to the source — verbose translations stretch the audio. 2. **Lock proper nouns** — `Preserve these names exactly: ["Camille", "Mémé"]`. 3. **Hold register** — keynote/poetic/technical/conversational, not generic colloquial. ``` Translate the following transcript into {{target_language}}. Register: {{register}}. Audience: general. Preserve these proper nouns exactly: {{proper_noun_lock}}. Match the original sentence COUNT and rhythm — do not add or merge sentences. Output only the translation, no notes. Source transcript: {{transcript}} ``` ## Source video requirements - **Single speaker, front-facing.** Multi-speaker clips: split with `ffmpeg-trim` first. - **Mouth visible the entire clip** (no occlusion). lipsync v3 hates side angles. - **30s ≤ duration ≤ 10min** per call. Longer: split + stitch. - **Constant fps** (24/25/30). Frame-rate jumps confuse lip-sync. - **16 kHz+ audio, SNR > 20 dB.** Background music: separate with `mirelo-sfx-v2v` first, then mix back after. - **Voice clone needs ≥10s** of clean source audio — under that, fall back to `gemini-tts`. ## Cost (per 30s source × 3 languages) | Stage | Cap | Per language (30s) | |---|---|---| | Transcribe (once) | wizper | ~$0.01 | | Translate | gemini-text | ~$0.01 | | Speak | chatterbox-tts / gemini-tts | ~$0.03 | | Lip-sync | veed-lipsync-v2 ($0.07/s) — default | ~$2.10 | | Lip-sync (budget) | lipsync / sync-v3 | ~$0.30 | | **Per language (Veed)** | | **~$2.15** | | **3 languages (Veed)** | | **~$6.45** (budget path ~$1.10) | yt-dlp + hosting is free. The whole stack is BYOC caps — no per-minute HeyGen tariff. ## Webapp / CLI / MCP parity | Surface | How to invoke | |---|---| | **MCP (agent)** | run Stages 3–6 as `create_media` calls per target (see pipeline above) | | **CLI** | `livepeer translate --to English,Spanish,Chinese --voice-clone` | | **Webapp** | right-click any video card → "Translate to…" → multi-select languages → renders N MP4s as a row | | **YouTube paste** | paste a YouTube URL in chat → preprocessor runs `yt-dlp` + re-hosts via `create_media(action:"publish")`, then routes the chain | ## The 2 reference showcases 1. **`/chapters/camille-multilingual-example.html`** — French → English + Spanish + Chinese. Three targets, single speaker, side-by-side player. 2. **`/chapters/david-multilingual-example.html`** — English → French + Chinese. Same chassis. Both ship the pipeline trace + cost breakdown. ## Honest limits - **YouTube ToS.** Downloading copyrighted YouTube videos is a license question. Use only content you own or CC-BY / CC0 / public-domain material. Rights clearance is on you. - **Lip-sync quality vs cost.** `veed-lipsync-v2` (default) handles emotion, style, timing and longer/higher-res clips (up to 4K / 10 min) and is more forgiving of angles; `lipsync` (sync-v3) is ~7× cheaper but hates side angles. Pick per budget. - **Voice clone needs ≥10s clean audio.** Under that, `gemini-tts` produces a cleaner (but generic) voice. - **Music + voice on one track gets murky.** Separate stems with `mirelo-sfx-v2v` first, translate the voice, mix the music back via `ffmpeg-mux`. - **Animated source (cartoon faces) fails lipsync.** For animation, ship audio-only: `wizper → gemini-text → chatterbox-tts → ffmpeg-mux` (no lip-sync step). - **Long docs need chunking.** lipsync v3 has a soft ~5–10 min limit; chunk the audio, lipsync each chunk, `ffmpeg-concat`. ## Trace The pipeline is four live BYOC caps — `wizper` (transcribe), `gemini-text` (register-locked translate), `chatterbox-tts`/`gemini-tts` (speak), `veed-lipsync-v2` / `veed/lipsync/v2` (lip-sync — default; `lipsync` / `fal-ai/sync-lipsync/v3` as the budget fallback) — plus `ffmpeg-*` finishing. No external translation service required; works directly off any video URL.