--- reliability: 3.7 # wizper .2 + moodboard image batch .3 + i2v batch .6 + 6 happy-path stages −.2; ffmpeg finishing deterministic; brand-kit refs + key art OPTIONAL; no proven E2E showcase yet --- # Suno song to music video — agent playbook > Paste this whole file (with the BRIEF filled in) into Claude cowork, chat, or Code. The agent walks through it with you and stops for your approval at every step that matters. ## What you'll get A complete music video package built around the song you already wrote. The scene structure is locked to the song's actual structure — intro / verse / chorus / bridge — not a vague mood mapping. - **Section-mapped scene plan** — the agent listens to the song (transcript + duration via `wizper`) and proposes 8-12 scenes synced to its intro / verse / chorus / bridge sections. - **Brand kit** — palette + reference imagery + voice keywords derived from `song_mood` and `lyric_themes`. Stays consistent across every visual. - **Scene moodboard** — 8-12 still frames at premium quality, one per song section. The visual storyboard. - **Animated scenes** — each still becomes a 3-5s i2v clip. The motion matches the energy of its section (slow for verses, dynamic for choruses). - **Full music video MP4** — your audio + the animated scenes, stitched and color-graded, in 16:9. - **9:16 short version** — the strongest 30-45 seconds re-cropped vertical for Reels / TikTok / Shorts. - **Key art** — 1920×1080 album / single cover art for streaming platforms. **Time**: ~35 minutes wall clock. **Your attention**: about 8 minutes across 7 approvals. **Cost**: $2.50 – $5.00 end to end. ## Tell the agent about the song ```yaml song_url: # the Suno mp3/wav — upload via /upload first, or paste the public Suno share URL song_title: # the song's title (used in key art + filenames) artist_name: # your artist name (key art credit) song_mood: # 3-5 words, e.g. "wistful, anthemic, late-summer, nostalgic, hopeful" lyric_themes: # 3-5 themes pulled from the lyrics, e.g. "first car, hometown highway, leaving, growing up, calling home" visual_style: # e.g. "16mm film grain, golden hour, candid documentary" · "neon synthwave, rain, urban night" · "cel-shaded anime, pastel" character_or_artist: # OPTIONAL — URL of a photo of you / a featured character. If present, the agent uses it as i2v keyframe anchor. aspect_ratio: # 16:9 (default) or 9:16 (vertical-native) song_slug: # kebab-case for filenames, e.g. hometown-highway ``` ## Quality controls (new — auto-applied) This playbook leverages the closed quality loop. The agent applies these by default; you don't have to ask: - **Cost upfront.** Before any multi-step batch fires, the agent calls `submit_plan` with `confirm:false` and shows you the proposed steps + total cost. Nothing runs until you approve. - **Auto-retry on weak shots.** The server default is `quality_threshold: 0.7`. For the moodboard frames the playbook step tells the agent to pass `quality_threshold: 0.85` so a low-confidence first try gets one auto-retry before you see it. - **Critic verdict on the result.** When a multi-scene step finishes, the agent reads the `critique` field on the response (verdict: ship / iterate, plus issues / missing items) and surfaces it for your review. - **One-click recovery.** If a step fails, `get_plan` returns a `recovery` hint pointing at `replan({plan_id})` — the agent classifies the failure and proposes a recovery plan you can approve. - **End-of-project scorecard.** The final wrap-up calls `get_scorecard({project_id})` with succeeded count, mean quality score, critic verdict, total cost, and wall time. You can override any of these per step: pass `quality_gate: false` on cost-sensitive tries, `quality_threshold: 0.6` for stylized / abstract work, or `submit_plan({plan_id, confirm:true})` to skip the cost-preview pause when you trust the chain. ## How this works Seven gates. The first one — confirming the beat-mapped scene plan — is the entire premise. Get the scene-to-section mapping right and the cuts feel motivated. Get it wrong and the video drifts from the song. ## How the agent should run this (interaction contract) 1. **Confirm first (one message).** Restate: "'{song_title}' → 8-12-scene music video + 9:16 cut, ~$3-4, ~35 min." Offer real choices: visual_style direction · 16:9 vs 9:16 · key art yes/no (OPTIONAL, +$0.40). Ask at most ONE question (usually "does song_url play?"); otherwise state assumptions and start. 2. **Preview checkpoint.** After the Step-1 scene plan, render ONE test still (~$0.10) in the proposed style BEFORE the full moodboard, and show the moodboard BEFORE the i2v wave. "Wrong style costs $0.10 here, $1.20 after the batch — redirect now." 3. **Narrate.** Every `create_media` returns a job_id — poll `get_create_media` every 10-15s and post one line per step: "Animating scene 5/10 (seedance-i2v, ~3-4 min each)". Never silent >2 min; heartbeat during the i2v wave. 4. **Fail gracefully.** seedance-i2v hang/queue → say "seedance queued at fal, switching", retry once, then `ltx-q-i2v` (`resolution:"auto"`, `num_frames` matched to the scene's seconds — 241 for a 10s clip). `wizper` failure → `nemotron-asr`. `ffmpeg-concat` drops the clips array via MCP → direct SDK call (see notes at bottom). NEVER regenerate the user's Suno track. ≤2 retries per step; if 2 scenes won't animate, stitch the rest + Ken-Burns the stills, and say so. 5. **Deliver.** Final MP4 + short cut + one honest line: "i2v scenes drift from the approved stills, and cuts land NEAR section boundaries, not sample-accurate on the beat." + ONE next step: "Post the 9:16 cut first — that's the discovery engine." ## The steps ### Step 0 — Confirm the brief The agent reads back the BRIEF and previews the song inline (audio player). Lets you re-listen before any inference fires. **STOP**: "Brief looks right? Song plays correctly? (approve / edit [field] [new value])" ### Step 1 — Extract structure + propose scene plan (~1 min, ~$0.05) The agent transcribes `song_url` with `wizper` (timestamps + lyrics) and reads the total duration, then has `gemini-text` infer the song structure: sections (intro / verse-1 / chorus-1 / verse-2 / bridge / outro) with start_time + duration in seconds. From that it writes a scene plan: 8-12 scenes, each labeled with its song section, duration, and a one-sentence visual brief derived from `lyric_themes` + `song_mood`. Slow sections → contemplative scenes (static framing, soft camera). Choruses → dynamic scenes (motion, color shift, energy). The plan is rendered as a timeline card you scrub through. > **For the agent**: the scene plan MUST explicitly list each scene's section + start_time + duration so Step 4 can match motion energy and Step 5 can align video length to song length. (There is no `beat-extractor` cap on BYOC — structure inference is `wizper` transcript + `gemini-text`, not an audio-DSP cap.) **STOP**: "Scene plan looks right? (approve / edit scene [#] [new description] / restructure)" ### Step 2 — Lock the brand kit (~30s, ~$0.05) Build a brand kit from `visual_style`, `song_mood`, `lyric_themes`. The agent infers 6-8 voice keywords ("16mm film grain", "golden hour", "candid hands holding steering wheel") and writes 3 reference prompts. Renders the references, sets as active brand kit. Every later moodboard frame inherits the palette + voice. **STOP**: "Approve the brand kit, or refine? (approve / refine [direction])" ### Step 3 — Scene moodboard (~6 min, ~$1.20) The agent calls `generate_project` with the scene plan from Step 1 — 8-12 scenes, each rendered as a still frame at premium quality. Brand-locked. Composition variety across the set (hero / lifestyle / atmospheric / detail) so the video doesn't feel monotonous. > **For the agent**: pass `quality_threshold: 0.85` per scene. If `character_or_artist` is set, pass it as `quality_anchor_url` on any scene the user's brief mentions the artist appearing. Surface the cross-asset critic verdict after the batch — flags any scene that drifts from brand or doesn't match its section's energy. **STOP**: "Open the moodboard viewer. Approve all scenes, or redo specific? (approve / redo [#] [direction])" ### Step 4 — Animate each scene (~10 min, ~$1.20) The agent calls `create_media({ action: "animate", model_override: "seedance-i2v", source_url: , resolution: "auto" })` on each moodboard still. Verses get slow-drift motion. Choruses get camera dynamics. Duration per clip matches its section: 3s for short transitions, 5s for chorus sustains. Runs in parallel via `submit_plan` so total wall-clock is bounded by the slowest single render. > **For the agent**: prefer `seedance-i2v` (best character/motion consistency) or `ltx-q-i2v` (LTX Quality tier) — both over base `ltx-i2v`. Always pass `resolution: "auto"` (the literal `"1080p"` 422s on ltx). Explicit motion brief per clip — "slow camera dolly forward, soft grain, no character motion" for verses; "handheld energy, character moves freely, dynamic light shift" for choruses. Use `quality_threshold: 0.75` (animation drift is normal at higher thresholds; relaxed gate). **STOP**: "All scenes animated. Approve, or redo? (approve / redo [#] [direction])" ### Step 5 — Stitch + lay the song over the video (MANDATORY finishing, ~2 min, ~$0.05) 1. `create_media({ model_override: "ffmpeg-concat" })` on the animated scenes in scene-plan order (transition: cut) → one silent stitched MP4. 2. `create_media({ action: "mux_audio", model_override: "ffmpeg-mux", source_url: , audio_url: , audio_fill: "loop" })` → lays your full Suno track over the video. > **Why `ffmpeg-mux` + `audio_fill: "loop"` is mandatory:** a bare mux truncates audio to the shorter stream and you get a silent tail (or a clipped song). `audio_fill: "loop"` makes the audio cover the FULL video; use `"pad"` if you'd rather end on silence than loop. This is the single most common "the music stops early" bug — do not skip it. Aim for total video duration ≈ song duration: have Step 4 size scene durations to the section timings from Step 1 so the stitched length lands within ~2s of the song. Optional `ffmpeg-overlay` LUT pass for color grade if `visual_style` calls for it (teal-and-orange, faded film, etc.). > **For the agent**: the standard MCP envelope drops array params on `ffmpeg-concat` with 3+ clips — fall back to direct SDK with the `clips` array in the params body. See the notes-for-agent section if this errors. ### Step 6 — 9:16 short cut (~2 min, ~$0.10) The agent picks the strongest 30-45s window (usually a chorus + immediate context) and calls `ffmpeg-export` to `preset: "tiktok-portrait"` with `crop: "center-crop"`. One call covers both the crop and the platform-ready bitrate. This is what gets posted to Reels / TikTok / Shorts. **STOP**: "Short cut approved? Or pick a different window? (approve / use [verse / chorus / outro])" ### Step 7 — Key art (~2 min, ~$0.40) A 1920×1080 promotional still — the "this is the song" key visual, derived from the strongest moodboard frame, re-rendered with `quality_threshold: 0.9`. Used as the streaming-platform cover (Spotify, Apple Music) and as the YouTube thumbnail. > **For the agent**: use `gpt-image` for this — it handles `song_title` + `artist_name` text overlay if the user wants typographic key art. Otherwise `flux-dev` at premium quality. **STOP**: "Key art approved? (approve / refine [direction])" ### Final step — scorecard wrap-up Call `get_scorecard({project_id: ...})` — paste output to your channel. ``` Scorecard for cjob_xxx N/N scenes shipped — critic: SHIP Title: {song_title} Steps: N/N succeeded (100%) Quality: mean X.XX · M/N passed · K retries Cost: $X.XX actual (est $Y.YY) · Ts wall time Viewer: ``` ## When you're done ``` ✅ Music video MP4 (16:9 or 9:16 native): ✅ Short cut (9:16 ~40s): ✅ Key art (1920×1080): ✅ Scene moodboard viewer: Total spent: $X.XX Total wall-clock: MM:SS ``` ## What to do next Upload the music video to YouTube with the key art as thumbnail. Post the 9:16 short cut to Reels / TikTok / Shorts — that's the discovery engine. Use the key art as your Spotify / Apple Music cover for the single. Run this playbook again on every Suno song you ship — the brand kit and visual_style stay consistent across an EP, so the videos form a cohesive aesthetic. By the time you have a full project, you have a visual identity, not just isolated tracks. --- ## Notes for the agent (only read if a step fails) **Stitching multiple clips**: the standard MCP envelope drops array params. For `ffmpeg-concat` with 3+ clips, fall back to direct SDK: ```bash curl -X POST https://sdk.daydream.monster/inference \ -H "Authorization: Bearer $DAYDREAM_API_KEY" \ -H "Content-Type: application/json" \ -d '{"capability":"ffmpeg-concat","params":{"clips":[,...,],"transition":"cut"}}' ``` **Audio sync drift**: with `audio_fill: "loop"` the song always covers the full video. If you want the song to END the video instead of looping, size scenes so the stitched length matches the song (re-render the longest scene at extended duration) and use `audio_fill: "pad"`. **On animate failure**: `seedance-i2v` is the default; on failure retry once with `ltx-q-i2v`, then `grok-imagine-video`. Always pass `resolution: "auto"`. ## Execution mode (autopilot / director / auto) This is a multi-scene project, so pass a `mode` to `generate_project` to control the **pre-finish gate** (after every scene renders, before delivery): `autopilot` (certain brief → ship in one call, no pause), `director` (always pause for a human pick/steer before delivery), or `auto` (recommended — ship UNLESS the auto-critic is unsure: score < `confidence_threshold` (default 0.8), verdict `iterate`, or a flagged shot). Setting `mode` runs the project async (returns a `job_id`); resume a director/auto pause with `review_checkpoint` + `resume_from_checkpoint` (decision `continue` to deliver / `abort` to discard). Omit `mode` for legacy straight-through behavior.