One network for AI media creation

AI generation, deterministic finishing tools, and external publishing — under a single envelope, a single bearer, and a single agent runtime. Built for teams shipping media-creation features without owning the GPU stack, the tool stack, or the orchestration glue.

AI capabilities
60+ generation models — image, video, audio, 3D, TTS, multimodal. New models land via config-only registration.
Tool capabilities
Deterministic Docker — ffmpeg-concat, ffmpeg-mux, ffmpeg-export, opencv-smart-crop, yolo-detect. Sub-second, near-zero cost.
MCP capabilities
Bridged external services — Slack, Drive, Linear, GitHub. First call surfaces OAuth, then succeeds silently.
One envelope. Three kinds of capability. One bearer.
Generation, finishing, and publishing are all reachable through the samePOST /inference {capability, input}shape, billed to the same Daydream key.

The three deployment shapes

ShapeWho runs the loopPick when
CLI / SDKYour terminal or Node serviceBackend workflows, batch jobs, CI integration
MCP serverClaude Code / Cursor / ChatGPTLowest-friction distribution. Your users live in their AI tool already.
Embedded runtimeYour app imports @livepeer/agentFull control. Add your own tools alongside the network.

Quick Start

Four supported integration paths. Pick by where your users live. Each one authenticates with the same Daydream API key (sk_…) — get yours at app.daydream.live.

① Set up the MCP server (Claude Code, Cursor, ChatGPT)

The Storyboard MCP exposes the entire network as tools your AI client can call:generate_project,create_media,list_capabilities,submit_agent_task.

claude-code (~/.claude.json)
{
  "mcpServers": {
    "storyboard": {
      "url": "https://storyboard.daydream.monster/api/mcp/sse",
      "transport": "sse",
      "headers": {
        "Authorization": "Bearer sk_<your-daydream-key>"
      }
    }
  }
}

Restart Claude Code. The tools appear as mcp__storyboard__*. Try:

chat
Use the storyboard MCP to give me a 12-scene illustrated short story about
a Scottish ferryman's last crossing — watercolor pencil, John Bauer x Ghibli,
3:2, character-locked across scenes.

② Install and use the CLI

One-line install. Runs locally as a terminal app or a one-shot verb.

bash
curl -fsSL https://storyboard.daydream.monster/cli/install.sh | bash
livepeer setup --non-interactive --api-key sk_<your-daydream-key>

# One-shot
livepeer agent run "make me a watercolor fox in Studio Ghibli style"

# Interactive REPL
livepeer
livepeer> /story 8-scene story about a fox in autumn
livepeer> /film 4-shot sci-fi opener about a derelict station
livepeer> /talk "Welcome to Aurora" --face img-1

Switch providers anytime — your Daydream key is the default (Livepeer-hosted), but you can swap:

bash
livepeer config llm gemini --model gemini-2.5-pro    # Gemini direct
livepeer config llm anthropic --model claude-sonnet-4-6  # Claude direct
livepeer config llm ollama --model llama3.1          # local
livepeer config llm livepeer                         # back to hosted

③ Use the Storyboard webapp

The visual surface — canvas, chat, projects, LoRA training, live streams. Best for visual approval and creative iteration.

quick steps
1. Open  https://storyboard.daydream.monster
2. Sign in with your Daydream account
3. Type in chat:
     /story 6 scenes of a fox catching salmon in autumn
4. Watch the scenes render on the canvas in parallel
5. Right-click any card for context-menu actions:
     • Animate (i2v)         • Variations (×4)
     • Restyle / Edit        • Talking Video
     • Convert to 3D         • Virtual Try-On
     • Analyze Media         • Cinematic Video
6. /render to stitch the canvas into a final video

④ Use the Agent SDK directly (@livepeer/agent)

Embed the runtime in your own app. Same agent loop, your tools alongside the network's.

bash
npm install @livepeer/agent
typescript
import {
  AgentRunner,
  ToolRegistry,
  WorkingMemoryStore,
  LivepeerProvider,
} from "@livepeer/agent";

// 1. Provider (uses YOUR users' Daydream key for billing)
const provider = new LivepeerProvider({
  apiKey: process.env.DAYDREAM_API_KEY!,
});

// 2. Tool registry — built-in tools work out of the box
const tools = new ToolRegistry();
// Or add your own:
tools.register({
  name: "send_to_slack",
  description: "Post an asset URL to a Slack channel",
  parameters: {
    type: "object",
    properties: {
      channel: { type: "string" },
      url: { type: "string" },
    },
    required: ["channel", "url"],
  },
  execute: async (args) => {
    await myAppSlackPost(args.channel as string, args.url as string);
    return JSON.stringify({ ok: true });
  },
});

// 3. Memory and runner
const memory = new WorkingMemoryStore();
const runner = new AgentRunner(provider, tools, memory);

// 4. Stream events
for await (const event of runner.runStream({
  user: "Generate a hero image for the spring launch and post it to #marketing",
})) {
  if (event.kind === "text") process.stdout.write(event.text);
  if (event.kind === "tool_call") console.log("→", event.name, event.args);
  if (event.kind === "tool_result") console.log("←", event.content);
}

Direct HTTP — for any language. Same envelope every capability shares:

bash
curl -X POST https://sdk.daydream.monster/inference \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk_<your-daydream-key>" \
  -d '{
    "capability": "flux-dev",
    "input": { "prompt": "a watercolor fox", "image_size": "landscape_16_9" }
  }'

Architecture

Three consumer surfaces (CLI / MCP / Embedded) → SDK service → BYOC + Scope orchestrators → AI / Tool / MCP adapters. Everything routes the same envelope, authenticated by the same bearer.

livepeer.architecture
Consumer surfaces (pick one or all three)
─────────────────────────────────────────
  • Direct CLI / SDK  (`@livepeer/agent`)
  • As MCP server     (Claude Code, Cursor, ChatGPT)
  • Embedded runtime  (import the SDK into your app)

                        │
                        │  Bearer: sk_<user-daydream-key>
                        │  Wire:  POST /inference { capability, input } → { output }
                        ▼

  Livepeer agent layer
  ┌───────────────────────────────────────────────┐
  │  • SDK service        sdk.daydream.monster     │
  │  • Storyboard MCP +   /api/mcp/sse + /api/llm  │
  │  • Hermes (long-task) agent.daydream.monster   │
  └───────────────────────────┬───────────────────┘
                              │
                              ▼
  Orchestrator network (go-livepeer, decentralized)
  ┌───────────────────────────────────────────────┐
  │  • BYOC orchs   AI / Tool / MCP capability    │
  │  • Scope orchs  live video-to-video           │
  │  • Signer       Arbitrum payment tickets      │
  └───────────────────────────┬───────────────────┘
                              │
                              ▼
  Adapters — all speak the same /inference envelope
  ┌───────────────────────────────────────────────┐
  │  AI   : Flux, Seedance, GPT-Image, Veo,       │
  │         Chatterbox, Music, Tripo, Gemini-text │
  │  TOOL : ffmpeg-concat / -mux / -export,       │
  │         opencv-smart-crop, yolo-segment/-detect│
  │  MCP  : Slack, Drive, Linear, GitHub          │
  └───────────────────────────────────────────────┘

Why this design wins

Tools Reference

The agent uses 21 tools to create and manipulate media. Each tool has typed parameters and returns structured results. Tools are composed into multi-step workflows automatically by the LLM.

Media Creation

Canvas Operations

Live Streaming (Scope)

Skills

Skills are the agent's soft-coded knowledge layer — markdown files that teach the agent how to think, not whattools to call. They're fetched at runtime from /skills/*.md, ranked against the current intent, and injected into the system prompt before each turn. Edit a skill, save the file, the next turn already runs the updated behavior — no code change, no deploy.

How skills fit into the tool architecture

Tools execute. Skills orient — they bias the agent toward the right tool, the right parameters, and the right finishing pass. The agent loop pulls relevant skills at every turn:

skills.flow
User turn
   │
   ▼
findSkills(userMessage, context)  ─── ranks all skills by frontmatter
                                       relevance (task, domain, persona,
                                       scope) — top 3-5 win
   │
   ▼
Skills concatenated into system prompt:
   "You are a creative agent. CRITICAL CONSTRAINTS:
    [base.md content]
    [director-mcp.md content]
    [finishing-quality.md content]
    [storyteller.md content]
    ..."
   │
   ▼
LLM call (with tools + system prompt)
   │
   ▼
Agent fires tools according to skill guidance:
   - "movie quality" → seedance-i2v (from director-mcp)
   - "watercolor pencil" → gpt-image (from director-mcp model table)
   - finishing pass mandatory (from finishing-quality)
   - cut transitions for 3+ clips (from director-mcp)
   │
   ▼
Skill outcomes scored after turn ends — feeds future
ranking (skills that produced good outcomes get bumped)

Skill frontmatter (the routing metadata)

Every skill file starts with YAML frontmatter that the ranker reads. The body is markdown the LLM reads.

director-mcp.md
---
name: director-mcp
description: "Director-language → multi-step MCP recipes; pick defaults, don't name models"
agent_rule: "User describes a deliverable — pick model/params, don't make them name caps."
task: [render, compose, animate, narrate]
domain: [video, image, audio, film]
persona: [agent-only]
scope: epic
---

# Director — natural language → MCP recipes

When the user says "30-second cinematic clip," you pick flux-schnell
for keyframes + seedance-i2v-fast for animation + ffmpeg-concat for
stitching. They don't name models; you do.

[... full markdown body teaches the agent ...]

Skills available today (40 in the registry)

Grouped by where they sit in a creative session.

Foundation (always loaded)
base.mdCore agent rules — when to call create_media vs project_create, finishing-pass discipline, refusal handling.
intent-classifier.mdIntent vocabulary — `compare_models`, `batch_generate`, `style_sweep`, `variations`, `story`, `single`, `unclear`.
Directing / Orchestration
director.mdGeneral multi-step director: plan, generate, review, iterate for multi-scene work.
director-mcp.mdDirector-language → MCP recipe table. Model defaults per intent (cinematic / cinemagraph / talking-head / social-reel).
episode-director.mdTurn images-into-episode workflow. Cohesion prefix, motion prompts, transitions.
claude-code-mcp.mdPlaybook for Claude Code orchestrating the Storyboard MCP — long-context, multi-tool, AAA quality.
live-director.mdChat commands → LV2V stream_control mapping for live streams.
scope-agent.mdScope Domain Agent: graphs, presets, LoRA, VACE, natural-language parameter mapping.
Storytelling
storyboard.mdMulti-shot storyboard from a pasted scene description — split into shots, call create_media ONCE.
storyteller.mdGenerates 6-scene visual stories: title, audience, arc, context, scenes.
character-design.mdPer-character anchor + LoRA training guidance for cross-shot identity.
Image & Style
prompt-craft.md7-layer prompt formula: camera, subject, surface, background, lighting, colors, atmosphere.
text-to-image.mdModel selection guide, size rules, common pitfalls per model family.
image-editing.mdkontext-edit usage, style transfer, reference-driven composition.
cinematography.mdCamera framing + movement vocabulary. Lens choices, depth-of-field cues.
style-presets.md15 built-in style presets with optimized prompt suffixes.
grok-imagine.mdxAI Grok Imagine quality + edit usage notes.
ideogram-bg-remove.mdHigh-fidelity subject isolation patterns.
Video & Animation
video.mdi2v/t2v pipeline selection, motion prompts, when to chain.
hifi-video.mdTwo-step pipeline: GPT Image 2 keyframe → Seedance 2.0 animation.
keyframe-video.mdBuilding cohesive multi-shot video from keyframes.
happy-horse-cinematic.mdWorked example — cinematic horse video from keyframe → animation → finishing.
scope-lv2v.mdScope live video-to-video parameters: noise, transitions, denoising step list.
scope-graphs.mdScope pipeline graphs and selection (simple-lv2v, depth-guided, etc.).
scope-pipelines.mdPipeline catalog: LongLive, LTX 2.3, Krea, MemFlow. 9 composable recipes.
stream-source-analysis.mdSource-quality gates before starting a live stream.
Finishing & Quality
finishing-quality.mdThe finishing checklist — concat / mux / export / crop / overlay / burn captions. "Don't ship a URL list."
refinement.mdIterative refinement: generate, evaluate, upscale.
remix.mdCombine multiple canvas cards into composites.
advanced-models.mdAdvanced model guide: video gen, audio, editing, 3D, export workflows.
non-ai-tools.mdDeterministic Docker tool catalog: ffmpeg-*, opencv-*, yolo-*. When to reach for each.
yolo-vision.mdSubject detection / segmentation / pose patterns for cinemagraphs + smart-crop.
mcp-bridged-tools.mdWhen to publish via Slack / Drive / Linear / GitHub — what payload shape each expects.
Vertical / Domain
marketing.mdBrand-locked imagery, audience considerations, A/B variant patterns.
commercial.mdCommercial-style pacing, hero-shot composition, brand frame conventions.
ugc.mdInfluencer / handheld / phone-shot aesthetics — when and how to use them.
daily-briefing.mdEmail-driven visual deck (Gmail MCP → slides).
Training & Specialized
lora-training.mdLoRA training: image selection, trigger words, step counts, drift detection.
nemotron-omni.mdNVIDIA Nemotron multimodal usage — text+image+video+audio reasoning.

How skills change agent behavior — concrete before/after

A skill is text the LLM reads before deciding what to do. Same prompt + same tools, but with a skill loaded, the agent picks different models, different parameters, and a different finishing pipeline. Three short examples:

User: "Make a 30-second cinematic clip about a lone surfer at sunset."
WITHOUT director-mcp loaded
  • 1× create_media(prompt='cinematic surfer sunset', capability='flux-dev')
  • Returns ONE image URL — call ends.
  • User has to ask for finishing, voice, music separately.
WITH director-mcp loaded
  • Wave 1 parallel: 6 × flux-schnell keyframes + chatterbox-tts narration + music score (30s, instrumental).
  • Wave 2 parallel: 6 × seedance-i2v-fast (5s, slow-dolly camera motion per shot).
  • Wave 3 sequential: ffmpeg-concat (cut) → ffmpeg-audio-mix (voice 1.0, music 0.30) → ffmpeg-mux (replace).
  • Returns ONE finished MP4 with audio. ~250s wall-clock, single deliverable.
User: "Generate 8 illustrated scenes for a Ghibli-style fox story."
WITHOUT studio-ghibli + storyteller
  • Agent uses base prompt as-is → flux-dev defaults.
  • Scenes vary in style (#1 watercolor, #4 photoreal, #6 cartoon). Character drifts between shots.
  • Outputs 8 inconsistent URLs.
WITH studio-ghibli + storyteller loaded
  • storyteller plans 8 scenes with title + arc + character anchor.
  • studio-ghibli auto-prepends 'studio ghibli style, hand-painted watercolor...' to every prompt.
  • Picks gpt-image (best stylization) over flux-dev for all 8 scenes.
  • Outputs 8 cohesive scenes, returned as one viewer URL via generate_project.
User: "Cut out the cat from this photo."
WITHOUT yolo-vision loaded
  • Agent guesses: tries kontext-edit with prompt 'remove background, keep cat'. ~6s.
  • Result: ok but jagged edges, half-faded paw.
WITH yolo-vision loaded
  • Agent recognizes 'cut out' intent → yolo-segment with classes=['cat'], output_mode='alpha-cutout'. ~1s.
  • Pixel-precise alpha PNG. Tells the user 'cinemagraph-ready'.

The skill doesn't add new capabilities — those already exist on the network. The skill teaches the agent which one to reach for, with what parameters, in what wave order, and what finishing pass to run. Same hands, better training.

Loading, adding, and unloading skills — across all three surfaces

Skills are managed the same way conceptually across MCP, CLI, and the Storyboard webapp — the syntax differs by surface. The active-set is per-user (persisted), so a skill loaded via the CLI is also active for your next REPL session and surfaces in the webapp's "active skills" pill bar.

ActionMCP (Claude Code / Cursor)CLIStoryboard chat
listNatural language:
"Which skills are available?"
Agent runs livepeer.list_skills
livepeer skills list/skills/list
show"Show me what director-mcp teaches."livepeer skills show <id>/skills/show <id>
load (pin for session)"Apply the studio-ghibli skill for the rest of this chat."
Agent runs livepeer.apply_skill_pack
livepeer skills load <id>/skills/load <id>
load by category"Activate all marketing skills."livepeer skills load --category marketing/skills/load-by-category marketing
unload"Drop the ugc skill, I want polished output now."livepeer skills unload <id>/skills/unload <id>
show active set"What skills are active right now?"livepeer skills activeTop of chat — always-visible pill bar
add (author + install)"Make a new skill for our brand voice."
Agent runs livepeer.gen_skill
livepeer skills new
livepeer skills validate
livepeer skills install
/skills/new
Or autogenesis (post-turn)
publish to public registryRoadmap — MCP publish toollivepeer skills publish <id>/skills/publish <id>
private sync across devicesBearer-scoped; happens automaticallylivepeer skills push
livepeer skills pull
Automatic on sign-in
auto-load on/offServer-side; admin-controlledlivepeer skills auto-load on|offSettings → Skills

The active-set is shared across surfaces

Skills you load via the CLI persist to ~/.livepeer/skills/active.json AND to a per-user blob mirror keyed by your Daydream bearer. The next time you sign into the Storyboard webapp the same skills are active. Sign into Claude Code with the same MCP key — same skills again. One bearer, one active set, all three surfaces.

examples — same skill, three surfaces
# Day 1 — author and pin on your laptop CLI
$ livepeer skills new --name aurora-brand-voice
$ livepeer skills load aurora-brand-voice
$ livepeer agent run "make me a launch hero image"
  # Hero respects the brand voice immediately.

# Day 2 — fire up the Storyboard webapp
  # The chat header pill bar shows: [aurora-brand-voice ●]
  # Every generation already brand-locked.

# Day 3 — switch to Claude Code (MCP) for an iterative session
  > "Use the storyboard MCP to draft 4 social cuts."
  # Claude calls list_skills, sees aurora-brand-voice active
  # and uses it without you asking.

# Roll back anywhere:
$ livepeer skills unload aurora-brand-voice
# OR in the webapp:
/skills/unload aurora-brand-voice
# OR ask Claude Code:
  > "Drop the aurora-brand-voice skill — I want generic output."

Under-the-hood — what each surface actually does

Every surface ends up doing the same thing: it modifies the user's active-set and the next agent turn reads the updated set when it builds the system prompt.

lifecycle
User action on any surface
   │
   ▼
1. Active-set update:
     local:   ~/.livepeer/skills/active.json
     remote:  storyboard-mcp/user-mirror/<user-hash>.json  (Vercel Blob,
                                                              mirrored across
                                                              devices)
   │
   ▼
2. Next agent turn fires:
     findSkills(userMsg, { tools, persona })
     → ranks ALL skills by relevance
     → top-N + every pinned skill from active-set
       always wins (pinned override ranking)
   │
   ▼
3. System prompt assembled:
     [base.md content]
     [pinned skill 1]                ← from your active-set
     [pinned skill 2]
     [ranked-in skill 3]             ← matched the current intent
     [ranked-in skill 4]
   │
   ▼
4. LLM call (with system prompt + tools)
   │
   ▼
5. Outcome scored at turn end:
     - did the agent reach for a tool the skill recommends?
     - was the result rated thumbs-up?
     - update skill's effectiveness score in the per-user ledger
   │
   ▼
6. Future rankings boost or demote based on outcomes
   (your skills get smarter with use)

Extending and personalizing — write your own

Skills are file-based. Drop a new .md in public/skills/ (or via the chat autogenesis flow), register a one-line entry in public/skills/_registry.json, and your skill is loaded by the ranker on next session.

public/skills/my-brand-voice.md
---
name: my-brand-voice
description: "Brand voice for Aurora Audio — premium, understated, hopeful"
agent_rule: "All copy targets Aurora Audio brand voice. Premium without superlatives."
task: [render, narrate]
domain: [image, video, audio]
persona: [agent-only]
scope: epic
---

# Aurora Audio brand voice

Tone: confident, understated, premium. Avoid superlatives ("uncompromising",
"the best", "world's first"). Prefer warmth ("the headphones that come home
with you", "made for the quiet hour").

Visual cues:
  - Always cool blue / charcoal / brushed silver palette
  - Studio lighting, never sunlight
  - Product on minimalist surfaces — concrete, walnut, marble

Caption patterns:
  - TikTok: short, sensory, second-person ("you'll hear...")
  - Instagram: longer, story-led, soft pull-quote
  - LinkedIn: hopeful, work-life balance angle

When generating campaign creative, always reference this skill alongside
director-mcp.md and finishing-quality.md.
public/skills/_registry.json (entry)
{
  "id": "my-brand-voice",
  "category": "vertical",
  "description": "Aurora Audio brand voice — premium understated hopeful",
  "tags": ["brand", "voice", "aurora"]
}

Skill autogenesis (the meta-skill)

Skills can be born from successful sessions. After a turn finishes well, the agent (via the skills B3 autogenesis flow) drafts a candidate skill capturing what made it work — proposed via the promotion UI; the user approves or rejects. Approved skills go into the registry and start informing future turns. Your agent gets smarter without a human writing markdown.

Agent Marketplace

The Agent Marketplace is the registry of pre-built, runnable agents you can summon by name. Where a skill teaches, an agent does — each marketplace entry bundles a system prompt + a curated skill stack + default tool gates + (optionally) a trained LoRA + a brand kit. Summon one and the conversation already knows who it is and how to behave.

How agents work in the network

agent.invocation
User chat:                       Or CLI:                   Or MCP:
  /agent aurora-marketer           livepeer agent run \        mcp__storyboard__
    "launch the X1 next Tuesday"     --agent aurora-marketer \    create_agent_task
                                     "launch the X1 next Tuesday"  agent_id: aurora-marketer
   │                                 │                              prompt: "launch ..."
   │                                 │                              │
   └──────────────┬─────────────────┘                              │
                  │                                                │
                  ▼                                                ▼
       ┌──────────────────────────────────────────────────────────────┐
       │ Agent registry resolves "aurora-marketer" to:                 │
       │   • system_prompt: "You are Aurora's brand director ..."     │
       │   • skill_refs:    [director-mcp, marketing, my-brand-voice] │
       │   • lora_refs:     [bkit_aurora_lora_v3]                     │
       │   • tool_allowlist: [create_media, project_create, …]        │
       │   • brand_kit_id:  bkit_aurora_a1b2c3                        │
       │   • author + signature + reputation score                    │
       └──────────────┬───────────────────────────────────────────────┘
                      │
                      ▼
              Agent loop runs with this bundle pre-loaded.
              First turn is already brand-aware, model-aware,
              and finishing-pass-aware.

Off-the-shelf agents (live in the marketplace today)

Available at storyboard.daydream.monster/marketplace. Each agent is callable from CLI, MCP, and the webapp.

Storytelling
story-architect12-scene illustrated story planner. Outputs scenes + style prefix + character anchor. Best for fiction, fables, brand stories.
graphic-novelistMulti-page graphic-novel pacing — panels, dialogue, character consistency, page transitions.
documentary-narratorB-roll planning + voice-over scripting from a brief. Pairs with chatterbox-tts.
Film & Video
short-film-directorEnd-to-end 2-min short film: keyframes + i2v + voice + score + finishing + export.
cinemagraph-artistMagic-moving-photo specialist. yolo-segment + i2v + ffmpeg-overlay pipeline.
music-video-directorLyric/music-driven video. Beat-synced cuts, prompt traveling across scenes.
happy-horse-cinematicWorked-example agent showing the gpt-image → seedance → finishing chain.
Marketing & Brand
campaign-launcherBrand-LoRA + multi-platform campaign — hero spot, social cuts, stills, carousel, scheduling.
social-shorts-creator8-episode TikTok-format series with continuity. Same character, evolving narrative.
product-stageSingle-product hero shots with brand frame and lifestyle context library.
Game & 3D
character-sheetGame character: front/3-4/side/back turnaround, expressions, LoRA training, 3D mesh export.
environment-conceptStylized environment passes with controlnet preprocessor pack.
Live
scope-vjLive VJ — multi-scene Scope stream with prompt traveling, beat-synced parameter modulation.
talking-presenterImage + script → talking-head video, lip-synced.
Utility & Free
image-analyzerGemini Vision-driven analysis: style, palette, characters, setting, mood.
asset-finisherPure finishing-pass agent — give it N urls, get a stitched MP4 with audio, captions, platform export.
brief-validatorPre-flight check on a campaign brief — compliance flags, missing fields, cost estimate.

The agent manifest

Every marketplace agent is a signed JSON manifest in Vercel Blob storage. Tamper-proof; reputation tracked by signature.

agent.manifest
{
  "id": "aurora-marketer",
  "version": "1.4.2",
  "title": "Aurora Audio brand director",
  "author": "@aurora-creative",
  "signature": "0xabcdef...",       // signed with author's wallet
  "reputation": 4.8,                 // 0-5 from rated runs

  "system_prompt": "You are Aurora's brand director. Premium, understated...",

  "skill_refs": [
    "director-mcp",
    "marketing",
    "finishing-quality",
    "my-brand-voice"
  ],

  "lora_refs": [
    "bkit_aurora_lora_v3"            // private LoRA, accessible to owners
  ],

  "brand_kit_id": "bkit_aurora_a1b2c3",

  "tool_allowlist": [                // what this agent is allowed to call
    "livepeer.create_media",
    "livepeer.project_create",
    "livepeer.project_generate",
    "livepeer.canvas_organize",
    "ffmpeg-*",
    "publora-schedule"
  ],

  "pricing": {
    "model": "free" | "per-run" | "subscription",
    "fee_eth_per_run": "0.001",       // marketplace fee on top of compute
    "creator_split": 0.7              // 70% to author, 30% to network
  }
}

Building and publishing your own agent

Three steps. The webapp has a guided builder at /marketplace/register — the API is also exposed for scripted submission.

bash
# 1. Build the manifest locally (CLI helper)
livepeer agent build \
  --id my-fashion-director \
  --title "Fashion campaign director" \
  --skills marketing,director-mcp,finishing-quality \
  --brand-kit bkit_aurora \
  --lora lora_aurora_v3 \
  --tool-allow create_media,project_create,publora-schedule \
  --pricing per-run --fee 0.0005 --output ./my-agent.json

# 2. Sign it with your wallet
livepeer agent sign ./my-agent.json --key ~/.daydream/wallet.key

# 3. Register on the marketplace
livepeer agent publish ./my-agent.json
# → Published as agent_a1b2c3d4 — your share of fees lands in
# → 0x1234... on Arbitrum within 24h of each invocation.

Sales & reputation

When users summon your agent, the marketplace records the run, optionally collects a rating, and updates your reputation. Earnings (your creator split of the per-run fee) settle on-chain to the wallet you signed the manifest with. Subscription pricing is supported for agents bundled with proprietary LoRAs or skills.

Extending an existing agent (forking pattern)

Pull any public agent manifest, modify, sign, re-publish under your own id. Original author retains reputation; your version starts fresh and competes on its merits. Forking is encouraged — the network rewards differentiation via the rating system.

bash
livepeer agent fork aurora-marketer --new-id studio-x-marketer
# Opens the manifest in your editor. Tweak the system prompt,
# swap skill_refs, change pricing. Sign + publish.
livepeer agent publish ./studio-x-marketer.json

Workflows

Slash commands orchestrate multi-step creative workflows. Each creates a draft, lets the user edit inline, then applies.

/story <concept>
LLM generates 6 scenes → StoryCard with inline editing → Apply creates project → project_generate → images on canvas
Models: flux-dev (default), auto-routed by style
/film <concept>
LLM generates 4 shots with camera → FilmCard → Apply: key frames (flux-dev or gpt-image) → animate each (seedance-i2v) → videos on canvas
Models: hifi mode: gpt-image → seedance
/stream <concept>
LLM plans multi-scene stream → StreamPlanCard → Apply: scope_start → prompt traveling with transitions → auto-stop
Models: longlive, ltx2, krea, memflow (via recipes)
/talk <text>
TTS (chatterbox or gemini-tts) → talking-head animation → video card
Models: chatterbox-tts, gemini-tts, talking-head

Conversation Continuity

text
User: /story a dragon adventure
→ 6-scene story card (editable)

User: add more scenes about finding treasure
→ System detects continuation → generates 3 matching scenes → appends
→ Updated card with NEW badges → user deletes unwanted → Apply

User: /story a space adventure
→ resetForNewWork() clears old context → fresh story, no bleed

Live Streaming

Real-time AI video generation using Scope. The stream processes input frames through an AI pipeline, transforming them based on text prompts.

Stream Recipes

classicLongLive
Default. Stable prompt traveling, LoRA, VACE
ltx-responsiveLTX 2.3
24fps native, fast prompt response
ltx-smoothLTX+RIFE
48fps buttery smooth output
depth-lockLongLive+Depth
Preserve 3D structure
krea-hqKrea 14B
Highest visual fidelity
memflow-consistentMemFlow
Best character consistency

Stream Lifecycle

typescript
import { ScopePlayer } from "@livepeer/scope-player";

// In your React component:
<ScopePlayer
  sdkUrl="https://sdk.daydream.monster"
  apiKey="sk_your_key"
  externalStreamId={streamId}
  onStateChange={(state) => console.log(state.status, state.fps)}
  onSourceReady={(setSource) => {
    // Drag an image onto the player to transform it live
    setSource({ type: "image", url: imageUrl });
  }}
/>

Prompt Traveling (Scene Transitions)

typescript
import { PerformanceEngine } from "./performance";

const engine = new PerformanceEngine();
engine.setScenes([
  { index: 0, title: "Dawn", prompt: "golden sunrise over mountains", preset: "cinematic", duration: 30 },
  { index: 1, title: "Storm", prompt: "dark storm clouds, lightning", preset: "abstract", duration: 15 },
  { index: 2, title: "Night", prompt: "starry sky, aurora borealis", preset: "dreamy", duration: 25 },
]);

// Transitions use noise_scale + kv_cache ramp for smooth morphing
engine.play(controlFn, onStateUpdate);
engine.pause();   // freeze at current point
engine.resume();  // continue from where paused

Creative Kit

Shared framework for building creative apps. Both Storyboard and Creative Stage use it.

Stores (Zustand)

typescript
import {
  createArtifactStore,   // Canvas cards (images, videos, audio)
  createChatStore,       // Chat messages
  createProjectStore,    // Multi-scene projects
  createGroupManager,    // Episode/collection grouping
  createConversationContext, // Active work tracking
} from "@livepeer/creative-kit";

const artifacts = createArtifactStore({ maxArtifacts: 200 });
artifacts.getState().add({ type: "image", title: "Dragon", url: "...", refId: "img-1" });

Model Router (Self-Learning)

typescript
import { routeModel, recordModelLatency, getModelStats } from "@livepeer/creative-kit";

// Scores: speed (60%) + style match (30%) + capacity (10%)
const result = routeModel({
  action: "generate",
  prompt: "a logo with text",
  availableModels: new Set(["flux-dev", "gpt-image", "recraft-v4"]),
});
// After each inference, feed back actual speed → router learns
recordModelLatency(result.model, elapsedMs);

// Check learned stats
const stats = getModelStats();
// → Map { "flux-dev" → { avgMs: 3200, count: 47, currentSpeed: 8.9 } }

Pipeline Registry

typescript
import { createPipelineRegistry } from "@livepeer/creative-kit";

const registry = createPipelineRegistry();

// Resolve user intent to the best recipe
const recipe = registry.resolve("smooth 24fps stream");
// → { id: "ltx-responsive", pipeline: "ltx2", graph: {...}, defaults: { kv_cache: 0.3 } }

// List all recipes by quality tier
const quality = registry.listRecipes("quality");
// → [depth-lock, ltx-smooth, krea-hq, memflow-consistent, ...]

REST API Reference

Base URL: https://sdk.daydream.monster · Auth: Authorization: Bearer sk_...

POST/inferenceRun AI inference (image, video, audio, 3D)
GET/capabilitiesList all 48 available models
GET/healthService health check
POST/stream/startStart live AI stream
POST/stream/{id}/publishSend input frame (JPEG)
GET/stream/{id}/frameGet output frame (JPEG)
POST/stream/{id}/controlUpdate stream params
POST/stream/{id}/stopStop stream
GET/streamsList active streams
POST/api/agent/geminiGemini proxy (app route)
POST/api/agent/chatClaude proxy (app route)
POST/api/agent/openaiOpenAI proxy (app route)

Example: Generate Image → Animate → Get Video

bash
# Step 1: Generate key frame
IMG=$(curl -s -X POST https://sdk.daydream.monster/inference \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk_your_key" \
  -d '{"capability":"flux-dev","prompt":"a majestic eagle soaring"}' \
  | jq -r '.image_url // .data.images[0].url')

echo "Image: $IMG"

# Step 2: Animate to video (10 seconds with audio)
curl -X POST https://sdk.daydream.monster/inference \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk_your_key" \
  -d "{
    \"capability\": \"seedance-i2v\",
    \"prompt\": \"eagle gliding through mountain valley, cinematic camera follow\",
    \"params\": { \"image_url\": \"$IMG\", \"duration\": \"10\", \"generate_audio\": true }
  }"

# → { "data": { "video": { "url": "https://..." } } }

AI Models

60+ AI capabilities live on the network. Each is reachable ascapability: <name>on the /inference envelope. Adding a new model is a config-only change to CAPABILITIES_JSONon the BYOC orchestrator — no client release needed. Live list: /capabilities.

Image Generation
flux-dev~3sDefault. Cinematic photoreal.
flux-schnell~1sFastest draft / preview.
flux-pro / flux-flex~8sHigh-fidelity product photography.
gpt-image~8sBest for text, logos, stylization.
recraft-v4~6sProfessional illustration, editorial.
seedream-5-lite~5sStrong photorealism.
gemini-image~6sPainterly, watercolor, artistic.
nano-banana~4sMultimodal image model.
grok-imagine-quality~6sxAI Grok Imagine text-to-image.
Image Editing
kontext-edit~6sReference-driven edit. Preserves identity.
gpt-image-edit~10sInstruction-based edit (strong stylization).
grok-imagine-edit~6sGrok edit model.
flux-fill~5sInpaint / outpaint.
Video — Image → Video
seedance-i2v~30sDefault. 15s cinematic + per-clip audio.
seedance-i2v-fast~12s5s preview clips. Storyboarding.
veo-i2v~20sGoogle Veo i2v. 8s max.
kling-v3-turbo-i2v~40sKling 3.0 Turbo. Fast, stable motion, superior lip-sync (720p; -pro = 1080p).
ltx-i2v~15sOpen-source. Fast.
pixverse-i2v~20sPixverse i2v.
Video — Text → Video
veo-t2v~20sGoogle Veo 3.1 fast text-to-video.
ltx-t2v~15sFast open-source t2v.
pixverse-t2v~20sPixverse t2v.
Video — Specialized
veo-transition~25sInterpolate between two keyframes.
pixverse-transition~25sSame, Pixverse.
pixverse-ref2v~25sReference-driven video.
void-inpaint~30sVideo inpainting.
talking-head~40sLip-sync image + audio → video.
lipsync~25sAdd lip-sync to existing video.
face-swap~20sIdentity transfer.
fashn-tryon~30sVirtual try-on (garment + person).
TTS / Audio
chatterbox-tts~3sTTS + voice cloning (pass audio_url).
gemini-tts~3sTTS with persona/style control.
inworld-tts~3sFast natural TTS — staging-reliable.
grok-tts~3sxAI Grok TTS.
music~20sScore / instrumental music.
sfx~10sSound-effects generation.
3D / Scene
tripo-i3d~30sImage → 3D mesh (.glb).
tripo-t3d~30sText → 3D mesh.
tripo-mv3d~40sMulti-view → 3D mesh.
tripo-p1-t3d~40sTripo Pro tier text-to-3D.
tripo-p1-i3d~40sTripo Pro tier image-to-3D.
Background / Segmentation / Upscale
bg-remove~3sBirefnet — fast bg removal.
ideogram-bg-remove~6sHigh-fidelity subject isolation.
sam3~5sSAM 3 prompt-conditioned segmentation.
topaz-upscale~15sAura-SR upscale (4x photo / video).
Multimodal Reasoning
nemotron-omni~10sNVIDIA Nemotron — text+image+video+audio.
gemini-text~2sGemini 2.5 Flash — LLM via /llm/chat.
Live Streaming (Scope)
longlive8-12fpsDefault Scope pipeline. LoRA + VACE.
ltx224fpsLTX 2.3. Fast prompt response.
krea_realtime_video6-8fps14B model. Highest quality.
memflow8fpsMemory bank. Best consistency.

Non-AI Tools (Deterministic Docker)

Finishing-pass capabilities. Same /inferenceenvelope, same Daydream key. These run as Docker containers on the orchestrator — sub-second, near-zero cost (compute only, no model fee). The agent uses these to turn AI output into deliverables: stitched videos, platform-cropped clips, watermarked assets, captioned exports.

FFmpeg — Composition / Finishing
ffmpeg-concat~5sStitch N clips with cut/crossfade.
ffmpeg-mux~5sCombine video + audio track (replace mode).
ffmpeg-audio-mix~5sMulti-track mix (voice + music + sfx, per-track volume + delay).
ffmpeg-overlay~5sComposite logo / watermark on video.
ffmpeg-burn-subtitles~10sBurn captions from cue list.
ffmpeg-export~10sPlatform preset export (tiktok / instagram / youtube / x / linkedin).
ffmpeg-loop~5sLoop a clip to target duration or count.
ffmpeg-color-grade~10sLUT application, log-to-Rec.709 (planned).
OpenCV / Vision Tools
opencv-smart-crop~1sFace-aware crop. Pair with yolo-detect for focus_x.
yolo-detect<1sObject detection. Returns bounding boxes + labels (annotated jpg optional).
yolo-segment~1sPixel-precise segmentation. output_mode='alpha-cutout' for cinemagraph.
yolo-pose~1sBody / face keypoints. Quality-gate before talking-head.
brand-style-extract<1sDominant palette, detected font family, logo location (planned).
Audio / Captions
whisperx-transcribe~5sAuto-caption SRT from audio (planned).
caption-cuesheet-build<1sPer-platform teaser captions from one master (planned).
Image / Effects
hyperframes~2sHyperframe stylization passes.
obscura~2sObscura-style image effects pack.
rife-interpolate~8sRIFE frame interpolation 2x/4x (planned).

MCP Bridges (External services)

Third-party services proxied through the Livepeer MCP bridge. Same envelope, same auth path — first call surfaces an OAuth link, subsequent calls succeed silently. The agent uses these to finish the loop: published, scheduled, delivered. No custom client SDKs to maintain.

Live today
slack-send-fileOAuthPost asset to a Slack channel with caption.
drive-uploadOAuthPush to Google Drive folder (creates folder if missing).
linear-create-issueOAuthFile a Linear issue with the campaign asset attached.
github-create-prOAuthOpen a PR with generated assets staged.
github-issue-commentOAuthComment on an existing issue with a generated mockup.
notion-create-pageOAuthDrop a launch brief into a Notion DB.
Coming in Splash 3 (marketing)
publora-scheduleOAuthSchedule a post across TikTok / IG / YouTube / X / LinkedIn in one call.
publora-analyticsOAuthEngagement + reach per scheduled post.
brandfetch-fetchAPI keyPull official palette / fonts / logo from a brand's domain.
figma-exportOAuthHand off final carousel to a Figma file.
frameio-uploadOAuthPush final cut to Frame.io for review.
meta-graph-insightsOAuthDirect Meta Graph analytics (Publora fallback).
tiktok-businessOAuthDirect TikTok Business API for analytics + boost.
Roadmap
salesforce-queryOAuthPull customer segments for per-segment campaigns.
hubspot-contactsOAuthAudience pull for marketing automation.
discord-postOAuthDrop generated assets into Discord channels.
buffer-scheduleOAuthBuffer scheduling as Publora alternative.
unity-asset-importAPI keyDrop .glb + spritesheet into a Unity project (Splash 2).
bynder-publishOAuthDAM integration for enterprise brands.
patreon-postOAuthPaid-tier delivery for artist creators.
artstation-publishOAuthPortfolio publishing for artists (Splash 4).
Want a deeper dive into how this all fits together?
Read the full architecture page — control flow, payment tickets, and the extension guide.