01 — OVERVIEW
One message, four boundaries
Every user prompt crosses four architectural boundaries: the agent framework that decides intent and picks tools, the tool / capability layer that knows how to make media, the SDK edge that meters and signs every call, and the Livepeer orchestration network that actually runs inference on fal.ai / Gemini and streams it back.
≤20
tool-use rounds per turn
2
delivery paths · batch + live
~50ms
multi-scene preprocess (no LLM)
02 — THE STACK
End-to-end layered architecture
Top to bottom is the full lifetime of a request. Each layer only talks to the one below it; the SDK edge is the single choke point where authentication, payment signing and capability metering happen.
Client / App
Agent Framework
Tools & Capability
MCP
SDK Edge
Orchestration
Inference / Chain
① Clients & Surfacesbrowser · CLI · bots · IDE
Storyboard Web
Next.js 16 on Vercel · infinite canvas + chat
components/chat/ChatPanel.tsx
MCP Clients
Cursor · Claude Desktop · livepeer CLI
POST /api/mcp
Bot Surfaces
Telegram · Discord webhooks
app/api/{telegram,discord}
↓ user message + canvas context
② Agent Frameworkintent → plan → tool-use loop
Preprocessor
@-refs, multi-scene extraction, plan short-circuit
lib/agents/preprocessor.ts
Creative Pipeline
classify → validate → execute → evaluate
creative-kit/.../creative-pipeline.ts
Plugin Registry
5 swappable agents · default Livepeer
lib/agents/registry.ts
AgentRunner
tool-use loop, ≤20 rounds, streams events
@livepeer/agent · runner-adapter.ts
↓ executeTool(name, input)
③ Tool & Capability Layer31 browser tools · 83 MCP tools
create_media
the workhorse · generate / restyle / animate / tts
lib/tools/compound-tools.ts
project_*
multi-scene lifecycle on canvas
lib/tools/project-tools.ts
scope_*
live video-to-video control (6 tools)
lib/tools/scope-tools.ts
MCP server tools
generate_project, director_*, brand_kit_*, voice_*
lib/mcp-server/tools/*
Capability Resolver
fuzzy-match LLM names → real model · fallback chains
lib/sdk/capabilities.ts
↓ runInference() · POST /inference · /stream/*
④ SDK Edge — sdk.daydream.monsterFastAPI · the single choke point
/inference
batch AI / tool / mcp dispatch
/stream/*
LV2V start · publish · frame · control · stop
/capabilities
live registry · 60s cache
Auth resolve
Bearer sk_ → daydream_user_id (Clerk)
↓ sign tickets · meter payment
⑤ Auth & PaymentsArbitrum-metered
Signer
signer.daydream.live · sign-orchestrator-info, generate-live-payment, sign-byoc-job
Daydream API
api.daydream.live · validates sk_ key (nginx auth_request)
↓ gRPC + payment ticket
⑥ Livepeer Orchestrationgo-livepeer · BYOC + Scope
BYOC Orch
byoc-staging-1 · batch inference · CAPABILITIES_JSON registry
inference-adapter
:9090 · registers caps, routes by model_id
serverless-proxy
:8080 · provider API translation
Scope Orchs
orch-staging-1/2/3 · live-video-to-video
↓ provider call (server-side keys)
⑦ Inference & Settlementexternal model execution
fal.ai
flux · recraft · ltx · seedance · veo · kontext …
Gemini
gemini-2.5-flash · -flash-image · text + LLM
Scope Runner
wss://fal.run/daydream/scope-livepeer/ws
Arbitrum One
ticket verification & settlement
Results bubble back up the same path: hosted URL (fal.media / Vercel Blob) → canvas card or viewer URL.
03 — AGENT FRAMEWORK
Pluggable agents over a shared tool registry
The framework's core idea: the LLM is swappable, the tools are not. Five agent plugins implement one interface (AgentPlugin.sendMessage()) and all drive the exact same 31-tool registry through a provider-agnostic tool-use loop.
flowchart TB
U["User message + canvas context"]:::client
U --> CI["classifyIntent()
regex · ~0ms"]:::agent
CI --> PP["Preprocessor
@-refs · multi-scene · plan short-circuit"]:::agent
PP -->|"multi-scene brief"| PC["project_create
(direct, no LLM)"]:::tool
PP -->|"compare / batch / style / variations"| CP["Creative Pipeline
classify→validate→execute→evaluate"]:::agent
PP -->|"single / story / passthrough"| REG["Active Plugin
registry.getActivePlugin()"]:::agent
REG --> P1["Livepeer ★default"]:::plugin
REG --> P2["Gemini"]:::plugin
REG --> P3["Claude"]:::plugin
REG --> P4["OpenAI"]:::plugin
REG --> P5["Built-in"]:::plugin
P1 & P2 & P3 & P4 --> AR["AgentRunner.runStream()
tool-use loop · ≤20 rounds"]:::agent
P5 --> ENR["SDK /enrich DAG
parallel waves, no LLM loop"]:::agent
AR --> PICK["pickToolsForIntent()
31 → 8-15 tools/turn"]:::agent
PICK --> EX["executeTool(name, input)"]:::tool
CP --> EX
PC --> EX
ENR --> EX
EX --> SDK["lib/sdk/client → SDK edge"]:::sdk
classDef client fill:#15233f,stroke:#6ea8fe,color:#dce8ff;
classDef agent fill:#241a3a,stroke:#c89bff,color:#ecdfff;
classDef plugin fill:#1d1730,stroke:#a987e0,color:#d8c9f5;
classDef tool fill:#13302c,stroke:#5fd0c0,color:#cdfff6;
classDef sdk fill:#33181f,stroke:#ff7a90,color:#ffd7df;
The 5 agent plugins
| Plugin | LLM | Route |
|---|
| Livepeer ★ | gemini-2.5-flash | /api/llm/chat via Daydream key |
| Gemini | gemini-2.5-flash | /api/agent/gemini |
| Claude | claude-sonnet-4-6 | /api/agent/chat |
| OpenAI | gpt-4o | /api/agent/openai |
| Built-in | none | SDK /enrich + direct |
The default Livepeer agent routes its LLM through the same Daydream key used for inference — no separate provider key needed.
Why preprocess before the LLM?
An 800-word storyboard brief + 21 tool schemas overwhelms the model's token budget and produces empty STOP or MALFORMED_FUNCTION_CALL. The fix is structural, not prompt-engineering:
Detect & extract client-side
Regex pulls scene titles + ≤25-word summaries and a style guide from the brief in <50ms.
LLM never sees the full brief
It receives a 1-line instruction — "project created, call project_generate" — and just drives the loop. Verified on a 20-scene / 8,651-char prompt.
05 — MCP DUAL-SURFACE
The same brain, two front doors
MCP runs in both directions. The app is an MCP client (it consumes external servers like Gmail). It is also an MCP server — POST /api/mcp exposes 83 storyboard tools so Cursor, Claude Desktop and the CLI can render real, hosted media without touching the UI.
flowchart LR
subgraph CLIENTS["MCP client side (app consumes)"]
direction TB
EXT["External MCP servers
Gmail · custom"]:::mcp
MC["lib/mcp/client.ts
tools/list · tools/call"]:::tool
EXT --> MC
end
subgraph SERVER["MCP server side (app exposes)"]
direction TB
CUR["Cursor / Claude Desktop / livepeer CLI"]:::client
API["POST /api/mcp
Streamable HTTP · maxDuration 800s"]:::mcp
SRV["lib/mcp-server/server.ts
83 tools"]:::tool
AI["auto-injection
project voice + LoRA"]:::tool
BLOB["Vercel Blob
projects · jobs · voices · LoRAs"]:::sdk
CUR --> API --> SRV --> AI
SRV --> BLOB
end
MC --> AG["Agent tool-use loop"]:::agent
AI --> CALL["sdk-call.ts (sdkPost)"]:::sdk
AG --> CALL
CALL --> EDGE["SDK edge → BYOC → fal / Gemini"]:::sdk
EDGE --> VURL["viewer_url
storyboard.daydream.monster/v/proj_*"]:::ok
classDef client fill:#15233f,stroke:#6ea8fe,color:#dce8ff;
classDef mcp fill:#2a2410,stroke:#ffb454,color:#ffe6bf;
classDef tool fill:#13302c,stroke:#5fd0c0,color:#cdfff6;
classDef agent fill:#241a3a,stroke:#c89bff,color:#ecdfff;
classDef sdk fill:#33181f,stroke:#ff7a90,color:#ffd7df;
classDef ok fill:#1d2a16,stroke:#8ee37a,color:#d7f6c8;
Stateless & durable
The MCP create_media is independent of the browser — it persists jobs to Vercel Blob and returns a viewer_url the user can open.
tools/create-media.ts
Auto-injection
When a project_id is set, attached voices & LoRAs are injected automatically — explicit args always win.
auto-injection.ts
Director tools
6 MCP-only editorial intents — re_render, insert, retime, variant, beat_lock, export — reuse create_media + finishing chains.
tools/director-*.ts
06 — DELIVERY FLOWS
Two metered paths to a pixel
Both flows pass through the same SDK edge and signer, but split at the orchestrator: batch inference hits the BYOC orch, live video hits the Scope orch fleet. Every call is payment-signed against Arbitrum before a model runs.
A · Batch inference — image / video / TTS / finishing tools
sequenceDiagram
autonumber
participant B as Browser / MCP
participant S as SDK edge
participant SG as Signer
participant DD as Daydream API
participant O as BYOC orch
participant AD as adapter + proxy
participant P as fal.ai / Gemini
B->>S: POST /inference (Bearer sk_…)
S->>DD: resolve user → daydream_user_id
S->>SG: sign-orchestrator-info / sign-byoc-job
SG->>DD: HEAD /v1/streams (validate key)
SG-->>S: address + signature
S->>O: gRPC + Livepeer-Payment ticket
O->>AD: /inference/{capability}
AD->>P: provider call (server-side key, by model_id)
P-->>B: hosted URL bubbles back up the stack
B · Live video-to-video — Scope streaming
sequenceDiagram
autonumber
participant B as Browser
participant S as SDK edge (passthrough)
participant SG as Signer
participant O as Scope orch (1/2/3)
participant F as fal Scope runner (wss)
B->>S: POST /stream/start {model_id:scope, params:{graph,…}}
S->>SG: sign-orchestrator-info + generate-live-payment
S->>O: start_lv2v (params passed through, not validated)
O->>F: WebSocket wss://fal.run/daydream/scope-livepeer/ws
O-->>S: control_url, events_url, trickle channels
loop ~10 fps
B->>S: POST /stream/{id}/publish?seq=N (input frame)
S->>O: trickle publish → F
B->>S: GET /stream/{id}/frame (output frame)
S->>O: trickle subscribe ← F
end
B->>S: POST /stream/{id}/control (prompt / noise / LoRA)
B->>S: POST /stream/{id}/stop
Why the SDK is a passthrough for Scope. The SDK does not validate Scope graph params — it proxies the full config to the fal runner, which validates against its own schema. Adding new Scope features therefore needs zero SDK changes. The SDK only changes when the transport protocol (channel types) changes.
07 — INFRA TOPOLOGY
Where it physically runs
GCP project livepeer-simple-infra. The web app is on Vercel; everything below the SDK edge is go-livepeer + adapters on staging VMs, defined declaratively in simple-infra/environments/ and provisioned via Pulumi.
| VM | Domain | Runs | Role |
|---|
| sdk-staging-1 | sdk.daydream.monster | sdk-service (FastAPI :8000) + Caddy | SDK edge — inference + LV2V proxy + capabilities |
| byoc-staging-1 | byoc-staging-1…:8935 | byoc-orch + inference-adapter :9090 + serverless-proxy :8080 | Batch inference orchestration |
| orch-staging-1/2/3 | orch-staging-N…:8935 | go-livepeer AI-Serverless + Caddy | Scope LV2V (us-west / us-east / us-central) |
| signer-staging-1/2 | signer.daydream.live | go-livepeer remoteSigner + nginx (HA pair) | Payment ticket signing + key validation |
| tool-staging-1 (optional) | tool-staging-1… | tool-adapter + ffmpeg / yolo containers | Deterministic finishing tools |
Capability source of truth: environments/staging/byoc.yaml → capabilities[], deployed into CAPABILITIES_JSON on the orch. The adapter re-registers each entry on an interval — no go-livepeer code change to add a model.
08 — HOW RESULTS LAND
From raw model output to a deliverable
The architecture is opinionated about the last mile: it doesn't stop at a list of fal.media URLs. It plans a finishing pass, persists durable artifacts, and hands back something the user can actually open or post.
Finishing pass
After AI generation the agent plans a deterministic finish: N clips → ffmpeg-concat, aspect re-frame → ffmpeg-export, brand mark → ffmpeg-overlay, captions, audio mix. Each step's output feeds the next.
Canvas as state
In the web app, every result is a card grouped by batchId from the prompt that made it. Layout modes (/organize grid · narrative) keep a prompt's outputs together. Undo/redo + snapshots persist locally.
Durable viewer URLs
MCP / CLI jobs persist to Vercel Blob and return storyboard.daydream.monster/v/proj_* — a shareable viewer that fills in per-scene URLs as the async job completes.
Async + polling
Long jobs (>4 scenes or any video) run async. Poll GET /api/creative/{id} every 8–15s; status moves running → partial → done, per-scene URLs populate as they land.
Resilient delivery
Capability fallback chains retry failed models with siblings. LV2V streams self-heal: clear 410 Gone on dead streams, a stream reaper, per-stream publish locks, and graceful SDK shutdown.
Humanized errors
Raw failures map to actionable messages — "no GPU available, try again", "can't reach SDK, check key" — and the SDK returns structured rejections with hints (e.g. low broadcaster deposit).
The result-delivery loop, end to end
flowchart LR
G["AI generation
fal / Gemini"]:::infra --> FIN["Finishing pass
ffmpeg / overlay / export"]:::tool
FIN --> PER["Persist
canvas card · Vercel Blob"]:::sdk
PER --> DEL{Delivery surface}:::agent
DEL -->|web| C["Canvas card
batchId grouped"]:::client
DEL -->|MCP / CLI| V["viewer_url
/v/proj_*"]:::ok
DEL -->|publish| PUB["Slack / Drive / Linear
bridge — roadmap"]:::mcp
classDef infra fill:#2a2410,stroke:#f4d35e,color:#fff0c2;
classDef tool fill:#13302c,stroke:#5fd0c0,color:#cdfff6;
classDef sdk fill:#33181f,stroke:#ff7a90,color:#ffd7df;
classDef agent fill:#241a3a,stroke:#c89bff,color:#ecdfff;
classDef client fill:#15233f,stroke:#6ea8fe,color:#dce8ff;
classDef ok fill:#1d2a16,stroke:#8ee37a,color:#d7f6c8;
classDef mcp fill:#2a2410,stroke:#ffb454,color:#ffe6bf;