Overall Architecture
Three consumer surfaces (CLI / MCP / Embedded) call the same SDK service, which routes through one of two orchestrator pools (BYOC for capabilities, Scope for live video-to-video), backed by a signer that issues payment tickets on Arbitrum. Every paid call is authenticated by the user's Daydream bearer; every capability — AI, deterministic tool, or MCP-bridged external service — speaks the same wire envelope.
Consumers
─────────
CLI (livepeer) MCP client (Claude Code / Cursor) Embedded SDK (your app)
│ │ │
│ Bearer: sk_<user-daydream-key> │
│ Wire: POST /inference { capability, input } → { output } │
└──────────────┬───────────┴────────────────────┬──────────────┘
│ │
▼ ▼
┌───────────────────┐ ┌──────────────────────┐
│ SDK service │ │ Storyboard webapp │
│ sdk.daydream │ │ /api/llm/chat │
│ .monster │ │ /api/mcp/sse │
│ │ │ /api/preview │
│ FastAPI wrapping │ │ Hermes (long tasks) │
│ livepeer-python │ └─────────┬────────────┘
│ gateway │ │
└────────┬──────────┘ │
│ payment tickets │ same SDK,
▼ │ user bearer
┌──────────────────┐ │
│ Signer service │ │
│ signer.daydream │ │
│ .live │ │
│ Arbitrum │ │
└────────┬─────────┘ │
│ verify tickets │
▼ ▼
┌─────────────────────────────────────────────────────────────┐
│ Orchestrator network (go-livepeer, decentralized) │
│ │
│ BYOC orchs Scope orchs │
│ byoc-staging-* orch-staging-* │
│ │
│ AI + Tool + MCP Live video-to-video │
│ capability dispatch per-second tickets │
└────────────────────┬────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ Adapters — all speak the SAME /inference envelope │
│ │
│ AI : serverless → fal.ai / Gemini │
│ TOOL : Docker → ffmpeg / opencv / yolo │
│ MCP : MCP bridge → Slack / Drive / Linear / GitHub │
└─────────────────────────────────────────────────────────────┘One envelope, three kinds of capability
The architectural decision that makes the rest of this work: every capability — whether it's an AI model on fal.ai, a Docker container running ffmpeg, or a bridge to an external MCP server — is invoked the same way and paid for the same way.
POST /inference
{
"capability": "<name registered on BYOC>",
"input": { ... capability-specific schema ... }
}
Headers:
Authorization: Bearer sk_<user-daydream-key>
Response:
{
"output": { "url": "...", "ok": true, "elapsed_ms": 2317 }
}BYOC — Bring Your Own Capability
The BYOC orchestrator is go-livepeerrunning in BYOC mode. On startup it reads CAPABILITIES_JSONfrom its environment and auto-registers each entry with the network. The same orchestrator dispatches all three kinds of capability — AI, deterministic tool, MCP — to the appropriate adapter behind it.
The unified adapter dispatch
# /opt/byoc/.env (truncated for readability)
CAPABILITIES_JSON=[
# AI capability — adapter calls fal.ai with FAL_KEY (server-side)
{ "name": "flux-dev", "kind": "ai",
"model_id": "fal-ai/flux/dev", "capacity": 4, "price_per_unit": 3 },
# AI capability via Gemini directly (for LLM completions)
{ "name": "gemini-text", "kind": "ai",
"model_id": "gemini/gemini-2.5-flash", "capacity": 8, "price_per_unit": 1 },
# Tool capability — adapter spawns a Docker container
{ "name": "ffmpeg-concat", "kind": "tool",
"image": "livepeer/ffmpeg-concat:1.4", "capacity": 8, "price_per_unit": 1 },
{ "name": "yolo-segment", "kind": "tool",
"image": "livepeer/yolo-segment:0.9", "capacity": 4, "price_per_unit": 1 },
# MCP capability — adapter is the MCP bridge; calls external service
{ "name": "slack-send-file", "kind": "mcp",
"mcp_server": "slack", "capacity": 16, "price_per_unit": 1 },
{ "name": "drive-upload", "kind": "mcp",
"mcp_server": "google-drive", "capacity": 16, "price_per_unit": 1 }
]The orchestrator doesn't branch on kind for dispatch — it routes by name to an adapter that knows what to do. kind is metadata for grouping, SLA, and the UI. This is the central design choice that lets generation + finishing + publishing live under one contract.
AI Capability Flow — fal.ai / Gemini
When the agent calls flux-dev (or any AI capability), the BYOC adapter forwards to a serverless function that holds the upstream provider key (FAL_KEY, GEMINI_API_KEY). The user never sees those keys; they pay through the Daydream bearer.
Client BYOC orch AI adapter
────── ───────── ───────────
POST /inference ─── ticket ────▶
{ capability: "flux-dev", verify ticket ────▶
input: { prompt: "...", on Arbitrum
image_size: "..." } }
resolve capability
"flux-dev" → ai adapter
────▶
POST fal.ai
model=fal-ai/flux/dev
key=FAL_KEY (server)
◀────
{ images: [{url}], ... }
normalize envelope ◀────
◀───── { output } ──
{ url: "https://v3b.fal.media/..." }Why this is decoupled from upstream provider
Tomorrow we swap flux-dev to point at replicate.com/flux/dev or a self-hosted GPU. The capability name stays. Every client — CLI, MCP, webapp, third-party apps — continues working without a release. That's the substrate guarantee.
Tool Capability Flow — Self-hosted Docker
Deterministic tools (ffmpeg, OpenCV, YOLO) run as Docker containers on the orchestrator hardware itself — no external SaaS, no per-call vendor fee. The adapter for these capabilities spawns a one-shot container, mounts the input URLs as files, runs the deterministic transform, and returns the output URL.
Client BYOC orch Tool adapter
────── ───────── ───────────────
POST /inference ─── ticket ───▶
{ capability: "ffmpeg-concat", verify ticket
input: { ─────▶
clip_urls: [ docker run --rm
"https://.../c1.mp4", livepeer/ffmpeg-concat:1.4
"https://.../c2.mp4", --input "<urls>"
"https://.../c3.mp4" --transition cut
], ────────────────────
transition: "cut" (downloads inputs,
} runs ffmpeg, uploads
} result to fal.media
or Vercel blob)
◀──── { output: { url: ... } }
◀──── { output } ────
{ url: "https://.../stitched.mp4" }Why self-hosted matters
Tool capabilities are fast, cheap, predictable. ffmpeg-concat is <5 s for typical 12-clip jobs. No upstream API rate limits, no billing surprises, no "are they having an outage today." For the agent loop, this means finishing-pass code (concat → mux → audio-mix → export) ships as confidently as generation does.
The same Docker container runs locally too — developers can debug a tool capability without hitting the network at all by invoking the image directly. That symmetry makes tool capabilities the most maintainable layer.
MCP Capability Flow — External Services
The third capability kind bridges to external SaaS via their MCP servers (Slack, Drive, Linear, GitHub today; TikTok, Figma, Frame.io, Publora coming in Splash 3). The BYOC adapter for MCP capabilities holds OAuth tokens per-user and forwards each call to the target MCP server using the user's identity.
Client BYOC orch MCP bridge External MCP
────── ───────── ────────── ────────────
POST /inference ── ticket ──▶
{ capability: verify ticket
"slack-send-file",
input: { ─────▶
channel: "#creative", resolve user
url: "...mp4", OAuth token for
title: "Q1 launch reel" channel "#creative"
}
} ─────────────────▶ invokeTool("send_file",
channel, file,
title)
◀────────
{ ok: true,
timestamp: ... }
◀───────────────── envelope normalize
◀──── { output: { ok: true } } ──First-call OAuth flow
When a user calls an MCP-bridged capability for the first time, the bridge returns a 401-like envelope with an OAuth URL. The client surfaces it ("Click to connect Slack"), the user completes OAuth in their browser, the token is stored against their Daydream bearer. Subsequent calls succeed silently. No app-side OAuth code.
On-Chain Routing and Payment Tickets
Every paid call carries a payment ticket signed by the signer service. The ticket commits to a small ETH amount (the call's cost) and is verified on Arbitrum by the orchestrator before the work is dispatched. This is what makes the network decentralized — orchestrators compete on price and reliability, and verifiable on-chain payment means no single vendor can hold the network hostage.
Stage 1 — Identity resolution
─────────────────────────────
Client request carries Authorization: Bearer sk_<user-daydream-key>
│
▼
SDK service forwards to Daydream API:
GET /api/v1/keys/info → { user_id: "user_2x...", balance_eth: 0.18 }
│
▼
SDK has the user's Clerk identity and wallet address.
Stage 2 — Ticket generation
───────────────────────────
SDK service forwards the bearer to signer.daydream.live:
POST /generate-orch-ticket
{ user_id, capability, est_cost_wei }
│
▼
Signer signs the ticket with the user's wallet:
{ ticket_id, signer_sig, expires_at, max_face_value_eth }
│
▼
SDK appends the signed ticket to the /inference request.
Stage 3 — Orchestrator verification (the orchestrator's job)
─────────────────────────────────────────────────────────────
go-livepeer on the BYOC orch receives the request:
1. Parses the ticket.
2. Calls Arbitrum to verify signer_sig + balance via Livepeer's
payment contract.
3. Records the spend on the orch's books.
4. If valid → dispatches to the capability adapter.
5. If invalid → 402 Payment Required (rare; usually means
the user's balance dropped between key check and dispatch).
Stage 4 — Settlement
────────────────────
Periodically the orchestrator submits batched tickets to the
payment contract. The contract pays the orch in ETH and debits
the user's balance. On-chain audit trail per call.Why this matters operationally
For the user: their Daydream key is a single bearer that bills per-call. For developers building on top: there's no Stripe to integrate, no usage-metering code to write, no chargeback risk to handle. The chain is the billing system.
For the network: orchestrators can be added or removed at any time. They bid for work via discovery; the cheapest valid orch wins. No single-vendor outage can take the network down. No price negotiation changes the protocol.
Control Flow vs Data Flow
A common question: "When the orchestrator dispatches an AI call to fal.ai, do the model weights pass through the orchestrator?" No — control and data are deliberately separated. The orchestrator handles control (which capability, which adapter, was the ticket valid). The adapter handles the data (the actual API call to fal, the actual ffmpeg run, the actual Slack post).
CONTROL FLOW (small payloads, on-chain)
────────────────────────────────────────
Client → SDK service → Signer → SDK service
(ticket request, ~200 bytes)
Client → SDK service → BYOC orch
(capability name, parameters, ticket; ~1-4 KB)
BYOC orch → Arbitrum
(ticket verification, ~500 bytes on chain; reads only)
BYOC orch → Adapter dispatch
(capability name, input JSON; ~1-4 KB)
DATA FLOW (large payloads, direct CDN)
──────────────────────────────────────
Adapter (ai/tool/mcp) → external service
(fal.ai API call: maybe a source_url; response with output URL)
Adapter response → SDK service
(output URL only; ~200 bytes — NOT the image/video bytes)
Client fal.media CDN
◀──────── fetch output URL ──────────▶ (images, videos, audio
served directly via CDN,
never through SDK or orch)The implications
The orchestrator never proxies media bytes. The SDK service handles only envelopes (JSON in, JSON out). Generated assets live on the fal CDN (or Vercel Blob for tool outputs) and clients fetch them directly. This is what makes the network scale: a 12-scene generation is 12 tickets + 12 control roundtrips, not 12 large file transfers through the SDK.
The corollary: the SDK service can run on a small VM (and does — a single uvicorn worker with async I/O handles arbitrary concurrent load because every blocking call is wrapped in asyncio.to_thread). The decentralized orchestrator pool handles compute scale; the SDK handles control-plane only.
Extending the Network — Adding a Capability
Three patterns, by capability kind. All three are config-only — no orchestrator code change, no client SDK release.
① Add an AI capability (the most common)
# 1. Verify the upstream model exists. fal renames models; always check:
# curl https://fal.ai/models/fal-ai/some/new-model
# 2. SSH to the BYOC VM and edit CAPABILITIES_JSON:
gcloud compute ssh byoc-staging-1 --zone=us-west1-b \
--project=livepeer-simple-infra
sudo python3 -c "
import json, re
with open('/opt/byoc/.env') as f: content = f.read()
for line in content.splitlines():
if line.startswith('CAPABILITIES_JSON='):
caps = json.loads(line.split('=', 1)[1])
break
caps.append({
'name': 'kling-motion-control',
'kind': 'ai',
'model_id': 'fal-ai/kling-video/v2.6-pro/motion-control',
'capacity': 2,
'price_per_unit': 5,
})
content = re.sub(r'^CAPABILITIES_JSON=.*$',
'CAPABILITIES_JSON=' + json.dumps(caps),
content, flags=re.MULTILINE)
with open('/opt/byoc/.env', 'w') as f: f.write(content)
print(f'Wrote {len(caps)} capabilities')
"
# 3. Restart the adapter (down && up — NOT restart):
sudo bash -c 'cd /opt/byoc && docker compose down && docker compose up -d'
# 4. Verify registration:
sudo docker logs byoc-adapter --tail 10
# Look for: "Registered capability 'kling-motion-control'"
curl -s https://sdk.daydream.monster/capabilities \
| python3 -c "import sys,json; \
[print(c['name']) for c in json.load(sys.stdin) if 'kling-motion' in c['name']]"That's it. The new capability is callable from CLI, MCP, and webapp immediately. No client release. list_capabilities surfaces it.
② Add a Tool capability (Docker-backed)
# 1. Build a Docker image that speaks the /inference envelope:
# Dockerfile expects input on stdin as JSON, writes output JSON on stdout.
FROM debian:bookworm-slim
RUN apt-get update && apt-get install -y python3 ffmpeg && rm -rf /var/lib/apt/lists/*
COPY ./tool.py /tool.py
ENTRYPOINT ["python3", "/tool.py"]
# tool.py reads stdin: { "input": {...} }, writes stdout: { "output": {...} }
# 2. Push the image to the registry:
docker build -t us-docker.pkg.dev/livepeer/tools/my-tool:1.0 .
docker push us-docker.pkg.dev/livepeer/tools/my-tool:1.0
# 3. Register it in CAPABILITIES_JSON exactly like an AI capability,
# but with kind=tool and an image field:
{
"name": "my-tool",
"kind": "tool",
"image": "us-docker.pkg.dev/livepeer/tools/my-tool:1.0",
"capacity": 8,
"price_per_unit": 1
}
# 4. Same down/up + verify as above.The tool runs on the orchestrator hardware. No external API key needed. Cost is just compute (which is why tools are price_per_unit=1 — they dominate the "fast and cheap finishing" tier).
③ Add an MCP-bridged capability
# 1. Register the external MCP server URL + auth flow with the
# Livepeer MCP bridge:
sudo vim /opt/mcp-bridge/config.yaml
mcp_servers:
- name: figma
transport: sse
url: https://figma.com/api/mcp
auth:
type: oauth2
client_id: ${FIGMA_CLIENT_ID}
client_secret: ${FIGMA_CLIENT_SECRET}
scopes: ["files:read", "files:write"]
# 2. Restart the MCP bridge so it discovers the tools the new server
# exposes:
sudo systemctl restart mcp-bridge
# 3. Register each tool you want to expose as a Livepeer capability
# in CAPABILITIES_JSON:
{
"name": "figma-export",
"kind": "mcp",
"mcp_server": "figma",
"mcp_tool": "export_file",
"capacity": 16,
"price_per_unit": 1
}
# 4. Same down/up + verify. First call from a user will surface the
# OAuth URL; subsequent calls succeed silently.What you DON'T have to do
Notably absent from every "add a capability" recipe:
- No orchestrator code change.
go-livepeeris generic. - No SDK service code change.
/inferenceis generic. - No client SDK release.
@livepeer/agentis generic. - No MCP server release. The Storyboard MCP exposes
list_capabilitiesdynamically. - No webapp deploy. Capabilities surface in the model picker via
/capabilities.
A new model joining the network is a 5-minute ops task. That tempo — new SOTA models landing on the network within minutes of going live on fal — is the compounding advantage that makes Livepeer the most comprehensive media network available.