For engineers · adding to the network

Add a new provider — AI, tool, or MCP

Three integration paths, one wire format. Pick the path that matches what you're adding. Each path is additive — no orch code change, no SDK change. The hardest case (a new AI provider) is ~150 lines of Python plus one JSON entry.

The mental model

A capability is one row in CAPABILITIES_JSON on the BYOC orchestrator's adapter. It declares a name, a container + endpoint to route to, a price, and a kind (ai, tool, or mcp). The container at that endpoint speaks the same /inference envelope as every other capability. That's it.

What you don't touch: go-livepeer, payment tickets, the signer, the agent, the SDK, the MCP server, the browser. Those are all generic.

§1The three paths

Kind	When to use	What you build
ai	You want to expose a model behind a paid API (fal, OpenAI, Replicate, Runway, Moonshot, Anthropic, …)	A Python provider class in the serverless-proxy that maps the standard `/inference` envelope into that vendor's HTTP API. Plus one JSON capability entry per model.
tool	You want a deterministic operation (ffmpeg op, OpenCV op, headless-browser scrape, image filter)	A small Docker container exposing an HTTP endpoint that takes JSON in and JSON out. Plus one JSON capability entry per endpoint.
mcp	You want to bridge an existing third-party MCP server (Slack, Drive, Linear, Notion, internal corp tools) so the agent can call it through the same pay-per-call rail	Nothing to build. Add one JSON capability entry pointing at the upstream MCP server and tool name. The tool-adapter's `mcp_bridge.py` handles the rest.

§2The zero-code case — a provider that's already wired

Before you write a new provider, check whether someone else already exposes the workflow you want. ComfyUI is the textbook example:

fal.ai hosts most of the popular ComfyUI workflows behind their own model endpoints (fal-ai/instant-id, fal-ai/pulid, fal-ai/ip-adapter-face, fal-ai/photomaker, fal-ai/illusion-diffusion, fal-ai/controlnet-canny, fal-ai/codeformer, fal-ai/ccsr, fal-ai/animatediff). Storyboard already has the FalAiProvider. The integration was nine JSON entries in CAPABILITIES_JSON and one skill markdown:

# /opt/byoc/.env on byoc-staging-1 — one entry per ComfyUI cap
{
  "name": "instant-id",
  "container": "serverless-proxy",
  "endpoint": "/inference",
  "capacity": 3,
  "price_per_unit": 4,
  "kind": "ai",
  "upstream_provider": "fal-ai",
  "model_id": "fal-ai/instant-id"
}

Restart the adapter (docker compose down && up -d), the orch re-registers, and within seconds the storyboard MCP advertises instant-id as a callable capability. No new provider code, no SDK change. If your upstream is already a wired provider, you're a config edit away. Always check this first.

Sanity check before writing code: does fal, Replicate, or RunPod already host the model you want? They host thousands. Grep storyboard-a3/lib/sdk/capabilities.ts for prior art, then check fal.ai/models / replicate.com/explore. The integration cost difference is "5 minutes of JSON" vs "a week of provider work."

§3New AI provider — worked example: Moonshot

You actually need a new provider when your upstream isn't already wired. Worked example: Moonshot AI (Kimi). Same path applies to OpenAI, Anthropic, xAI, Mistral, any HTTP-based model API.

Architecture

         agent → SDK service → BYOC orch ───▶ inference-adapter
                                                   │
                                                   ▼
                                          serverless-proxy
                                                   │
                                                   ▼
                                            MoonshotProvider   ← you write this
                                                   │
                                                   ▼
                                          api.moonshot.cn      ← Moonshot API

Step 1 — write the provider class

Path: NaaP/containers/livepeer-serverless-proxy/src/serverless_proxy/providers/moonshot.py

Subclass InferenceProvider (see providers/base.py). Implement two methods:

from __future__ import annotations
import aiohttp
from .base import InferenceProvider
from typing import Optional

class MoonshotProvider(InferenceProvider):
    """Forward inference to api.moonshot.cn (OpenAI-compatible)."""

    BASE_URL = "https://api.moonshot.cn/v1"

    def __init__(self, api_key: str, model_id: Optional[str] = None):
        self._api_key = api_key
        self._default_model_id = model_id

    async def health(self) -> bool:
        try:
            async with aiohttp.ClientSession() as session:
                async with session.get(
                    f"{self.BASE_URL}/models",
                    headers={"Authorization": f"Bearer {self._api_key}"},
                    timeout=aiohttp.ClientTimeout(total=5),
                ) as resp:
                    return resp.status == 200
        except Exception:
            return False

    async def inference(self, request_body: dict, session: aiohttp.ClientSession,
                        model_id: Optional[str] = None) -> dict:
        # Resolve the model to call
        model = model_id or self._default_model_id
        if not model:
            raise ValueError("No model_id provided")

        # Map the standard /inference envelope into the vendor's shape.
        # Moonshot is OpenAI-compatible; the envelope's `prompt` becomes a user msg.
        prompt = request_body.get("prompt", "")
        params = request_body.get("params", {})
        payload = {
            "model": model,
            "messages": [{"role": "user", "content": prompt}],
            "temperature": params.get("temperature", 0.7),
            "max_tokens": params.get("max_tokens", 2048),
        }

        async with session.post(
            f"{self.BASE_URL}/chat/completions",
            json=payload,
            headers={
                "Authorization": f"Bearer {self._api_key}",
                "Content-Type": "application/json",
            },
            timeout=aiohttp.ClientTimeout(total=120),
        ) as resp:
            data = await resp.json()
            if resp.status != 200:
                return {"error": data.get("error", str(data))[:200]}

            # Map the response back into the envelope. Text models surface
            # `text`; image / video models would surface `url`.
            return {
                "text": data["choices"][0]["message"]["content"],
                "model": model,
                "usage": data.get("usage"),
            }

Step 2 — register the provider in the factory

Path: NaaP/containers/livepeer-serverless-proxy/src/serverless_proxy/server.py

# in create_provider():
if config.provider == "moonshot":
    return MoonshotProvider(api_key=config.api_key, model_id=config.model_id)

# and in create_extra_providers() so multi-provider mode picks it up:
elif name == "moonshot":
    extra[name] = MoonshotProvider(api_key=cfg["api_key"])

Step 3 — add the capability rows

Path: /opt/byoc/.env on byoc-staging-1. Same Python helper pattern as adding any other capability (don't sed/awk JSON — see CLAUDE.md):

{
  "name": "moonshot-k2",
  "container": "serverless-proxy",
  "endpoint": "/inference",
  "capacity": 4,
  "price_per_unit": 2,
  "kind": "ai",
  "unit_kind": "1k-chars",
  "display_price_usd": 0.012,
  "upstream_provider": "moonshot",
  "model_id": "moonshot-v1-32k"
}

Step 4 — deploy

# In NaaP/containers/livepeer-serverless-proxy:
docker build -t serverless-proxy:moonshot .
docker push us-docker.pkg.dev/livepeer-simple-infra/simple-infra/serverless-proxy:moonshot

# On byoc-staging-1:
gcloud compute ssh byoc-staging-1 --zone=us-west1-b --project=livepeer-simple-infra
sudo docker pull us-docker.pkg.dev/livepeer-simple-infra/simple-infra/serverless-proxy:moonshot
cd /opt/byoc && sudo docker compose down && sudo docker compose up -d

# Verify the registration
sudo docker logs byoc-adapter --tail 10  # should see "Registered capability 'moonshot-k2'"
curl -s https://sdk.daydream.monster/capabilities | grep moonshot

Step 5 — wire into storyboard (optional)

Add to the keyword resolver + fallback chain so the agent picks the capability naturally. Path: storyboard-a3/lib/sdk/capabilities.ts:

// FALLBACK_CAPABILITIES — extend the known cap set
"moonshot-k2",

// FALLBACK_CHAINS — pick siblings if moonshot is down
"moonshot-k2": ["gemini-text", "claude-sonnet"],

Build, deploy storyboard-a3 (Vercel auto-deploys on merge to main), and the agent will route to moonshot-k2 for matching queries.

Total work

Step	File / system	Lines / time
1	`providers/moonshot.py`	~80 lines
2	`server.py` factory	4 lines
3	`CAPABILITIES_JSON` on the VM	1 JSON object per model
4	Docker build + push + adapter restart	5–10 min
5 (optional)	`lib/sdk/capabilities.ts`	2 lines

§4New deterministic tool — Docker container pattern

Use this when you need a deterministic operation: an ffmpeg op, an OpenCV op, a Pillow filter, a Rust CLI wrapped in HTTP. Reference implementations in simple-infra/tool-host-build/:

tool-ffmpeg — concat, mux, audio-mix, overlay, export, trim, loop, burn-subtitles, grid
tool-obscura — headless-browser web extract (Rust CLI in Docker)
tool-yolo — YOLO detect + segment
tool-hyperframes — HTML→video composition (caption, lower-third)

Container contract

One HTTP endpoint per operation. Input + output are JSON. Sample worker (Python with FastAPI):

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel

app = FastAPI()

class ConcatReq(BaseModel):
    clips: list[str]            # input URLs
    transition: str = "cut"     # cut | crossfade-300 | fade-black

@app.post("/run/ffmpeg-concat")
async def concat(req: ConcatReq):
    out_url = await run_ffmpeg_concat(req.clips, req.transition)
    return {"url": out_url, "clips": len(req.clips)}

Endpoint name (here /run/ffmpeg-concat) is what you put in the capability's endpoint field. The container name (the docker-compose service name, e.g. ffmpeg) goes in container.

The capability row

{
  "name": "ffmpeg-concat",
  "container": "ffmpeg",
  "endpoint": "/run/ffmpeg-concat",
  "capacity": 8,
  "price_per_unit": 1,
  "kind": "tool",
  "unit_kind": "call",
  "display_price_usd": 0.002
}

Deploy steps are identical to the AI-provider path: docker build + push, restart the adapter, verify in /capabilities.

SSRF + abuse guards

If your tool takes a URL as input (like obscura-* or any image / video op), the tool-adapter's URL normalizer rejects RFC 1918 / link-local / cloud-metadata IPs at the network boundary. You don't need to re-implement it — but you also can't trust input URLs that arrive through other channels. See tool-adapter/proxy.py for the live guards.

§5New external service — MCP bridge (no code)

Want to expose an existing third-party MCP server (Slack, Drive, Linear, Notion, Publora, an internal corp service) to agents via the same pay-per-call rail? You write zero code. The tool-adapter/mcp_bridge.py already proxies any MCP server that speaks the standard protocol.

The capability row

{
  "name": "slack-send-file",
  "container": "tool-adapter",
  "endpoint": "/mcp",
  "capacity": 8,
  "price_per_unit": 1,
  "kind": "mcp",
  "unit_kind": "call",
  "display_price_usd": 0.005,
  "mcp_server": "https://mcp.slack.com",
  "mcp_tool": "send_file",
  "auth_strategy": "bearer",
  "auth_header": "Authorization"
}

The bridge resolves the auth token from the bearer's per-user MCP-credentials store (OAuth tokens land there via the existing /api/mcp/auth flow). The first call surfaces an OAuth link to the user if they haven't authorized; subsequent calls succeed silently.

What you DON'T do

You do not run the MCP server. It runs on the vendor's infrastructure.
You do not handle the vendor's auth flow. The MCP server does (OAuth, API key, whatever).
You do not write a provider class. mcp_bridge.py is generic.

What you do

Confirm the upstream is a real MCP server (speaks the standard protocol).
Add the JSON capability entry pointing at mcp_server + mcp_tool.
Restart the adapter.
Pick a reasonable price_per_unit + display_price_usd — typically low ($0.001–$0.01 per call) because you're just relaying.

§6The /inference envelope — what every capability speaks

One JSON shape, all three kinds:

// Request — agent / SDK / orch all use this
{
  "capability": "moonshot-k2",
  "prompt": "Summarize this article in three bullets",
  "params": { "temperature": 0.3, "max_tokens": 512 },
  "image_data": "data:image/png;base64,..."     // optional, multimodal
}

// Response — provider returns this shape
{
  "url": "https://fal.media/output.mp4",    // image / video / audio path
  "text": "summary text...",                // text-model path
  "elapsed": 1240,                            // ms
  "model": "moonshot-v1-32k",                // echoed back
  "error": "..."                            // only on failure
}

Your provider's job is to translate this in both directions. The agent and the SDK don't care which kind ran — they just see the response shape.

§7Common mistakes (verbatim from the runbook)

Using docker restart instead of down && up -d — adapter won't re-read the updated .env.
Editing CAPABILITIES_JSON with sed — shell quoting breaks the JSON. Use Python or jq.
Wrong model_id — fal renames models. Curl the model URL on fal.ai before registering.
Forgetting display_price_usd — your cap will work but the storyboard cost-preview shows $0.00 and the agent can't compare to alternatives.
Wire price_per_unit too high — every call rejects with "Insufficient balance for request" because the bearer's signer doesn't carry enough deposit. See the pricing doc for the wei magnitude relationship.
Two BYOC adapters — byoc-staging-1 runs inference-adapter, tool-staging-1 runs tool-adapter. They're different codebases. Editing one ≠ updating both.

§8Where to read next

Pricing & metering & on-chain payment — what price_per_unit actually means, how wei → USD works, where payment tickets fit.
Architecture overview — the four layers, who owns what.
livepeer/storyboard — agent layer source.
Provider reference impls — NaaP/containers/livepeer-serverless-proxy/src/serverless_proxy/providers/{fal_ai,replicate,gemini,runpod,custom}.py
Tool reference impls — simple-infra/tool-host-build/{tool-ffmpeg,tool-yolo,tool-obscura,tool-hyperframes}/