Skills System — Token Economy Benchmark

Configurations

Scenario A · thin brief → quality output

User types a short prompt like “12 bear scenes catching salmon, Ghibli style.” Without the SDK they must manually orchestrate plan → develop → anchor → optimize, with realistic 1.7× iteration, before any render. The agent SDK collapses the entire pipeline into a single cached agent context.

Scenario B · pre-detailed prompts → render only

User has done the scene authoring themselves and provides finished per-scene render prompts. The SDK still saves on architecture overhead (single static prefix + tool schemas registered once + cache discipline). The wow story lives in scenarios A and C.

Scenario C · fragmented APIs (40-model catalog)

Same thin brief as A, BUT the user must integrate against ~40 separate model providers (fal, gemini, runway, …) without a unified SDK. Per-scene MODEL-PICKER round (all 40 schemas pasted in context) + PARAM-MAPPING round (translate scene → chosen-provider schema), iterated 1.5× because picks fail and params get rejected. The unified SDK collapses this into one normalized capability registry.

Scenario C · fragmented APIs (100-model catalog)

The next bar — 100-provider saturated marketplace. Per-scene model-picker has 2.5× more provider schemas to scan. This is where the wow saving lives: complex long-form content from a thin prompt, against a fragmented model marketplace.

🎬 Scenario D · Professional 4-step workflow

Real professional creators don't ship single text-to-image renders. The canonical pro workflow per scene is a 4-step chain that fans out across 4 different modalities, each requiring a distinct model and parameter mapping:

1. Text → Image — generate the hero shot / key frame for the scene (prompt + style + character lock)
2. Image → Video — animate the key frame into a motion clip via i2v (motion prompt + duration + dimensions match)
3. Text → Audio — TTS dialogue or music score for the scene
4. Video + Audio → Final asset — sync, format-coerce, and compose the deliverable scene clip

Without a unified SDK, EACH step needs its own model-picker round (different modality-aware schemas) + param-mapper round + URL plumbing between providers. With the SDK, it's 4 tool calls in one cached agent context.

Scenario D · minimal 4-step pro workflow (40-model catalog)

The ship-something baseline: t2i → i2v → t2a → mix. 4 model calls per scene, each requiring its own picker + mapper round in C1 plus chain-reasoning rounds for inter-step URL plumbing.

Scenario D · minimal 4-step pro workflow (100-model catalog)

Same 4-step minimal pipeline against the saturated 100-provider marketplace.

🎯 Scenario E · Typical 7-step pro workflow

What real production teams actually ship — voice + score + color consistency + upscale on top of the minimal D pipeline:

1. Text → Image — hero shot / key frame
2. Image → Upscaled image — sharpen for clean i2v input
3. Image → Video — animate the key frame
4. Video → Color-graded video — palette consistency across scenes
5. Text → Voice — TTS dialogue
6. Text → Music — background score
7. Video + Voice + Music → Final asset — sync, format, compose

7 model calls per scene. Each step needs its own model-picker round (different modality-aware schemas) and param-mapper round in C1, plus URL plumbing between steps. The agent SDK absorbs all of it as 7 cheap tool calls in one cached conversation context.

Scenario E · typical 7-step pro workflow (40-model catalog)

What real production teams ship at today's catalog size. Voice + score + color grade + upscale + the canonical t2i/i2v/mix backbone.

Scenario E · typical 7-step pro workflow (100-model catalog)

The peak case in the bench — typical pro workflow × saturated 100-provider marketplace. The largest honest synthetic saving.

What you get on top of the savings

• 100% routing accuracy on 50 curated multi-scene briefs across 9 intent categories
• ~70 added tokens per turn for compiled skill rules — well below the 1.5KB hard cap
• Auto-pinned skills survive page reloads via localStorage; user preferences become session-stable
• Self-learning — after enough thumbs-up votes on a pattern not yet covered by an existing skill, the system drafts a new one for you to promote
• Privacy-isolated telemetry — per-bucket cookie auth, hashed blob keys, no cross-user mixing

Methodology — Scenario A (thin)

C1 user runs four phases per project before render: Planner (1 call), Scene Developer (N calls reading brief + style + ALL prior scene full texts), Anchor Checker (N calls), Prompt Optimizer (N calls). 1.7× iteration multiplier. Output sized realistically (1000-1500 chars per scene from a quality LLM).

Methodology — Scenario C (fragmented)

Same as A, plus per-scene model-picker round (all provider schemas pasted in context: 40 or 100 × ~500 chars each) and param-mapping round (translate scene → chosen-provider schema). Both rounds iterate 1.5× because real users retry when picks fail or params get rejected.

Methodology — Scenario D (multistep)

Inherits Scenario A pipeline + Scenario C's fragmented overhead, then fans out per scene across 4 steps. Each step pays its own picker (full catalog scan) + mapper, with step-specific extra context (i2v needs image dimension constraints, mix needs sync requirements). Plus 4 chain-reasoning rounds per scene for inter-step URL plumbing (3 transitions + 1 extra coordination round because mix consumes BOTH step 2 and step 3 outputs). Picker/mapper iterate at 1.5×; chain reasoning at 2× (chain failures dominate retries — format mismatches and codec issues across providers). C2/C3 absorb all of this via tool chaining in one cached agent loop.

Both scenarios — C2 and C3

Single agent loop. Static prefix (system + tool schemas + style guide) cached at 90% rate after turn 0. C3 adds a ~360-char compiled-rules block per turn that doesn't cache; trade-off is marginal cost per turn vs better routing → fewer retries.

Sources: plan §6 · bench README · token-model.ts · demo walkthrough