---
title: "Remove an object from a video — the Vanishing Act"
tier: standard
format: video-inpaint
theme: vfx | video-editing | magic | content
persona: editor, vfx-artist, content-creator, social-media
duration: a short clip + one sentence → the subject erased, background reconstructed, in ~3–6 min
budget_usd: ~$0.30–$0.90 per clip (priced per video-second of inpainting)
caps: ["void-inpaint", "marlin-video", "sam3", "yolo-segment", "ffmpeg-concat"]
skills: ["livepeer-cli"]
showcases:
  - title: "Vanishing Act · a man erased off a city sidewalk (BEFORE | AFTER)"
    href: "/chapters/remove-object-magic-example.html"
status: live — validated end-to-end on a CC0 Pexels clip; subject fully erased, background reconstructed, before|after split ships. See public/chapters/remove-object-magic/.
reliability: 4.1 # 5 − marlin .8 (noisy locate) − void-inpaint .6 (video) + proven showcase .5; ffmpeg recompose deterministic; biggest variable is CLIP SUITABILITY (see works/struggles table), not the caps
---

# Remove an object from a video — the Vanishing Act

Photo object-removal is a solved trick — one frame, one mask, one inpaint. **Video** is the hard version: the subject moves, the camera drifts, the lighting shifts, and the hole you punch has to be filled *consistently across every frame* or it flickers, smears, and ghosts. Do it well and you get a magic effect: a person walks into frame, and the scene behind them was simply always empty. A "ghost street." A "floating object." The impossible shot.

`void-inpaint` is the video-inpainting capability on the Livepeer network — the VOID model (`fal-ai/void-video-inpainting`). You hand it a clip, a one-line description of *what to remove* (`mask_prompt`), and a one-line description of *what the background should be* (`prompt`). It runs SAM-3 internally to track + mask the subject across frames, then a temporally-coherent diffusion pass to reconstruct the background. No frame-by-frame rotoscoping. No After Effects content-aware-fill marathon.

> **`void-inpaint`** accepts `video_url` (required), `prompt` (required — describe the *clean background*), and `mask_prompt` (describe the *subject to remove* — it auto-masks via SAM-3). It also takes `quad_mask_video_url` if you'd rather supply your own mask video, plus `negative_prompt`, `num_inference_steps`, `strength`, `num_frames`, `enable_pass2_refinement`. It returns `{ video: { url } }` — the inpainted clip with the subject gone. LOCAL clip? Host it first: `scripts/host-local-file.sh <path>` -> pass the returned public URL as `video_url` (auto-deletes ~2 days; see CLAUDE.md "GCS ingest").

## When it works — and the honest limits

This is the single most important part of the playbook. Video inpainting is **not** magic-for-everything. It is magic for *one specific shape of problem*.

| Works beautifully | Struggles / don't promise it |
|---|---|
| **One clear foreground subject** (a person, a skateboarder, someone holding a prop) | A crowd, or several overlapping subjects — the mask gets confused |
| **Static or slow-panning camera** (tripod, locked-off, gentle follow) | Fast handheld, whip-pans, big parallax — the background reconstruction drifts |
| **Simple, low-frequency background** (a brick wall, a sky, a road, a plain floor) | Busy, high-detail backgrounds (dense foliage, text, repeating signage) — seams show |
| **Subject occupies a modest part of the frame** | Subject fills 60%+ of frame — there's too little real background to borrow from |
| **Short clips** (3–8s) | Long clips — temporal drift accumulates; cut into shots first |
| **REMOVAL** (erase + reconstruct) | **REPLACEMENT** — void-inpaint removes; it does not swap-in a new object. For replacement, composite a separate generated element on top. |

**Rule of thumb:** if a human editor could plausibly clone-stamp the background in a few frames, video inpainting will nail the whole clip. If even a human would struggle (busy background, fast motion, subject dominates frame), expect a blurry hole or ghosting — and *don't ship it*.

## How the agent should run this (interaction contract)

1. **CONFIRM (one message, ≤1 question):** restate in 1 line ("erase <subject> from your ~8s clip, reconstruct the background, ship clean + BEFORE|AFTER split, ~$0.35–0.85, ~5–10 min") + the real choices: fast draft (`enable_pass2_refinement:false`, ~100s) vs max quality (true, ~5 min) · clean plate only vs + before/after split. If the clip looks unsuitable per the table above (crowd, whip-pan, busy background), say so NOW — don't take the money first.
2. **PREVIEW CHECKPOINT:** run `marlin-video` (~$0.02) to confirm ONE clear subject + static background, then a FAST pass (pass2 off, half the wall-clock) as the draft — only re-run the winner on max quality. Never iterate prompts at 5 min/attempt.
3. **NARRATE:** "SAM-3 mask ~19s → pass-1 ~69s → pass-2 ~197s"; post a 1-liner per phase and poll the job — never silent >2 min on the max-quality run.
4. **FAIL GRACEFULLY:** auto-mask grabs the wrong thing → tighten `mask_prompt` from marlin's own nouns → build a manual mask (`sam3`/`yolo-segment` → `quad_mask_video_url`); ghost/smear in the fill → stronger `negative_prompt` ("person, figure, silhouette, ghost, blur") + pass2 on → downscale to 720p. ≤2 retries; if it still ghosts, the clip failed the gate — say so honestly and suggest a better-suited clip. A janky removal is worse than no removal.
5. **DELIVER:** clean plate + BEFORE|AFTER hstack (the split IS the deliverable) + one honest line ("removal only — swapping in a new object is a separate composite step; verified frames at 0s/3s/6s") + ONE next step ("run the next clip, or composite a replacement element on the clean plate").

## The recipe

```
clip ─→ marlin-video (locate + confirm subject) ─→ void-inpaint (mask_prompt + prompt) ─→ ffmpeg (clean result + BEFORE|AFTER hstack) ─→ GATE (frames verified) ─→ download
                                                          ↑ SAM-3 auto-mask runs inside void-inpaint
```

1. **`marlin-video` — locate + confirm the subject.** Before you spend on the inpaint, confirm what's actually in the clip and where. marlin returns a scene description + timestamped events. You're checking: is there *one* clear subject? Is the background static? This is also where you write your `mask_prompt` ("the man in the green jacket") and `prompt` (the clean background) from marlin's own words.
2. **Produce the removal mask.** The easy path: let `void-inpaint` do it — pass `mask_prompt` and it runs **SAM-3** internally to segment + track the subject across frames. The manual path (more control): run `sam3` or `yolo-segment` yourself to get per-frame masks, encode them into a grayscale mask video, and pass `quad_mask_video_url`. Start with `mask_prompt`; only build a manual mask if the auto-mask grabs the wrong thing.
3. **`void-inpaint` — erase + reconstruct.** Pass `video_url`, `mask_prompt` (subject), `prompt` (clean background), and a `negative_prompt` that bans "person, man, human, figure, silhouette, ghost, blur, smear". Turn on `enable_pass2_refinement` for a cleaner fill. Set `num_frames` to the clip's actual frame count.
4. **`ffmpeg` — make two deliverables.** (a) the clean "vanished" result, and (b) a **BEFORE | AFTER** split via `hstack` (original on the left, vanished on the right) so the magic is undeniable in one glance. (`ffmpeg-concat` / direct ffmpeg both work.)
5. **The GATE.** Extract sample frames from the result and *look*. Is the subject actually gone? Is the background plausible? Any severe flicker, ghost-trail, or smear? **Ship only a clean, magical removal.** If void-inpaint left a blurry hole or a ghost, the clip failed the gate — pick a better-suited clip (static camera, simpler background) or stop and report the limitation honestly. A janky removal is worse than no demo.

## Example `/inference` calls

```bash
SK="$YOUR_DAYDREAM_KEY"   # from ~/.daydream/credentials or your .mcp.json bearer

# 1. LOCATE — confirm the subject + background (cheap, ~$0.02).
curl -s https://sdk.daydream.monster/inference \
  -H "Authorization: Bearer $SK" -H "Content-Type: application/json" \
  -d '{"capability":"marlin-video","params":{
        "video_url":"https://.../clip.mp4",
        "prompt":"Describe the single main moving subject, where it is in frame over time, and the static background."}}'
# → { "data": { "scene": "A man in a green jacket walks ... in front of a red brick wall ...",
#               "events": [ { "start": 0.0, "end": 1.5, "text": "..." }, ... ] } }

# 2. REMOVE — erase the subject, reconstruct the background.
curl -s https://sdk.daydream.monster/inference \
  -H "Authorization: Bearer $SK" -H "Content-Type: application/json" \
  -d '{"capability":"void-inpaint","params":{
        "video_url":"https://.../clip.mp4",
        "mask_prompt":"the man in the green jacket walking",
        "prompt":"empty city sidewalk in front of a red brick wall, daytime, no people",
        "negative_prompt":"person, man, human, figure, silhouette, blur, ghost, smear",
        "num_inference_steps":30,
        "enable_pass2_refinement":true,
        "num_frames":194}}'
# → { "video_url": "https://.../void_out.mp4", "data": { "video": { "url": "https://.../void_out.mp4" } } }

# 3. RECOMPOSE — clean result + a BEFORE | AFTER split for the reveal.
ffmpeg -i original.mp4 -i vanished.mp4 -filter_complex \
  "[0:v]drawtext=text='BEFORE':x=20:y=20:fontsize=36:fontcolor=white[a];\
   [1:v]drawtext=text='AFTER':x=20:y=20:fontsize=36:fontcolor=white[b];\
   [a][b]hstack=inputs=2" -c:v libx264 before-after.mp4
```

## Tips (learned building the showcase)

- **Pick the clip for the model, not the other way around.** The whole game is choosing footage that fits the "works beautifully" column. A locked-off shot of one person on a plain wall will look like sorcery; a handheld pan through a crowded plaza will look like a smeared mess. Spend your effort on clip selection, not on coaxing a bad clip.
- **Write the prompts from marlin's mouth.** marlin-video describes the scene in plain English — lift its nouns straight into your `mask_prompt` (the subject) and `prompt` (the background). It already saw the clip; trust its description.
- **`prompt` describes the RESULT, not the action.** It's "empty sidewalk, brick wall, no people" — what you want the frame to *become* — not "remove the man". The `mask_prompt` is what to remove; the `prompt` is what's left.
- **Always ban the subject in `negative_prompt`.** "person, human, figure, silhouette, ghost, blur" steers the diffusion away from re-hallucinating a faint version of who you just erased — the classic "ghost" failure.
- **Downscale to 720p first.** void-inpaint is heavy. 1280×720 is plenty for a social reel and runs far faster than 4K. Re-encode to clean H.264 yuv420p so the model ingests it cleanly.
- **Fast vs max quality (`enable_pass2_refinement`).** void-inpaint runs in two passes, and the second one is most of the wall-clock. For an 8s / ~194-frame / 720p clip the timings are: SAM-3 mask ~19s + pass-1 ~69s + **pass-2 refinement ~197s ≈ ~5 min total**.
  - **Max (`enable_pass2_refinement: true`, default):** the cleanest fill — pass-2 removes residual ghosting/seams. Use it for the keeper you ship.
  - **Fast (`enable_pass2_refinement: false`):** skips pass-2 → **roughly half the time (~100s)**. Use it while you're iterating on `mask_prompt`/`prompt`/clip choice, or when pass-1 is already clean on a simple background.
  - `num_inference_steps` (default 30) is the other dial — lower = faster but softer.
  - **Workflow: iterate on Fast, then re-run the winner on Max.** Don't burn ~5 min per attempt while you're still tuning the prompts.
- **The BEFORE | AFTER split is the deliverable.** A bare "empty street" clip is unremarkable on its own — viewers don't know a person was ever there. The hstack reveal *is* the magic. Always ship it.
- **Removal, not replacement.** If the brief is "swap the person for a robot", that's two steps: void-inpaint to clean the plate, then composite a separately-generated element on top. void-inpaint alone only erases.

## Costs

| Operation | Cost | Notes |
|---|---|---|
| `marlin-video` (locate) | ~$0.02 | one cheap pass to confirm subject + write prompts |
| `void-inpaint` (the removal) | ~$0.30–$0.80 | priced per video-second; pass2 refinement adds a little |
| `ffmpeg` recompose / hstack | ~$0.00–$0.01 | local or tool-cap; negligible |
| **Typical 6–8s clip, end to end** | **~$0.35–$0.85** | a clean vanished plate + a before/after reveal |

## See also

- **Showcase:** [`/chapters/remove-object-magic-example.html`](/chapters/remove-object-magic-example.html) — the source clip (CC0, credited), the BEFORE | AFTER reveal, the pipeline, and an honest note on what works and what doesn't.
- **Related capabilities:**
  - `sam3` / `yolo-segment` — build a manual mask if the auto-mask grabs the wrong thing
  - `marlin-video` — video understanding; locate the subject + timestamps before you spend on the inpaint
  - `ffmpeg-concat` / `ffmpeg-overlay` — recompose, label, and stack the before/after