subject-compositing

Subject Compositing

For shots nobody can film and no generator produces convincingly: him in a spaceship, in a clown car, on the moon. A real take is segmented out of its real surroundings and laid over replacement footage, with a prop in front. It lives or dies on the source take — settle the footage before you promise the shot.

Requires

pip install 'video-studio-engine[vision] @ https://github.com/scrollmark/social-skills/archive/refs/heads/master.tar.gz', then video-studio composite_subject … and video-studio track_pointing …. Neither is bundled here — both import MediaPipe, NumPy and OpenCV and download model files on first run, so neither can ship as a plain file inside a skill; [vision] is the heaviest extra in the package. Without it nothing here executes: no matte, no feathered occlusion seam, no arm punch-through, no gesture timing. What still holds is the footage guidance below — it tells the user what to shoot, and is worth giving before they shoot. The pointing grammar lives in video-formats.

What the Segmenter Needs From the Take

This decides whether the shot works, and it is decided on set, not in post. The model is a selfie segmenter — one person, front-facing, upper body, at conversational distance. Everything it fails at follows from that.

• One person, near arm visible. Pose landmarks pick whichever wrist the segmenter can see. A second person in frame, or an arm crossing out of shot, breaks free-arm.
• Tonal separation from the background. Hair, dark clothing on a dark wall, or a subject the same value as what is behind them gives a matte that chews the silhouette. It does not error; it just looks cheap.
• Flat even light, no motion blur. Fast pans and hard shadow edges leave the matte flickering frame to frame, which reads instantly as fake. Lock the camera off and let the subject move instead.
• Frame wide enough to cut. --occlude-below slices the subject at the prop's top edge with a feathered seam — it sells "inside it" and hides the real chair they were sitting on, but it needs body below the cut line to work with.
• Short takes. Every frame goes through the segmenter one at a time; a couple of seconds is the working unit. This is a shot, not a sequence.

The Composite

video-studio composite_subject out.mp4 --in take.mp4 --ss 39.9 --t 2.5 \
    --bg crowd.webm --bg-crop 405:600:430:15 --bg-eq eq=contrast=1.9:saturation=0 \
    --fg art/car.png --occlude-below 1215 --free-arm --travel -330 430 --flip-h

Layering is background → subject → prop → the gesturing forearm punched back through the prop. --free-arm intersects a capsule around elbow/wrist/hand with the real mask, so only body pixels come forward; without that intersection a drifting pose estimate cuts a person-shaped hole of background through the prop. --travel, --flip-h and --flip-v move subject and prop as one object over a static background, and mirroring happens before travel — flip afterwards and the flip reverses the travel direction too. --scale anchors on the subject, not the frame origin.

Draw the prop crudely, with an alpha channel. Wobbly hand-drawn art forgives the matte's soft edges; clean vector art makes it look broken beside them. A PNG without alpha is rejected outright. And check the background crop before compositing into it — a tight crop of soft archival footage becomes grey mush, and stadium footage is full of legible third-party advertising you have just put in a brand video.

Pointing

video-studio track_pointing analyze emits events — windows where an extended index finger holds a stable position — with the pointed-at location extrapolated past the fingertip, as canvas fractions. layers turns those into storyboard layers offset toward frame centre so the popup never covers the hand. Here the footage dictates the layout and the storyboard does not, so reconcile the event list with the user before composing: pull frames at each timestamp and check the count. Holds under half a second are dropped as jitter by design, and only one hand is tracked — quick tappers and two-handed presenters both come back short, and it reads as a broken tracker rather than a threshold working.

Anti-patterns

• Promising the shot before seeing the take. Background contrast and camera motion decide it, and neither is fixable in post.
• A clean vector prop. It makes the matte look broken; crude art hides it.
• Trusting the event list. Extract frames and reconcile the count with the user.