video-to-game

Video to Game

The video is the game design document, and you cannot watch it. A video model is good at what you are bad at: the look of a game, its camera, its one memorable mechanic. Your job is to reverse-engineer that clip into a game someone can play, with assets generated to match it, and to prove it with pictures, not adjectives.

Workflow

1. Generate or take the video. One original concept, one mechanic you can see, a third-person or side camera, no HUD text. prompts/gameplay-video-prompts.md (repo root) has the recipe and 31 prompts that worked.
2. Study it. python3 scripts/study_video.py gameplay.mp4 -o study/ saves a frame every 0.25 s, a contact sheet, the palette and the cuts. Open the sheet, then the frames. Write the spec before any code: camera (distance, height, lens), hero, the one core mechanic, world layout, palette, light, materials, mood, and the key moment the video sells. See references/study-the-video.md.
3. Generate every asset with fal, one job at a time, from the video's own frames as references: concept sheet, background removal, image to 3D, rigging, textures, parallax layers, sound effects, music. python3 scripts/fal_asset.py <endpoint> --arg k=v --file k=path -o assets/ runs a model and logs it. Nothing drawn by hand, nothing downloaded elsewhere. See references/fal-assets.md.
4. Build the game from templates/game.html: Three.js only, one import map, a real loop (controls, goal, progression, feedback, end, restart), keyboard and gamepad, ?autostart and ?seed=N, window.__game state. See references/game-loop.md.
5. Match the look. Palette, fog, light, grade, bloom, vignette and grain are part of the spec from day one, not polish. Compare a game frame with a video frame at fixed spots after every change: python3 scripts/side_by_side.py gameplay.mp4 shot.png -o pairs/ --times 2,5,8. See references/look-and-post.md.
6. Play it yourself. Write play.mjs from templates/play.mjs: keyboard only, reading window.__game. node scripts/record_play.mjs "http://127.0.0.1:8000/?autostart" play.mjs -o take.mp4 films it at its real frame rate; python3 scripts/best_window.py take.mp4 --length 10 finds the liveliest stretch; look at those frames. Fix what a player would hit: falls, walls, dead seconds, unreadable moments. See references/playtest.md.
7. Deliver: index.html + assets/ + README.md (concept, controls, every fal asset with its endpoint), the side-by-side pairs, the best take, and an honest list of what differs from the video.

Scripts need Python 3.9+, ffmpeg, Node 18+ with playwright-core for the recorder, and pip install fal-client with FAL_KEY set for fal_asset.py. Run them from this skill's directory or with full paths.

Hard rules

• The game looks like its video. Same camera, palette, light and materials. Every look decision is checked on a side-by-side pair, at the same three spots each time. "Looks right" without a pair image is not a check.
• The video's mechanic is the game. If the clip shows a girl turning the world upside down, the game is about turning the world upside down, not a generic runner with her model in it.
• Every asset comes from fal, generated from the video's frames as references. Log each call (file, endpoint). One job at a time: look at each result before asking for the next.
• No text on screen beyond a minimal title and one controls line. The world explains itself.
• Original worlds only. No characters, logos, names or level designs from existing games or films.
• Post-processing is in the spec: tone mapping, one grade, bloom, fog for depth, vignette, grain. Keep 60 fps on a laptop; 30 fps is acceptable only for a much better image.
• Calm the world, let the game read. One harmonized palette, no acid colors; dense but grouped; hero and interactive things stand out on a quiet background.
• Play before you ship. A recorded take with zero console errors, no fall, no dead second, the key moment of the video visible.

Reference routing

File	Read when
references/study-the-video.md	starting: frames, palette, camera, mechanic, the spec to write
references/fal-assets.md	generating assets: which model for what, the character pipeline, prompts that work, costs
references/game-loop.md	building: structure, input, camera, state contract, multiplayer, performance
references/look-and-post.md	matching the video's look: grade, fog, bloom, density, hierarchy, the side-by-side routine
references/playtest.md	playing your own game: scenarios, recording, best window, what to fix

Templates: templates/game.html (Three.js skeleton with the post stack, input, state contract), templates/play.mjs (a keyboard playtest scenario).

Output expectations

• Say what was checked and how: which pairs, which take, fps, console errors.
• List the fal calls: count per endpoint and total cost.
• List honestly what differs from the video (a simplified mechanic, a missing effect), never "identical".