beat-sync-video-editing

Beat-Sync Video Editing

Purpose

Provide domain expertise for creating beat-synced video edits: taking a source video and an audio track, selecting clips from the video that align with the music's rhythm, and rendering the final output with FFmpeg.

Core Concept: The EditPlan

Every edit starts as an EditPlan — a JSON structure that describes which video clips to use and where in the audio to place them:

{
  "audio_start": "00:13",
  "audio_duration": 6.5,
  "clips": [
    { "video_start": "00:08", "duration": 2.0, "description": "Opening shot" },
    { "video_start": "00:45", "duration": 1.5, "description": "Action moment" },
    { "video_start": "01:22", "duration": 3.0, "description": "Build-up" }
  ],
  "reasoning": "Matches rising intensity with beat drops"
}

Timestamp format: audio_start and video_start use MM:SS strings (e.g. "01:15" for 1 minute 15 seconds). audio_duration and clip duration use numbers in seconds.

Critical constraints:

• audio_start must be valid MM:SS format, non-negative
• audio_duration must be positive (seconds)
• Every clip must have valid MM:SS video_start and positive duration (seconds)
• Sum of all clip durations must equal audio_duration (within 0.5s tolerance)
• Clip order is intentional — not necessarily chronological. Non-linear ordering creates dynamic edits.

Workflow: From Files to Final Video

Step 1: Generate EditPlan via Gemini

Run the Gemini script to analyze video + audio and produce a plan:

# Fresh upload (default: files kept 48h for reuse, ECLIPTIC_FILES printed on stderr):
bash ${CLAUDE_PLUGIN_ROOT}/scripts/gemini-edit-plan.sh \
  --video <video_path> \
  --audio <audio_path> \
  --prompt "<user's edit description>"

# Reuse previously uploaded files (skips upload, much faster):
bash ${CLAUDE_PLUGIN_ROOT}/scripts/gemini-edit-plan.sh \
  --video-uri <uri> --video-mime <mime> \
  --audio-uri <uri> --audio-mime <mime> \
  --prompt "<different description>"

# One-shot mode (delete files immediately after use):
bash ${CLAUDE_PLUGIN_ROOT}/scripts/gemini-edit-plan.sh \
  --video <video_path> \
  --audio <audio_path> \
  --prompt "<description>" --cleanup

• Outputs EditPlan JSON to stdout, progress to stderr
• Requires GEMINI_API_KEY, curl, and jq
• Gemini watches the video and listens to the audio simultaneously

Supported formats (the Interactions API enforces these — anything else is rejected):

• Video: mp4, mov, avi, mpeg, mpg, webm, wmv, 3gpp (not mkv — see Troubleshooting)
• Audio: mp3, wav, aac, ogg, flac, m4a, opus, aiff

File lifecycle. The script keeps uploaded files by default so subsequent runs can reuse them without re-uploading. Gemini enforces a 48-hour TTL — after that, URIs go stale and must be re-uploaded. Each run emits an ECLIPTIC_FILES JSON line on stderr containing video_name, audio_name, URIs, and MIME types — capture this to iterate with --video-uri / --audio-uri. The script also prints the exact cleanup-gemini-files.sh command at the end of every fresh-upload run so deletion is explicit and copy-pasteable. If the ECLIPTIC_FILES line is lost, bash ${CLAUDE_PLUGIN_ROOT}/scripts/list-gemini-files.sh lists everything currently uploaded — names, URIs, MIME types, and expiration times. To flip the default to one-shot deletion, pass --cleanup or export GEMINI_CLEANUP=1 in the environment. --no-cleanup is accepted as a no-op alias for the default.

Model selection. The script defaults to gemini-3.6-flash. To use a different model, export GEMINI_MODEL before invocation:

export GEMINI_MODEL=gemini-3.1-pro-preview   # stronger reasoning, slower
# or
export GEMINI_MODEL=gemini-3.5-flash-lite    # cheaper/faster for simple edits

The model name is passed straight into the request body, so any Gemini model that supports video + audio input and structured JSON output will work. Avoid deprecated gemini-2.x and gemini-1.5 models.

Tuning (optional env vars). Both default to unset, which uses the API/model defaults:

export GEMINI_MEDIA_RESOLUTION=high    # low | medium | high | ultra_high
export GEMINI_THINKING_LEVEL=medium    # minimal | low | medium | high

• GEMINI_MEDIA_RESOLUTION — how much visual detail the model sees per frame. Higher values improve clip selection on detailed footage at higher token cost.
• GEMINI_THINKING_LEVEL — how long the model reasons about the edit. minimal is fastest/cheapest for simple edits; high helps complex pacing requests.

API note. The script calls the Gemini Interactions API (POST /v1beta/interactions) with the video and audio referenced as typed input parts and the EditPlan schema enforced via response_format. Requests are sent with store: false, so interactions are not retained server-side (uploaded files are separate — see the file lifecycle above).

Step 2: Validate the Plan

echo '<plan_json>' | bash ${CLAUDE_PLUGIN_ROOT}/scripts/validate-plan.sh

• Outputs {"valid": true, "errors": []} or {"valid": false, "errors": [...]}
• Exit code 0 = valid, 1 = invalid

Step 3: Build FFmpeg Filters

echo '<plan_json>' | bash ${CLAUDE_PLUGIN_ROOT}/scripts/build-filter.sh

• Outputs {"videoFilter": "...", "audioFilter": "...", "fullFilter": "..."}
• The fullFilter field is what goes into FFmpeg's -filter_complex argument

Step 4: Render with FFmpeg

ffmpeg -y -i "<video_path>" -i "<audio_path>" \
  -filter_complex "<fullFilter>" \
  -map "[outv]" -map "[outa]" \
  -c:v libx264 -preset fast -crf 23 \
  -c:a aac -shortest \
  "<output_path>"

FFmpeg Filter Anatomy

For a 3-clip edit, the fullFilter looks like:

[0:v]trim=start=8.000:duration=2.000,setpts=PTS-STARTPTS[v0];
[0:v]trim=start=45.000:duration=1.500,setpts=PTS-STARTPTS[v1];
[0:v]trim=start=22.000:duration=3.000,setpts=PTS-STARTPTS[v2];
[v0][v1][v2]concat=n=3:v=1:a=0[outv];
[1:a]atrim=start=13.000:duration=6.500,asetpts=PTS-STARTPTS[outa]

• [0:v] = first input (video), [1:a] = second input (audio)
• trim extracts a segment, setpts=PTS-STARTPTS resets timestamps
• concat joins all video segments in order
• atrim extracts the audio section

For the full FFmpeg filter reference, see ${CLAUDE_SKILL_DIR}/references/ffmpeg-filters.md.

Troubleshooting

Duration mismatch error: Clip durations don't sum to audio_duration. Fix by adjusting the last clip's duration to absorb the difference, or re-run Gemini with a stricter prompt.

FFmpeg "Error" in stderr: FFmpeg writes progress and warnings to stderr. Only treat it as a real error if the output file wasn't created. Check for actual error patterns like No such file, Invalid data, or Conversion failed.

Unsupported format (.mkv): The Interactions API does not accept Matroska. Remux losslessly first: ffmpeg -i input.mkv -c copy input.mp4.

Expired file URIs: Uploaded files expire after 48 hours. If a --video-uri/--audio-uri reuse run fails with a Gemini error about the file (e.g. not found or permission denied), the URIs are stale — re-run with --video/--audio local paths to upload fresh copies.

Gemini returns poor clips: Add specificity to the prompt. Instead of "make an edit", say "make a fast 30-second action edit, cut every 1-2 seconds on the beat drops, start from the chorus".

Additional Resources

Reference Files

• ${CLAUDE_SKILL_DIR}/references/ffmpeg-filters.md — Detailed FFmpeg filter_complex syntax, encoding options, common flags
• ${CLAUDE_SKILL_DIR}/references/edit-plan-schema.md — Full EditPlan JSON schema, validation rules, edge cases

Scripts

• ${CLAUDE_PLUGIN_ROOT}/scripts/gemini-edit-plan.sh — Upload to Gemini via REST API, get EditPlan (supports --cleanup and reuse of uploaded files via --video-uri/--audio-uri)
• ${CLAUDE_PLUGIN_ROOT}/scripts/validate-plan.sh — Validate EditPlan JSON
• ${CLAUDE_PLUGIN_ROOT}/scripts/build-filter.sh — Convert EditPlan to FFmpeg filters
• ${CLAUDE_PLUGIN_ROOT}/scripts/list-gemini-files.sh — List uploaded files with names, URIs, and expiration times
• ${CLAUDE_PLUGIN_ROOT}/scripts/cleanup-gemini-files.sh — Delete uploaded files from Gemini when done iterating