mcpField
A
ai-multimodal

ai-multimodal

github.com/bmad-labsCreative & media19176 installsUpdated 1w ago
Repository

About ai-multimodal

AI Multimodal is a skill and CLI toolkit for processing media and generating images or videos with Google Gemini's multimodal API. It includes batch scripts, media preparation, and document conversion workflows.

Key features

  1. 01Audio, image, video, and document analysis
  2. 02Transcription and content extraction
  3. 03Image and video generation
  4. 04Media optimization and conversion
  5. 05Batch processing with structured output

Use cases

  1. 01Transcribing audio recordings
  2. 02Analyzing images, videos, and documents
  3. 03Generating image or video assets
  4. 04Preparing large media files for processing

Install

Copy the command and run it in a terminal.

$npx skills add https://github.com/bmad-labs/skills --skill ai-multimodal

After installing, type/ai-multimodalin Claude Code to confirm it loaded.

Bundled files

References and scripts installed along with SKILL.md.

.env.example
references/
audio-processing.md
image-generation.md
video-analysis.md
video-generation.md
vision-understanding.md
scripts/
check_setup.py
document_converter.py
gemini_batch_process.py
media_optimizer.py
requirements.txt
scripts/tests/
requirements.txt
test_document_converter.py
test_gemini_batch_process.py
test_media_optimizer.py

Details

From the project README

AI Multimodal

Process audio, images, videos, documents, and generate images/videos using Google Gemini's multimodal API.

Setup

export GEMINI_API_KEY="your-key"  # Get from https://aistudio.google.com/apikey
pip install google-genai python-dotenv pillow

Quick Start

Verify setup: python scripts/check_setup.py Analyze media: python scripts/gemini_batch_process.py --files <file> --task <analyze|transcribe|extract>

  • TIP: When you're asked to analyze an image, check if gemini command is available, then use "<prompt to analyze image>" | gemini -y -m gemini-2.5-flash command. If gemini command is not available, use python scripts/gemini_batch_process.py --files <file> --task analyze command. Generate content: python scripts/gemini_batch_process.py --task <generate|generate-video> --prompt "description"

Stdin support: You can pipe files directly via stdin (auto-detects PNG/JPG/PDF/WAV/MP3).

  • cat image.png | python scripts/gemini_batch_process.py --task analyze --prompt "Describe this"
  • python scripts/gemini_batch_process.py --files image.png --task analyze (traditional)

Models

  • Image generation: imagen-4.0-generate-001 (standard), imagen-4.0-ultra-generate-001 (quality), imagen-4.0-fast-generate-001 (speed)
  • Video generation: veo-3.1-generate-preview (8s clips with audio)
  • Analysis: gemini-2.5-flash (recommended), gemini-2.5-pro (advanced)

Scripts

  • gemini_batch_process.py: CLI orchestrator for transcribe|analyze|extract|generate|generate-video that auto-resolves API keys, picks sensible default models per task, streams files inline vs File API, and saves structured outputs (text/JSON/CSV/markdown plus generated assets) for Imagen 4 + Veo workflows.
  • media_optimizer.py: ffmpeg/Pillow-based preflight tool that compresses/resizes/converts audio, image, and video inputs, enforces target sizes/bitrates, splits long clips into hour chunks, and batch-processes directories so media stays within Gemini limits.
  • document_converter.py: Gemini-powered converter that uploads PDFs/images/Office docs, applies a markdown-preserving prompt, batches multiple files, auto-names outputs under docs/assets, and exposes CLI flags for model, prompt, auto-file naming, and verbose logging.
  • check_setup.py: Interactive readiness checker that verifies directory layout, centralized env resolver, required Python deps, and GEMINI_API_KEY availability/format, then performs a live Gemini API call and prints remediation instructions if anything fails.

Use --help for options.

References

Load for detailed guidance:

TopicFileDescription
Audioreferences/audio-processing.mdAudio formats and limits, transcription (timestamps, speakers, segments), non-speech analysis, File API vs inline input, TTS models, best practices, cost and token math, and concrete meeting/podcast/interview recipes.
Imagesreferences/vision-understanding.mdVision capabilities overview, supported formats and models, captioning/classification/VQA, detection and segmentation, OCR and document reading, multi-image workflows, structured JSON output, token costs, best practices, and common product/screenshot/chart/scene use cases.
Image Genreferences/image-generation.mdImagen 4 and Gemini image model overview, generate_images vs generate_content APIs, aspect ratios and costs, text/image/both modalities, editing and composition, style and quality control, safety settings, best practices, troubleshooting, and common marketing/concept-art/UI scenarios.
Videoreferences/video-analysis.mdVideo analysis capabilities and supported formats, model/context choices, local/inline/YouTube inputs, clipping and FPS control, multi-video comparison, temporal Q&A and scene detection, transcription with visual context, token and cost guidance, and optimization/best-practice patterns.
Video Genreferences/video-generation.mdVeo model matrix, text-to-video and image-to-video quick start, multi-reference and extension flows, camera and timing control, configuration (resolution, aspect, audio, safety), prompt design patterns, performance tips, limitations, troubleshooting, and cost estimates.

Limits

Formats: Audio (WAV/MP3/AAC, 9.5h), Images (PNG/JPEG/WEBP, 3.6k), Video (MP4/MOV, 6h), PDF (1k pages) Size: 20MB inline, 2GB File API

Resources

Comments

Markdown supported