ag2-multimodal-input

Multimodal inputs

When to use

The user wants the agent to process non-text input: an image to describe, audio to transcribe, video to summarise, or a PDF / document to extract from. The same factory pattern works across providers; per-provider support varies.

60-second recipe

from ag2 import Agent
from ag2.config import GeminiConfig
from ag2.events import ImageInput

agent = Agent(
    "vision",
    "You describe images.",
    config=GeminiConfig(model="gemini-3-flash-preview"),
)

image = ImageInput("https://example.com/photo.jpg")
reply = await agent.ask("Describe this image in detail.", image)
print(reply.body)

Multiple inputs in one ask are fine:

reply = await agent.ask(
    "Compare these two images.",
    ImageInput("https://example.com/before.jpg"),
    ImageInput("https://example.com/after.jpg"),
)

Input factories

Factory	Formats
ImageInput(...)	JPEG, PNG, GIF, WebP
AudioInput(...)	WAV, MP3, OGG, FLAC, AAC
VideoInput(...)	MP4, WebM, MOV, MKV, MPEG
DocumentInput(...)	PDF, TXT, HTML, Markdown, CSV, JSON, Office formats

Each accepts the same four data sources:

from ag2.events import ImageInput

ImageInput("https://example.com/photo.jpg")     # URL
ImageInput(path="photo.jpg")                    # local file
ImageInput(data=raw_bytes, media_type="image/png")  # bytes
ImageInput(file_id="file-abc123")               # provider-uploaded

Provider matrix

Input type	OpenAI	OpenAI Responses	Gemini	Anthropic
Text	✓	✓	✓	✓
Image (URL)	✓	✓	✓	✓
Image (binary)	✓	✓	✓	✓
Audio (URL)	–	–	✓	–
Audio (binary)	✓	–	✓	–
Video (URL)	–	–	✓	–
Video (binary)	–	–	✓	–
Document (URL)	–	✓	✓	✓
Document (binary)	✓	✓	✓	✓
File ID	✓	✓	✓	✓

Unsupported combinations raise UnsupportedInputError with a clear message.

Gemini has the broadest multimodal support. If you don't know which provider to pick for a multimodal task, start there.

Provider-specific niceties

Gemini — YouTube URLs work directly

from ag2.events import VideoInput

video = VideoInput("https://www.youtube.com/watch?v=dQw4w9WgXcQ")
reply = await agent.ask("Summarize this video.", video)

Gemini — large files (> 20MB) via Google Files API

from google import genai
from ag2.events import VideoInput
import time

client = genai.Client()
uploaded = client.files.upload(file="large_video.mp4")
while uploaded.state.name == "PROCESSING":
    time.sleep(2)
    uploaded = client.files.get(name=uploaded.name)

video = VideoInput(uploaded.uri)

Gemini — vendor_metadata

Key	Purpose
media_resolution	MEDIA_RESOLUTION_LOW/MEDIUM/HIGH/ULTRA_HIGH — token vs cost
video_metadata	Clipping (start_offset, end_offset) and fps
display_name	Display name for the file

ImageInput(data=raw, media_type="image/jpeg", vendor_metadata={"media_resolution": "MEDIA_RESOLUTION_LOW"})

VideoInput(path="lecture.mp4", vendor_metadata={
    "video_metadata": {"start_offset": "60s", "end_offset": "120s", "fps": 0.5},
})

OpenAI — image detail

ImageInput(data=raw, media_type="image/png", vendor_metadata={"detail": "low"})  # "low" | "high" | "auto"

Anthropic — File ID + prompt caching

import anthropic
from ag2.events import ImageInput, DocumentInput

client = anthropic.Anthropic()
uploaded = client.beta.files.upload(file=("photo.jpg", open("photo.jpg", "rb"), "image/jpeg"))

# filename determines block type (image vs document)
image = ImageInput(file_id=uploaded.id, filename="photo.jpg")

# Cache an attachment so subsequent turns skip re-uploading
doc = DocumentInput(path="report.pdf", vendor_metadata={"cache_control": {"type": "ephemeral"}})

FilesAPI — upload lifecycle, provider-agnostic

For any provider that has a file API (OpenAIConfig, OpenAIResponsesConfig, AnthropicConfig, GeminiConfig):

from ag2 import FilesAPI
from ag2.config import OpenAIResponsesConfig

files = FilesAPI(OpenAIResponsesConfig(model="gpt-5-mini"))

uploaded = await files.upload(path="report.pdf", purpose="assistants")
print(uploaded.file_id)

# Or from bytes (filename required)
uploaded = await files.upload(data=b"...", filename="hello.txt", purpose="assistants")

# List, read, delete
all_files = await files.list()
data = await files.read(uploaded.file_id)        # NotImplementedError on Gemini
await files.delete(uploaded.file_id)

Pass the file_id to DocumentInput, ImageInput, etc.:

from ag2.events import DocumentInput

doc = DocumentInput(file_id=uploaded.file_id)
reply = await agent.ask("Summarize this report.", doc)

Going deeper

• website/docs/user-guide/multimodal/inputs.mdx — full provider matrix and vendor_metadata reference.
• website/docs/user-guide/advanced/files.mdx — FilesAPI reference (upload / list / read / delete).
• For tools that return images / binary back to the LLM, see ag2-add-custom-tool (ImageInput, BinaryInput, ToolResult).

Common pitfalls

• Picking a provider that doesn't support your input type — silently you'll get UnsupportedInputError. Check the matrix; Gemini is broadest.
• FilesAPI.read() on Gemini — raises NotImplementedError. Gemini doesn't expose download.
• Calling files.upload(data=...) without filename= — raises ValueError. Filename is required for in-memory uploads.
• Supplying more than one source to a factory — not an error. The factory resolves in priority order url > file_id > path > data, so extra sources are silently ignored. Pass exactly one to get what you intend. Supplying zero sources raises ValueError.
• Anthropic ImageInput(file_id=...) without filename= — Anthropic decides block type (image vs document) by filename extension. Pass it.
• Gemini vendor_metadata keys are nested — video_metadata itself takes a dict. Check the doc table for shape.
• Forgetting to wait for Gemini file processing — large uploads have a PROCESSING state. Poll client.files.get(name=...) until ready before referencing the URI.