vision-multimodal

视觉与多模态技能

利用 Claude 的视觉能力进行图像分析、文档处理和多模态理解。

何时使用此技能

• 图像分析与描述
• 文档/PDF 处理
• 截图分析
• 类 OCR 文本提取
• 视觉比较
• 图表和示意图解读

支持的格式

格式	状态	最适用于
JPEG	✓	照片、自然场景
PNG	✓	截图、UI、文本
GIF	✓	动画（仅首帧）
WebP	✓	现代格式、压缩图像
PDF	✓	文档（通过 Files API）

图像尺寸指南

• 最小： 200 像素（更小会降低准确性）
• 最佳： 1000x1000 像素
• 最大： 8000x8000 像素
• Token 成本： ~(宽度 × 高度) / 1000
• 提示： 将最大尺寸调整至 1568px，可节省 30-50% 的 token

核心模式

模式 1：单张图像分析

import anthropic
import base64

client = anthropic.Anthropic()

# Load and encode image
with open("image.jpg", "rb") as f:
    image_data = base64.standard_b64encode(f.read()).decode("utf-8")

response = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=1024,
    messages=[{
        "role": "user",
        "content": [
            {
                "type": "image",
                "source": {
                    "type": "base64",
                    "media_type": "image/jpeg",
                    "data": image_data
                }
            },
            {
                "type": "text",
                "text": "Describe this image in detail."
            }
        ]
    }]
)

模式 2：来自 URL 的图像

import httpx

# Fetch and encode from URL
image_url = "https://example.com/image.jpg"
response = httpx.get(image_url)
image_data = base64.standard_b64encode(response.content).decode("utf-8")

# Then use same pattern as above

模式 3：多张图像

# Compare multiple images (up to 100 per request)
messages = [{
    "role": "user",
    "content": [
        {"type": "image", "source": {"type": "base64", "media_type": "image/jpeg", "data": image1}},
        {"type": "image", "source": {"type": "base64", "media_type": "image/jpeg", "data": image2}},
        {"type": "text", "text": "Compare these two images and list the differences."}
    ]
}]

模式 4：使用图像进行少样本学习

# Teach by example
messages = [
    # Example 1
    {"role": "user", "content": [
        {"type": "image", "source": {...}},
        {"type": "text", "text": "Classify this image."}
    ]},
    {"role": "assistant", "content": "Category: Landscape\nElements: Mountains, lake, trees"},

    # Example 2
    {"role": "user", "content": [
        {"type": "image", "source": {...}},
        {"type": "text", "text": "Classify this image."}
    ]},
    {"role": "assistant", "content": "Category: Portrait\nElements: Person, indoor, professional"},

    # Target image
    {"role": "user", "content": [
        {"type": "image", "source": {"type": "base64", "media_type": "image/jpeg", "data": target_image}},
        {"type": "text", "text": "Classify this image."}
    ]}
]

模式 5：PDF 处理

# Using Files API (beta)
with open("document.pdf", "rb") as f:
    pdf_data = base64.standard_b64encode(f.read()).decode("utf-8")

response = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=4096,
    messages=[{
        "role": "user",
        "content": [
            {
                "type": "document",
                "source": {
                    "type": "base64",
                    "media_type": "application/pdf",
                    "data": pdf_data
                }
            },
            {"type": "text", "text": "Summarize this document."}
        ]
    }]
)

视觉提示词工程

策略 1：分配角色

prompt = """You have perfect vision and exceptional attention to detail,
making you an expert at analyzing technical diagrams.

Analyze this architecture diagram and identify:
1. All components
2. Data flow between components
3. Potential bottlenecks"""

策略 2：逐步思考

prompt = """Before answering, analyze the image systematically:

<thinking>
1. What is the overall subject?
2. What are the key elements?
3. How do elements relate to each other?
4. What details stand out?
</thinking>

Then provide your answer based on this analysis."""

策略 3：结构化输出

prompt = """Extract information from this receipt and return as JSON:

{
    "vendor": "",
    "date": "",
    "items": [{"name": "", "price": 0}],
    "total": 0
}"""

图像优化

from PIL import Image
import io

def optimize_for_claude(image_path, max_dimension=1568):
    """Resize image to reduce token usage by 30-50%"""
    with Image.open(image_path) as img:
        # Calculate new dimensions
        ratio = min(max_dimension / img.width, max_dimension / img.height)
        if ratio < 1:
            new_size = (int(img.width * ratio), int(img.height * ratio))
            img = img.resize(new_size, Image.LANCZOS)

        # Convert to bytes
        buffer = io.BytesIO()
        img.save(buffer, format="JPEG", quality=85)
        return base64.standard_b64encode(buffer.getvalue()).decode("utf-8")

常见用例

文本提取（类 OCR）

prompt = """Extract all text from this image.
Preserve the original formatting and structure as much as possible.
If text is unclear, indicate with [unclear]."""

表格提取

prompt = """Extract the table data from this image.
Return as a markdown table with proper headers and alignment."""

图表分析

prompt = """Analyze this chart:
1. What type of chart is this?
2. What are the axes/labels?
3. What are the key data points?
4. What trends or patterns are visible?"""

最佳实践

应做：

• 使用高质量图像（≥1000px）
• 调整大图尺寸以节省 token
• 提供需要关注的内容背景
• 使用少样本示例以获得一致的输出

不应做：

• 发送小于 200px 的图像
• 期待手写内容能被完美 OCR 识别
• 发送超大图像（>8000px）
• 忽略多张图像的 token 成本

限制

• 无法识别特定个人
• 可能难以识别非常小的文本
• 动态 GIF：仅分析首帧
• 某些专用符号可能会被误读

另请参阅

• [[llm-integration]] - API 基础知识
• [[extended-thinking]] - 复杂推理
• [[citations-retrieval]] - 文档引用