mcpField
V
vision-bench

vision-bench

github.com/glebisTesting & quality401173 installsUpdated 1d ago
Repository

About vision-bench

vision-bench is a CLI for evaluating and comparing images with one or more vision LLM judges. It suits image-generation and image-understanding evaluations, using preset or custom YAML criteria and several judge providers.

Key features

  1. 01Score images with vision LLM judges
  2. 02Compare multiple images
  3. 03Use preset evaluation criteria
  4. 04Define custom YAML criteria
  5. 05Export Markdown, JSON, or tables

Use cases

  1. 01Compare AI image generator outputs
  2. 02Assess portrait or product photo quality
  3. 03Evaluate OCR and chart understanding

Install

Copy the command and run it in a terminal.

$npx skills add https://github.com/glebis/claude-skills --skill vision-bench

After installing, type/vision-benchin Claude Code to confirm it loaded.

Bundled files

References and scripts installed along with SKILL.md.

.gitignore
.sops.yaml
bench.py
judge.py
report.py
requirements.txt
vault.py
.claude-plugin/
plugin.json
criteria/
alt_text.yaml
artistic_style.yaml
chart_analysis.yaml
document_ocr.yaml
invoice.yaml
photorealism.yaml
portrait.yaml
product_photo.yaml
scientific.yaml
text_to_image.yaml
ui_screenshot.yaml

Details

From the project README

Vision Bench — LLM Image Evaluation

Compare images by scoring them with one or more vision LLM judges against structured rubric criteria.

Quick Start

# Install dependencies
pip install pyyaml openai anthropic mistralai

# Score a single image
python bench.py image.png --criteria photorealism --judge gemini-2.5-flash

# Compare two AI-generated images
python bench.py img_a.png img_b.png \
  --criteria text_to_image \
  --prompt "a fox in a snowy forest" \
  --judge gpt-4o

# Multi-judge consensus
python bench.py img.png \
  --criteria portrait \
  --judges gpt-4o gemini-2.5-flash claude-opus-4-5-20251022

# OpenRouter models (any vision-capable model)
python bench.py img_a.png img_b.png \
  --criteria artistic_style \
  --judges "openrouter/meta-llama/llama-4-maverick" "openrouter/mistralai/pixtral-large-2411"

# List all presets
python bench.py --list-presets

# Save report to file
python bench.py img.png --criteria chart_analysis --save report.md

Presets

PresetUse Case
text_to_imageCompare AI image generators (Midjourney, DALL-E, Flux)
photorealismHow convincingly an image looks like a photo
artistic_styleStyle consistency, composition, color harmony
portraitAI-generated portrait quality and realism
product_photoE-commerce product image quality
document_ocrDocument text extraction and layout understanding
chart_analysisChart and data visualization comprehension
invoiceFinancial document field extraction accuracy
ui_screenshotApp/web screenshot understanding
scientificScientific/medical image accuracy
alt_textAccessibility image description quality

Custom criteria: pass any .yaml file as --criteria path/to/my.yaml.

Judge Providers

PrefixProviderExample
gpt-, o1, o3, o4OpenAIgpt-4o
claude-Anthropicclaude-sonnet-4-5-20251022
gemini-Google Geminigemini-2.5-flash
pixtral-, mistral-, ministral-Mistralpixtral-12b-2409
openrouter/OpenRouter (any model)openrouter/meta-llama/llama-4-maverick

API Keys

Keys are loaded from secrets.enc.yaml (SOPS + age encrypted) with fallback to environment variables.

Supported keys: OPENAI_API_KEY, ANTHROPIC_API_KEY, GEMINI_API_KEY, OPENROUTER_API_KEY

To encrypt your own keys:

sops --config .sops.yaml --encrypt --input-type yaml --output-type yaml secrets.yaml > secrets.enc.yaml

Output Formats

--output markdown (default) · --output json · --output table

Files

  • bench.py — CLI entry point
  • judge.py — Multi-provider LLM judge logic
  • report.py — Report generation
  • vault.py — SOPS secrets decryption
  • criteria/ — 11 YAML preset files
  • .sops.yaml — Age key config for encryption
  • secrets.enc.yaml — Encrypted API keys

Comments

Markdown supported