evalsnap

evalsnap

You re-ran your agent and the answers look slightly different — but did anything actually get worse? evalsnap takes two snapshots of case outputs, a baseline run and a current run, matches them by case_id, and tells you exactly what moved:

• Matched, added, removed — cases present on both sides, cases that only exist in the current run, and cases that disappeared from the baseline, each list sorted by id
• Per-case results — for every matched case: whether the output is unchanged or changed under exact string equality, the score delta and the duration delta (each null when either side did not record that measurement), and a regression flag that is set only when the score went down
• Regressions and improvements — the sorted list of regressed case ids and the count of cases whose score rose
• Average scores — each side averaged over the matched cases that carry a score on both sides, so the comparison covers the same set on both sides; both averages are null when no matched case qualifies
• Verdict — fail when at least one regression exists, otherwise pass; added and removed cases are reported but never fail the verdict on their own

Deltas and averages are rounded to four decimal places. The computation is pure and local: no network, no configuration, nothing read beyond the snapshots you pass in. Invalid input — an empty snapshot, a case without an id or output, a duplicate id inside one side, a non-numeric score, a negative duration — is rejected with an error naming the offending path and side, and no partial diff is produced.

CLI

npx @reneturcios/evalsnap diff --input args.json
npx @reneturcios/evalsnap --version

args.json contains the full arguments object:

{
  "baseline": [
    { "case_id": "greeting", "output": "hello", "score": 0.9, "duration_ms": 120 },
    { "case_id": "summary", "output": "short text", "score": 0.7 }
  ],
  "current": [
    { "case_id": "greeting", "output": "hello", "score": 0.95, "duration_ms": 110 },
    { "case_id": "summary", "output": "short text", "score": 0.6 }
  ]
}

score and duration_ms are optional per entry. The diff is printed as JSON.

Exit codes: 0 on success, 1 on a runtime error (including an unreadable or invalid input file, or invalid input), 2 on a usage error.

MCP server

npx -y @reneturcios/evalsnap mcp

MCP client config:

{
  "mcpServers": {
    "evalsnap": {
      "command": "npx",
      "args": ["-y", "@reneturcios/evalsnap", "mcp"]
    }
  }
}

Tools

• diff_snapshots — Diff two eval snapshots matched by case_id.

  • baseline (array, required, at least one) — entries as {case_id, output, score?, duration_ms?} objects
  • current (array, required, at least one) — same entry shape

  Returns {total_baseline, total_current, matched, unchanged, changed, added, removed, case_results, regressions, improved_count, baseline_avg_score, current_avg_score, verdict}.

Docker

docker run -i ghcr.io/cloudcorprecords/evalsnap
docker run --rm -v "$PWD:/w" ghcr.io/cloudcorprecords/evalsnap diff --input /w/args.json

Development

npm install
npm run build   # tsc -> dist/
npm test        # tsx --test test/*.test.ts (pure local, no network)
npm run dev     # run the CLI from source

License

MIT — Copyright (c) 2026 Rene Turcios