evals-mcp-server

---

Overview

Build verifiable eval records with server-enforced graders. Draft a record with its own executable grader, revise it field by field, and submit through a gate that requires the gold answer to pass, a negative case to fail, and an independent check to agree. Export to JSONL, CSV, Inspect AI, or lm-evaluation-harness.

Tools

Tool	Description
evals_describe_schema	Return the required and optional fields plus grader options for a task type. Call before drafting.
evals_create_draft	Create a draft eval record carrying its own grader; returns the parsed record, a review protocol, and a verification subagent prompt.
evals_get_record	Read a draft or submitted record by id; the id is stable across submit.
evals_revise_draft	Apply a surgical set / append / unset patch to a draft by dotted path; re-runs the self-consistency check.
evals_discard_draft	Delete a draft record by id. Draft-only.
evals_run_check	Run a grader spec against candidate answers and get PASS/REJECT per candidate, decoupled from any saved record.
evals_submit_draft	Finalize a draft through the committability gate, then freeze it.
evals_list_records	Browse and filter records by status, domain, task type, or tag. Returns a compact summary per record.
evals_export_records	Compile submitted records to JSONL, CSV, Inspect AI, or lm-evaluation-harness and write the artifact under exports/.

Resources

Resource	Description
eval://record/{id}	A single draft or submitted record by id — the same payload evals_get_record returns, for resource-capable clients.

All record data is also reachable through the tool surface — evals_get_record for a single record, evals_list_records to browse. The resource is a convenience mirror for clients that support resources, not the access path.

Capability reference

evals_describe_schema _tool

• Accepts task_type: numeric, exact_answer, set_answer, mcq, regex_answer, json_answer, or free_response.
• Returns the gold shape, grader kinds, required/optional fields, and authoring notes from static schemas; mcq requires choices, and free_response requires llm_rubric.

---

evals_create_draft _tool

• Accepts a record with its grader and discrimination cases, plus optional verification and captures (EvalsIDs); mcq requires choices, and free_response requires llm_rubric.
• Saves the record as a draft once its grader runs, and returns the normalized record, a review protocol, a verification-subagent prompt, and what submit still needs. A draft whose gold fails its grader is still saved.
• Typed errors: grader_unexecutable, task_type_constraint, mcq_choice_mismatch.

---

evals_get_record _tool

• Reads a draft or submitted record by id, which stays stable across submit.
• Returns the full grader, discrimination cases, and verification evidence; not_found points to evals_list_records.

---

evals_revise_draft _tool

• Applies set, append, and unset by dotted path to a draft; task_type and server-owned fields cannot be changed, and arrays are replaced whole or appended, not edited by index.
• Re-validates the record and grader before writing, then returns the record plus changed entries (op, path, before, after). An unknown path rejects the whole patch.
• Typed errors: not_found, record_frozen for a submitted id, invalid_patch_path, grader_unexecutable, task_type_constraint, mcq_choice_mismatch.

---

evals_discard_draft _tool

• Deletes a draft by draft_id and returns discarded: true.
• A submitted record reports record_frozen; a missing or already discarded id reports not_found.

---

evals_run_check _tool

• Runs a grader against one or more candidates without a saved record; supply gold for exact_match, while target-embedding kinds such as numeric and mcq ignore it.
• Returns PASS/REJECT, detail, and the resolved reference per candidate. llm_rubric can't be run here.
• Numeric grading accepts only finite real values; unit-bearing, complex, nonfinite, and non-scalar candidates are rejected. units is documentation only.
• Typed errors: grader_unexecutable, mcq_choice_mismatch.

---

evals_submit_draft _tool

• Finalizes by draft_id: gold must PASS, ≥1 negative must be REJECTED, and decorrelated verification must agree. Also checks captures against EVALS_CAPTURE_DIR, rejects duplicate content_hash, and can ask for confirmation via confirm or EVALS_REQUIRE_CONFIRMATION.
• On success, freezes the record with submitted_at and checksum and returns status: submitted; a refusal leaves the draft intact. free_response records rely on verification alone and carry server_verified: false.
• Typed errors: not_found, record_frozen, verification_incomplete, grader_failed_on_gold, verification_disagrees_with_gold, missing_negative_case, negative_case_passed, duplicate, decorrelation_violation, capture_unresolved, submit_declined.

---

evals_list_records _tool

• Filters by status (draft/submitted), domain, task_type, or tag; limit is a positive integer and defaults to 50.
• Returns summaries (id, status, task_type, domain, tags, timestamps), newest-first, with shown, cap, totalCount, and truncated.

---

evals_export_records _tool

• Exports submitted records, optionally filtered by domain, task_type, or tag, as jsonl (lossless), csv (lossy summary), inspect (Inspect AI), or lm-eval (lm-evaluation-harness).
• Writes under exports/ and returns the path, record count, byte size, and preview.

---

eval://record/{id} _resource

• Reads id from evals_list_records or a draft/submit response.
• Returns the evals_get_record payload as application/json, or not_found when no record matches.

Features

Built on @cyanheads/mcp-ts-core: stdio and Streamable HTTP transports, pluggable auth (none / jwt / oauth), swappable storage (in-memory, filesystem, Supabase, Cloudflare KV/R2/D1), structured logging with optional OpenTelemetry tracing.

Eval authoring:

• A draft → review → surgical-revise → submit loop, with the server both storing records and running each record's own grader to reject what doesn't hold up
• Seven task types: numeric, exact_answer, set_answer, mcq, regex_answer, json_answer, free_response
• A grader serialized with each record; deterministic kinds (numeric via math.js, exact_match, set_match, regex, mcq, json_match) run server-side, while llm_rubric relies on recorded independent verification
• A submit gate: the gold must pass its own grader, ≥1 negative must be rejected, and a recorded decorrelated verification must agree with the gold
• Plain JSON files under EVALS_DATA_DIR, with drafts, submitted records, and exports kept separate

Agent-friendly output:

• evals_create_draft and evals_revise_draft return the parsed record, a per-field review protocol, and a ready-to-paste verification-subagent prompt
• Draft and revise responses report pass/reject results for each positive and negative case
• evals_list_records reports shown / cap / total count when the limit is hit
• The submit gate fails with a typed reason plus a recovery hint

Getting started

Add the following to your MCP client configuration file. Set EVALS_DATA_DIR to a writable folder — the server manages drafts/, submitted/, and exports/ under it.

{
  "mcpServers": {
    "evals-mcp-server": {
      "type": "stdio",
      "command": "bunx",
      "args": ["@cyanheads/evals-mcp-server@latest"],
      "env": {
        "MCP_TRANSPORT_TYPE": "stdio",
        "MCP_LOG_LEVEL": "info",
        "EVALS_DATA_DIR": "/absolute/path/to/evals-data"
      }
    }
  }
}

Or with npx (no Bun required):

{
  "mcpServers": {
    "evals-mcp-server": {
      "type": "stdio",
      "command": "npx",
      "args": ["-y", "@cyanheads/evals-mcp-server@latest"],
      "env": {
        "MCP_TRANSPORT_TYPE": "stdio",
        "MCP_LOG_LEVEL": "info",
        "EVALS_DATA_DIR": "/absolute/path/to/evals-data"
      }
    }
  }
}

Or with Docker:

{
  "mcpServers": {
    "evals-mcp-server": {
      "type": "stdio",
      "command": "docker",
      "args": [
        "run", "-i", "--rm",
        "-e", "MCP_TRANSPORT_TYPE=stdio",
        "-v", "evals-data:/usr/src/app/evals-data",
        "ghcr.io/cyanheads/evals-mcp-server:latest"
      ]
    }
  }
}

For Streamable HTTP, set the transport and start the server:

MCP_TRANSPORT_TYPE=http MCP_HTTP_PORT=3010 EVALS_DATA_DIR=./evals-data bun run start:http
# Server listens at http://localhost:3010/mcp

Prerequisites

• Bun v1.4.0 or higher (or Node.js v24+).
• A writable directory for EVALS_DATA_DIR. No external API key is required.

Installation

1. Clone the repository:

git clone https://github.com/cyanheads/evals-mcp-server.git

2. Navigate into the directory:

cd evals-mcp-server

3. Install dependencies:

bun install

4. Configure environment:

cp .env.example .env
# edit .env and set EVALS_DATA_DIR

Configuration

All server configuration is validated at startup via Zod schemas in src/config/server-config.ts.

Variable	Description	Default
EVALS_DATA_DIR	Root folder for record JSON; the store manages drafts/, submitted/, and exports/ under it.	./evals-data
EVALS_REQUIRE_CONFIRMATION	When true, evals_submit_draft requests human confirmation through multi-round input before finalizing.	false
EVALS_DEFAULT_LICENSE	Default metadata.license applied when a draft omits one (e.g. CC-BY-4.0).	—
EVALS_CAPTURE_DIR	Directory of framework-written tool-call captures; when set, captures EvalsIDs resolve to full dumps.	—
MCP_TRANSPORT_TYPE	Transport: stdio or http.	stdio
MCP_HTTP_PORT	Port for the HTTP server.	3010
MCP_SESSION_MODE	HTTP session mode: auto, stateful, or stateless; stateless is refused because submit confirmation needs a live session.	stateful
MCP_AUTH_MODE	Auth mode: none, jwt, or oauth.	none
MCP_LOG_LEVEL	Log level (RFC 5424).	info
OTEL_ENABLED	Enable OpenTelemetry instrumentation.	false

See .env.example for the full list of optional overrides.

Submit confirmation expires after 10 minutes and must be repeated if the draft changes. When running more than one instance, use shared storage (STORAGE_PROVIDER_TYPE=filesystem with a shared STORAGE_FILESYSTEM_PATH, supabase, or cloudflare-d1; not cloudflare-kv) and set the same MCP_REQUEST_STATE_KEY (at least 32 bytes) on each.

Running the server

Local development

• Build and run:

  # One-time build
  bun run rebuild
  
  # Run the built server
  bun run start:stdio
  # or
  bun run start:http
  

• Run checks and tests:

  bun run devcheck   # Lint, format, typecheck, security
  bun run test       # Vitest test suite
  bun run lint:mcp   # Validate MCP definitions against spec
  

Docker

docker build -t evals-mcp-server .
docker run --rm -e MCP_TRANSPORT_TYPE=stdio -v evals-data:/usr/src/app/evals-data evals-mcp-server

The Dockerfile defaults to HTTP transport, stateful session mode, and logs to /var/log/evals-mcp-server. Mount record storage at /usr/src/app/evals-data, the default directory owned by the runtime user. OpenTelemetry peer dependencies are installed by default — build with --build-arg OTEL_ENABLED=false to omit them.

Project structure

Directory	Purpose
src/index.ts	createApp() entry point — registers tools and the resource, inits the record-store and exporter services.
src/config	Server-specific environment variable parsing and validation with Zod.
src/mcp-server/tools	Tool definitions (*.tool.ts).
src/mcp-server/resources	Resource definitions (*.resource.ts).
src/services/eval-record	The record schema, draft builder, and submit gate.
src/services/grader	Deterministic grader DSL execution and the committability check.
src/services/record-store	On-disk JSON record CRUD, the draft→submitted move, and export writes.
src/services/exporter	Compiling submitted records to JSONL/CSV/Inspect/lm-eval.
tests/	Unit and integration tests mirroring src/.

Development guide

See CLAUDE.md/AGENTS.md for development guidelines and architectural rules. The short version:

• Handlers throw, framework catches — no try/catch in tool logic
• Use ctx.log for request-scoped logging; records persist to disk via the record-store service, not ctx.state
• Register new tools and resources in the createApp() arrays in src/index.ts
• The server is the source of truth — validate inputs, run the grader as a hard gate, and never admit a record on assertion alone

Contributing

Issues are welcome. Run checks and tests before submitting:

bun run devcheck
bun run test

License

Apache-2.0 — see LICENSE for details.