About digline
digline is an MCP server for coding agents to inspect and run LLM evaluation suites in a repository. It returns verdict-focused results and requires an explicit call-count acknowledgement before running a suite; baseline promotion is not available.
Key features
- 01List runs and inspect baselines
- 02Compare runs and explain verdicts
- 03Run evaluation suites with call-count acknowledgement
- 04Return verdict-only results
- 05No baseline promotion tool
Use cases
- 01Review stored evaluation results
- 02Compare a run with its baseline
- 03Measure a suite while acknowledging planned calls
MCP support
The server exposes eight tools for reading digline runs and measuring a suite. It serves one repository per instance and does not expose a tool to promote a baseline.
Configuration
Local server config is the same for every client. Copy it into your client’s MCP config file.
Tools
Local servers expose tools only once running on your machine, and the maintainer has not provided a tool list.
Details
From the project READMEdigline-mcp
The MCP server for digline: it lets a coding agent read a digline result and measure a new one, and it does not let the agent promote a baseline.
Not because promotion is refused. Because there is no such tool.
The eight tools
| tool | what it does |
|---|---|
list_runs | every stored run of a suite, newest first, with the baseline marked |
get_run | one stored run — the verdicts, never the payload |
get_baseline | the approved reference, same shape |
log | which model answered, read down the stored runs; never a gate |
compare | a run against the baseline: did it get worse? |
diff | two runs, neither of them a reference: should I switch? |
explain | one run read back as typed facts, with the exit code |
run | execute the suite, with the call count acknowledged first |
promote is absent by construction, and so are migrate, view and
report. A baseline is an approved reference, and promote writes into
.digline/<tenant>/baselines/, which is committed — anything landing there
arrives in somebody's diff and must arrive because they put it there. A refusal
would be a conversation an agent could argue with; an absence is not.
Every tool, the reading ones included, loads the suite it is named. For a
suite.py that executes the project's code, in the server's environment.
The reasoning is ADR 0011, and the
judgement layer it turns into surface is AGENTS.md.
Install and configure
uv pip install digline-mcp
One server serves one repository: the perimeter is the repo, and multi-project is N named servers rather than a registry.
{
"mcpServers": {
"digline-northwind": {
"command": "digline-mcp",
"args": ["--root", "/Users/you/src/northwind"]
}
}
}
--tenant and --env are accepted and verify; they never override. The
suite decides, as it does everywhere else in digline.
What run costs, before it costs it
run takes a mandatory acknowledge_calls that must equal the suite's planned
calls to the target. Called without it, the tool refuses and hands back the
number:
run(suite="suite.py")
→ refused: this suite plans 100 calls to the target.
20 cases × 5 samples = 100 calls to the target; each answer is judged
3 times by llm_rubric.
Call again with acknowledge_calls=100.
run(suite="suite.py", acknowledge_calls=100) → executes
The first call is the probe. An agent cannot spend a hundred model calls without
having stated the number — AGENTS.md §7 as a contract rather than as advice.
What never crosses
An MCP response goes into a model's context, and from there into transcripts and caches nobody controls. So the run tools return a verdict-only projection: the name, the identity, the status, the score, the threshold, the tolerance, the measured interval, and the metadata a suite disclosed.
Never the judge's reason — the judge quotes the output, so the reason is the
output. Never the sentence explaining a suspension, never undisclosed metadata,
never a case's inputs, and never a prompt or its digest unless the suite
declares Disclosure(artifacts=True).
A declared file outside the perimeter is refused when the run is made; one under
.digline or .git is refused by the command that would let it cross. Neither is
dropped in silence.
Comments
New to MCP?
From what the protocol is to installing your first server, these answers cover most first questions.
An open protocol that defines how AI applications connect to external tools and data in a uniform way. Clients talk to the model; MCP servers provide tools, resources and prompt templates. They exchange standard messages, so one server can be reused by many clients.