fable-eval

Fable Eval

Measure whether an agent-control change improves the behavior it claims to improve without quietly overfitting the benchmark or breaking neighboring behavior.

Mission

An eval is a decision instrument, not a scoreboard. It needs a frozen comparison point, representative semantic families, oracle isolation, explicit failure costs, and a rollback decision.

A candidate should not win because the prompts resemble its instructions, because holdouts leaked into authoring, or because one average score hides a severe regression.

Activate When

• changing Skills, prompts, routers, hooks, agent profiles, policies, or model-control logic;
• comparing candidate prompt/agent configurations;
• measuring trigger precision/recall or action compliance;
• validating a new behavioral maturity claim;
• investigating whether an apparent improvement is robust or benchmark-specific.

Do Not Activate When

• verifying ordinary application behavior (fable-verify);
• authoring a Skill before its intended behavior is clear (skill-creator);
• running a one-off subjective prompt demo with no acceptance decision.

Evaluation Classification

Change	Primary eval risk
Router/trigger	false positives, false negatives, precedence
Skill instruction	action correctness, forbidden shortcuts, boundary behavior
Spark/next-action	top-1 action, unsafe suggestion, silence precision
Hook/guard	enforcement, false blocking, bypasses
Prompt/persona	task quality + regressions + instruction conflicts
Tool policy	correct tool choice, unsafe/missing action
Model/config	quality/latency/cost variance across representative tasks

Protocol

Stage 1 — Define the decision before running tests

State:

• candidate being evaluated;
• baseline/control;
• exact behavior expected to improve;
• metrics and thresholds;
• unacceptable regressions;
• rollback action.

Avoid inventing metrics after seeing results.

Stage 2 — Build semantic scenario families

Cover distinct decisions, not wording variants. Include as applicable:

• straightforward positive case;
• non-trigger/boundary case;
• ambiguous competing action;
• adversarial shortcut pressure;
• partial/contradictory evidence;
• failure/recovery path;
• legacy/constrained environment;
• unseen holdout.

Record family coverage separately from raw prompt count.

Stage 3 — Freeze baseline and corpus identity

Bind the run to:

• corpus hash/version;
• candidate/baseline identity;
• evaluator/scorer version;
• provider/model/config where external;
• timestamp/repository revision.

Changing the subject, oracle, corpus, or scoring logic invalidates direct comparability unless explicitly normalized.

Stage 4 — Protect the oracle

Provider-facing requests must not reveal expected actions, forbidden actions, category labels, holdout identity, scoring implementation, or answer-bearing metadata.

Do not draft the candidate while repeatedly reading holdout failures. Promote discovered cases into a future checked corpus and preserve a new unseen holdout.

Stage 5 — Execute baseline and candidate consistently

Use the same task inputs, tool availability, context budget, temperature/configuration, timeout policy, and scoring rules where comparison requires them.

Capture provider errors/timeouts as failures or explicit unavailable states; do not fill missing outputs from the oracle.

Stage 6 — Score by slices, not average alone

Inspect:

• overall metric;
• each semantic family;
• negative/adversarial forbidden violations;
• high-cost regressions;
• variance/repeated-run stability where stochasticity is material;
• routing confusion pairs where applicable.

A 1% average gain is not acceptable if it introduces a severe release/security/recovery regression.

Stage 7 — Investigate suspicious gains

Check for:

• prompt leakage;
• duplicated/near-duplicate scenarios;
• benchmark-specific keyword matching;
• changed tool/context budget;
• scorer drift;
• cherry-picked seeds/runs;
• examples copied into the candidate.

Stage 8 — Decide and preserve rollback

Verdict:

• ACCEPT: thresholds met, no prohibited regression, evidence representative/fresh;
• REJECT: candidate regresses required behavior or fails threshold;
• INCONCLUSIVE: evidence lacks breadth/stability/holdout integrity.

Record baseline artifact so rollback remains possible.

Decision Rules

• Semantic family breadth matters more than raw scenario count.
• Surface rewrites of one case do not create independent coverage.
• Holdouts stop being holdouts once used repeatedly to tune the candidate.
• Compare slices before averages; safety-critical forbidden violations can veto a higher average score.
• If stochastic variance could change the decision, repeat enough runs to estimate stability rather than cherry-picking one seed.
• If provider/runtime errors differ between candidate and baseline, separate infrastructure failure from behavior score.
• Never preserve an old maturity result after the evaluated Skill/corpus/oracle changes unless freshness validation proves identity.
• Do not lower thresholds after a candidate fails simply to ship it.

Invariants

• Baseline is reproducible/frozen before candidate judgment.
• Oracle/holdout data remains hidden from the evaluated agent.
• Candidate and baseline are compared under equivalent conditions where claimed.
• Missing/failed provider outputs are never replaced by expected answers.
• Every acceptance decision has a rollback path.
• High-cost regressions remain visible even when aggregate score improves.

Failure Taxonomy

Benchmark overfit

Candidate improves checked cases but fails unseen family/holdout. Increase semantic breadth and reject/generalize candidate.

Oracle leakage

Expected/forbidden/category data reaches provider/candidate authoring loop. Discard contaminated evidence and create fresh blind cases.

Metric blindness

Average improves while important slice worsens. Use per-family and veto metrics.

Non-comparable runs

Different model/tool/context/scorer settings produce apparent gain. Re-run under controlled conditions.

High variance

Repeated runs change verdict. Increase samples/control nondeterminism or mark inconclusive.

Corpus drift

Skill/scenario/oracle changed after evidence capture. Mark evidence stale and rerun.

Anti-Patterns

• five paraphrases counted as five independent tests;
• reading holdouts while tuning every candidate;
• accepting on average score alone;
• changing thresholds after seeing failure;
• treating provider timeout as skipped rather than failed/incomplete;
• evaluating only positive examples;
• copying expected action vocabulary in a way that reveals the answer per case;
• claiming M4 because an evidence JSON file exists even though corpus hash changed.

Eval Report

Candidate / baseline:
Corpus + hashes:
Provider/config:
Semantic families:
Metrics + thresholds:
Per-family results:
Forbidden/high-cost regressions:
Variance/repeats:
Leakage/comparability checks:
Verdict: ACCEPT | REJECT | INCONCLUSIVE
Rollback:
Evidence freshness:

Completion Criteria

Evaluation completes when:

• baseline and candidate identities are explicit;
• semantic coverage and blind holdout integrity are credible;
• metrics are inspected by meaningful slices;
• regressions/forbidden actions are not hidden by averages;
• verdict and rollback are evidence-backed;
• evidence freshness is tied to the exact evaluated corpus/control.

Progressive Resources

• Deep guide: references/benchmark-design-and-overfit-control.md
• Existing protocol: references/eval-harness-protocol.md
• Example: examples/eval-regression-run.md