mcpField
M
model-eval-error-analysis

model-eval-error-analysis

github.com/jurgendnTesting & quality1164 installsUpdated 6w ago
Repository

About model-eval-error-analysis

A workflow skill for analyzing model evaluation results beyond a single headline metric. It helps ML researchers and practitioners verify metrics, examine performance slices, classify failures, and identify next steps.

Key features

  1. 01Metric and data-split verification
  2. 02Performance slicing by relevant factors
  3. 03Manual failure review and clustering
  4. 04Separation of noise, failures, and evaluator bugs
  5. 05Baseline comparisons and follow-up actions

Use cases

  1. 01Investigating a model's weak evaluation slices
  2. 02Checking whether a reported gain hides regressions
  3. 03Preparing evaluation findings for a paper or deployment claim

Install

Copy the command and run it in a terminal.

$npx skills add https://github.com/jurgendn/agent-skills --skill model-eval-error-analysis

After installing, type/model-eval-error-analysisin Claude Code to confirm it loaded.

Details

From the project README

Model Eval Error Analysis

Do not stop at one headline metric.

Intake

Collect:

  • task, dataset, split, metric, and baseline;
  • model outputs or failure examples if available;
  • aggregate numbers with seeds/uncertainty;
  • deployment or paper claim the evaluation must support;
  • known risky slices.

Workflow

  1. Verify the metric implementation and data split.
  2. Report the headline result with provenance.
  3. Slice performance by relevant factors such as:
    • difficulty
    • length / scale
    • class imbalance
    • domain shift
    • prompt family
    • robustness setting
  4. Inspect failures manually and cluster them.
  5. Separate:
    • random noise
    • systematic failure modes
    • evaluator bugs
  6. Compare against the strongest baseline on the same slices.
  7. End with concrete next actions.

Rules

  • Never smooth away inconvenient variation.
  • If confidence intervals or multiple seeds matter, say so.
  • A higher average may still hide a worse model for important slices.

Output shape

Return:

  • Metrics checked
  • Slice analysis
  • Failure taxonomy
  • Suspected causes
  • Recommended fixes or follow-up experiments

Comments

Markdown supported