mcpField
P
principle-explain-the-number

principle-explain-the-number

github.com/ericlitmanTesting & quality53213 installsUpdated 6d ago
Repository

About principle-explain-the-number

A skill for evaluating performance and evaluation results by identifying the limiter, ruling out alternative explanations, and recording supporting evidence. It helps teams avoid overstating what a benchmark or trial shows.

Key features

  1. 01Identify the resource limiting a result
  2. 02Rule out alternative explanations with evidence
  3. 03Record run count, spread, and limiter
  4. 04Apply checks to benchmarks and evals

Use cases

  1. 01Investigate why performance did not improve further
  2. 02Check whether a benchmark result reflects completed work
  3. 03Compare eval outcomes across trials and models

Install

Copy the command and run it in a terminal.

$npx skills add https://github.com/ericlitman/open-pstack --skill principle-explain-the-number

After installing, type/principle-explain-the-numberin Claude Code to confirm it loaded.

Details

From the project README

Explain the Number

  • Ask "why not double?" Name the resource or code path that bounds the result, such as a core, a lock, the disk, the network, or the load generator itself. Get it from a profile or from system counters taken during a run, then map it to source. A guess from reading the code is not a limiter. If you cannot say why the number is not twice as good, you do not know what you measured.
  • List what else the number could be measuring, and rule out each one with evidence. The usual suspects are errors, skipped or cached work, code that never ran, an untuned side (one left on default settings), noise, and a piece too small to matter end to end.
  • Keep the evidence with the number. Put the run count, the spread, and the limiter in the notes or a linked artifact.

For a performance number, run the full procedure with the benchmark-checklist skill. For an eval result, ask the same of the trials: did every run do the task, does the gap hold across trials and models, and does the scenario matter.

You skipped this when the evidence behind a number has no run count, no spread, or no named limiter, or when the time saved is larger than the time the changed piece took.

Distinct from Prove It Works, which checks that an output is real.

Comments

Markdown supported