mcpField
H
harness-bench

harness-bench

github.com/ruvnetTesting & quality80.0k203 installsUpdated today
Repository

About harness-bench

harness-bench creates and verifies fixed task suites for developers evaluating harness evolution. It supports comparable scoring across commits or forks, with graceful handling when its dependency is unavailable.

Key features

  1. 01Create suites from existing tests
  2. 02Verify suite task structure
  3. 03Check suites in CI
  4. 04Degrade gracefully without dependency

Use cases

  1. 01Set up a fixed benchmark for flaky tests
  2. 02Validate a suite in each relevant PR
  3. 03Retarget evaluation for a domain fork

Install

Copy the command and run it in a terminal.

$npx skills add https://github.com/ruvnet/ruflo --skill harness-bench

After installing, type/harness-benchin Claude Code to confirm it loaded.

Details

From the project README

Surfaces metaharness-darwin bench <create|verify> — the supporting verb for harness-evolve --bench. Use when you want evolution scored against a fixed corpus (independent of npm test) so champion fitness is comparable across commits or across forks of the same harness.

When to use

  • Setting up a new evolution pipeline for a repo whose npm test is flaky, slow, or undersized — scaffold a deterministic bench suite once, then evolve against it repeatedly.
  • CI: bench verify the checked-in suite on every PR that touches it (cheap; ~5s).
  • Forking a harness to a new domain: copy and edit the suite to retarget the evaluation without losing comparability to the parent.

Algorithm

Implementation: scripts/bench.mjs.

--op create
  1. Resolve --repo path; reject if missing.
  2. Shell to metaharness-darwin bench create <repo> [--out <suite.json>].
  3. Default output path: <repo>/.metaharness/bench/suite.json (chosen by upstream).
  4. Suite shape (per upstream): array of { input, expectedOutput, weight } tasks derived from existing test cases.
--op verify
  1. Resolve --suite path; reject if missing.
  2. Shell to metaharness-darwin bench verify <suite.json>.
  3. Exit 1 if any task malformed (upstream's signal).

Output shape

{
  "success": true,
  "data": {
    "op": "verify",
    "taskCount": 42,
    "wellFormed": true,
    "durationMs": 870
  }
}

Exit codes

CodeMeaning
0OK (or degraded — Darwin absent)
1--op verify and suite malformed
2Config error or upstream invocation failure

Graceful degradation

When @metaharness/darwin is absent, emits the standard {degraded: true, reason: 'metaharness-darwin-not-available'} payload and exits 0.

Comments

Markdown supported