mcpField
H
honey-gain

honey-gain

github.com/green-ptAgent workflows34440 installsUpdated 1w ago
Repository

About honey-gain

Honey Gain is an agent skill for reporting reproducible benchmark results from committed records. It emphasizes per-task statistics, test pass rates, and separating measured findings from unsupported claims.

Key features

  1. 01Recomputes committed benchmark results
  2. 02Reports per-tier LOC and output deltas
  3. 03Includes p-values and test pass rates
  4. 04Distinguishes ties from wins

Use cases

  1. 01Summarizing benchmark outcomes by variant
  2. 02Checking whether a reported result is statistically supported

Install

Copy the command and run it in a terminal.

$npx skills add https://github.com/green-pt/honey-for-devs --skill honey-gain

After installing, type/honey-gainin Claude Code to confirm it loaded.

Details

From the project README

Honey Gain

Report the committed benchmark results — never a guessed or per-session number, and never an embedded copy that can drift from the bench.

Do

  1. Recompute from the committed records at use time — don't recite from memory, and prefer the raw records over any rendered table (renderings go stale, the records don't):

    cd bench && node src/report.js --stamp full-opus48 --by-type

    Offline, no API spend. Swap --stamp full-gpt55 for the cross-provider figure, drop --by-type for the whole suite. Hive handoff numbers → bench/hive/RESULTS.md.

  2. Report the tier table terse: Δ LOC and Δ output, each with its p, judge as win/loss/tie, and the test pass-rate, per variant. The tier split is the finding — deepest on code and handoffs, output a statistical tie on user-facing (the polish carve-out). Lead with Δ LOC: it measures Lever 1 directly, while output tokens mix code with the prose around it, and the two come apart (Ponytail cuts lines but narrates at length).

Rules

  • Never quote a delta without its p-value, and call (ns) results ties, not wins. Every figure is a paired per-task median; a ratio of arm totals is not quotable.
  • Prose renderings (bench/README.md, results/combined.md) are secondary. If one disagrees with a fresh --stamp recompute, the recompute wins — say the rendering is out of sync.
  • Quality is a tie, not a gain — that's the honest claim. Don't upgrade it.
  • Cost/CO₂ savings are a modelled counterfactual, not measured; those belong to honey-eco, which labels them. Don't state a dollar saving here.
  • Asked for numbers on this repo? The bench measures the skill on a fixed task suite, not the user's codebase — offer cd bench && npm run bench, don't extrapolate.
  • One honest caveat, once: 23 author-written tasks, judge noise — the objective test-pass column is the trustworthy correctness signal.
  • Never resurrect the old unreproducible 92%/78%/73% / −57%/−65%/−70% numbers, or the superseded arm-total figures (−49% code, −15% aggregate) — see bench/METHODOLOGY.md.

Comments

Markdown supported