mcpField
R
reproduce-regression

reproduce-regression

github.com/trailofbitsTesting & quality7.9k163 installsUpdated today
Repository

About reproduce-regression

reproduce-regression is a skill for testing a suspected regression by comparing live S0 and S1 deployments with the same input. It targets investigators who need a portable proof and a sealed evidence package; synthetic tests and simulated systems are excluded.

Key features

  1. 01Validate inputs and project guidance
  2. 02Exercise real interfaces and deployments
  3. 03Compare S0 and S1 using identical input
  4. 04Record setup, commands, and separate logs
  5. 05Seal reproduction evidence

Use cases

  1. 01Testing a candidate regression across two revisions
  2. 02Checking whether a reported failure reaches a real interface
  3. 03Preparing reproducible evidence for independent replay

Install

Copy the command and run it in a terminal.

$npx skills add https://github.com/trailofbits/skills --skill reproduce-regression

After installing, type/reproduce-regressionin Claude Code to confirm it loaded.

Details

From the project README

Reproduce a candidate

When to Use

Input: sealed preparation/discovery packages, candidate and registry assignment. Follow project guidance before target work.

When NOT to Use

Independent verification of an existing PoC; use replay.

Reproduce

  1. Read the standard and file contract. Verify inputs; read security guidance. Stop execution if prerequisites fail.
  2. Build and run actual S0/S1 with required services. Trace the candidate to a real user/attacker entry point. Build/install CLIs and libraries; exercise supported interfaces. For build findings, execute the supported build workflow.
  3. Create and execute a portable PoC against both live deployments with identical input and comparable configuration. Record whether S0 behaves correctly and S1 exhibits the claimed failure. No mocks, extracted functions or simulated systems.
  4. Save setup, tool/dependency versions, sanitized configuration, input, PoC, revisions, commands and separate logs. Record observations and false-positive checks in reproduce.json. If blocked, retain attempted setup and exact prerequisites. Build and deploy only inside your attempt directory, and list every build directory in build_dirs.
  5. Release temporary resources without deleting evidence. Record their disposition, then seal the package.

Report Back

Keep detailed evidence in the sealed package. Return only:

  • Status: complete or blocked, matching the record.
  • Record: sealed reproduce.json path, or why none was produced.
  • Result: one sentence describing observed S0/S1 behavior.
  • Blocker: missing prerequisite or required caller action, if blocked.

Completion means the experiment finished, not that the regression is confirmed.

Rationalizations to Reject

  • “The unit test proves it”: synthetic tests cannot establish a regression.

Comments

Markdown supported