Flaky Test Triage
FeaturedOn demandAbout Flaky Test Triage
A manually started loop for a named failing test suite. Each round runs the failing tests, labels every failure as flaky or a real regression, fixes only the real regressions, then reruns the suite to check stability. It runs at most 5 rounds.
Key features
- 01Manual trigger, capped at 5 rounds
- 02Changes code only for real regressions, not flaky tests
- 03Regressions that cannot be fixed now can be explicitly deferred
- 04Supports Claude Code and Cursor
Use cases
- 01A test suite on CI passes and fails unpredictably
- 02Several failures appear after a merge and the cause is unclear
- 03Clearing test failures before a release
- Goal
- Every failing test is classified as flaky or a real regression, and each real regression is fixed or explicitly deferred.
- Exit when
- Stop when every failure is classified and real regressions are fixed or explicitly deferred, or after 5 rounds.
- Between iterations run
- npm test -- --testPathPattern=<failing-suite>
Launch prompt
Paste into Claude Code / Cursor. The agent repeats the steps below until the exit condition is met.
Start the "Flaky Test Triage" loop. Goal: Every failing test is classified as flaky or a real regression, and each real regression is fixed or explicitly deferred. Max iterations: 5. Step 1: Run failing tests — Run the failing suite selected with --testPathPattern and record which tests fail and with what errors. `npm test -- --testPathPattern=<failing-suite>` Step 2: Classify each failure — For each failure, judge whether the same code sometimes passes and sometimes fails. Inconsistent results are flaky; failures that reproduce reliably are real regressions. Step 3: Fix real regressions — Change only the code behind real regressions. For any that cannot be handled now, note the reason and mark it as deferred. Step 4: Confirm stability — Run the same suite again and check that results are consistent after the fixes. Unclassified failures send the loop into the next round. `npm test -- --testPathPattern=<failing-suite>` Between iterations run: `npm test -- --testPathPattern=<failing-suite>` Exit when: Stop when every failure is classified and real regressions are fixed or explicitly deferred, or after 5 rounds.
Steps
Each round runs in order; if the exit condition isn’t met, another round starts, up to 5.
Run the failing suite selected with --testPathPattern and record which tests fail and with what errors.
npm test -- --testPathPattern=<failing-suite>For each failure, judge whether the same code sometimes passes and sometimes fails. Inconsistent results are flaky; failures that reproduce reliably are real regressions.
Change only the code behind real regressions. For any that cannot be handled now, note the reason and mark it as deferred.
Run the same suite again and check that results are consistent after the fixes. Unclassified failures send the loop into the next round.
npm test -- --testPathPattern=<failing-suite>Supported agents
Coding agents that can run this loop directly.