flakyguard

flakyguard

"That test is flaky" is usually a feeling. flakyguard turns it into evidence. Feed it the per-test outcomes from a set of CI runs and it builds the record for every test:

• Per-test history — how many runs contained the test, and how many times it passed, failed, or was skipped
• Evidence runs — passes plus failures; a skip is recorded but never counts as evidence, because a test that did not run proves nothing either way
• Classification — a test needs a minimum amount of evidence (two evidence runs by default) before it earns a verdict: stable_pass if it never failed, stable_fail if it never passed, flaky if it did both, and insufficient_data until then
• Flakiness rate — the minority outcome's share of the evidence runs, so a test that splits its results evenly rates 0.5 and a mostly reliable one rates close to 0
• Quarantine list — the flaky tests whose rate meets the quarantine threshold (0.5 by default), worst first, each entry carrying the pass/fail counts behind its rate
• Verdict — quarantine_recommended when the list is non-empty, otherwise clean

Rates are rounded to four decimal places. Everything is computed locally from the runs you supply: no network, no configuration, no history kept between calls. Bad input — a run with no results, a status outside passed / failed / skipped, the same test twice in one run — is rejected with an error naming the offending path, and no partial analysis is produced.

CLI

npx @reneturcios/flakyguard analyze --input args.json
npx @reneturcios/flakyguard --version

args.json contains the full arguments object:

{
  "runs": [
    {
      "run_id": "run-101",
      "results": [
        { "test": "checkout totals", "status": "passed" },
        { "test": "search ranking", "status": "failed" }
      ]
    },
    {
      "run_id": "run-102",
      "results": [
        { "test": "checkout totals", "status": "failed" },
        { "test": "search ranking", "status": "failed" }
      ]
    }
  ],
  "min_runs": 2,
  "quarantine_threshold": 0.5
}

min_runs and quarantine_threshold are optional. The analysis is printed as JSON.

Exit codes: 0 on success, 1 on a runtime error (including an unreadable or invalid input file, or invalid input), 2 on a usage error.

MCP server

npx -y @reneturcios/flakyguard mcp

MCP client config:

{
  "mcpServers": {
    "flakyguard": {
      "command": "npx",
      "args": ["-y", "@reneturcios/flakyguard", "mcp"]
    }
  }
}

Tools

• analyze_test_flakiness — Analyze test flakiness across CI runs.

  • runs (array, required, at least one) — runs as {run_id, results} objects, where each result is {test, status} and status is one of passed, failed, skipped; a test may not appear twice in one run
  • min_runs (integer, optional, >= 1, default 2) — minimum evidence runs before a test is classified as stable or flaky
  • quarantine_threshold (number, optional, 0 to 1, default 0.5) — flakiness-rate threshold for the quarantine list

  Returns {run_count, test_count, tests, quarantine, summary, verdict}.

Docker

docker run -i ghcr.io/cloudcorprecords/flakyguard
docker run --rm -v "$PWD:/w" ghcr.io/cloudcorprecords/flakyguard analyze --input /w/args.json

Development

npm install
npm run build   # tsc -> dist/
npm test        # tsx --test test/*.test.ts (pure local, no network)
npm run dev     # run the CLI from source

License

MIT — Copyright (c) 2026 Rene Turcios