mcpField
C
cc-canary

cc-canary

github.com/delta-hqTesting & quality7646 installsUpdated 27w ago
Repository

About cc-canary

cc-canary is a skill for assessing whether Claude has regressed on a user's work. A bundled script analyzes session data and produces a Markdown report skeleton, with findings grouped into model-side, user-side, and ambiguous factors.

Key features

  1. 01Analyze sessions across a chosen window
  2. 02Detect inflections and compare versions
  3. 03Track prompt and reasoning-depth changes
  4. 04Render a Markdown report skeleton
  5. 05Classify findings into three buckets

Use cases

  1. 01Check whether Claude's recent performance has declined
  2. 02Compare versions on the same user's work
  3. 03Relate project or prompt shifts to metric changes

Install

Copy the command and run it in a terminal.

$npx skills add https://github.com/delta-hq/cc-canary --skill cc-canary

After installing, type/cc-canaryin Claude Code to confirm it loaded.

Bundled filesscripts/compute_stats.py

Details

From the project README

cc-canary — long-form regression writeup

Primary question: has Claude regressed on this user's work, and when?

Bundled script scripts/compute_stats.py does ~95% of the work in ~2.5s: scans JSONLs, runs inflection + transition-day detection, builds pre/post aggregates, cross-version comparison, hour-of-day, word frequency, three- period thinking depth, visibility transition, per-turn rates, and abnormalities — then renders a complete markdown skeleton with every table filled. Narrative slots are marked <!-- C: ... -->.

Default window: 60d. Accept 7d / 14d / 30d / 60d / 90d / 180d.

Framing — three-bucket classification

  1. Model-side — same user/task, worse outcomes; cross-version worse; reasoning-depth dropping
  2. User-side — project-mix shift; shorter/imperative prompts; new codebase; shortcut vocab rising
  3. Ambiguous — mixed confounds, borderline effect size, either-way-explainable

Your 3-step job

1. Run the script
Bash(python3 <SKILL_DIR>/scripts/compute_stats.py --window {window} --render-md /tmp/cc-canary-skeleton-{window}.md > /dev/null 2>&1)

<SKILL_DIR>: .claude/skills/cc-canary/ → fallback to ~/.claude/skills/cc-canary/.

Flags: --window {Nd} (required); --include-agents (include subagent sessions); --min-user-words N (default 10).

If the script fails: report error, retry once with --include-agents, else stop. Never fall back to hand-computation — that's the slow path.

2. Read the skeleton
Read /tmp/cc-canary-skeleton-{window}.md
3. Fill every <!-- C: ... --> placeholder and save
Write ./cc-canary-{YYYY-MM-DD}.md

End your message with the absolute path: Wrote /Users/.../cc-canary-{date}.md · paste-ready.

Narrative placeholders

Each placeholder's inline comment already spells out what to write. Summary:

  • verdict-line — HOLDING / SUSPECTED REGRESSION / CONFIRMED REGRESSION / INCONCLUSIVE + brief justification
  • summary — 2–4 sentences: verdict + inflection + biggest pre→post delta. No counter-evidence hedging.
  • timeline — 1–2 paragraphs on the daily series shape
  • xv-para — 1 paragraph on cross-version (if §3 present)
  • finding-N-class × up to 5 — inline classification label
  • finding-N-reason × up to 5 — 2–3 sentences max, evidence-first. Lead with the strongest number (cross-version Δ, §2 value, appendix rate). No signal-line restating, no rhetorical buildup
  • root-cause — 3–5 paragraphs tying strongest signals together
  • what-would-help — 2–4 concrete bullets
  • appendix-a1…a4, b, c, d, e, f, g, h — 1 paragraph each (see table context in skeleton)
  • meta-note — 2–5 sentences, first person, honest, no claimed feelings, acknowledge the recursion

Verdict calibration

  • HOLDING: ≤1 model-side signal
  • SUSPECTED REGRESSION: 2–3 model-side signals
  • CONFIRMED REGRESSION: ≥3 model-side signals + non-empty cross-version showing decline + session_count ≥ 15 + ≥2 models + inflection.gap_sigma ≥ 1.0
  • INCONCLUSIVE: session_count < 15 OR inflection.method == "fallback_split_half" with overlapping confounds

Cap at SUSPECTED when: only one model; <15 sessions; single-project with project starting mid-window; inflection coincides with a visible user-side event.

All the data you need (session_count, model mix, inflection method, cross-version presence) is rendered as plain text in the skeleton.

Grounding example (for finding-N-reason)

Classification: model-side. Read:Edit dropped 9.0 → 1.0 (-89%, concerning) while cross-version shows opus-4-7 at 0.39 vs opus-4-6 at 1.00 on the same user's workload. No project-mix shift near the inflection — model is defaulting to edit-first.

Hard rules

  • Never read, grep, or glob ~/.claude/projects/**/*.jsonl. Never run jq/awk/wc on session files. Script owns all that.
  • Never touch tables or numbers — they came from real data.
  • Every finding gets a classification label.
  • Hedge when cross-version is empty or session_count < 15.
  • Do not verdict CONFIRMED REGRESSION without the full checklist.
  • Do not save the skeleton as-is — replace every <!-- C: ... --> first.

Failure modes

  • Script import error → check python3 -V ≥ 3.8; retry once with --include-agents; else stop.
  • Skeleton < 5KB → likely no sessions in window. Check script error.
  • inflection.method == fallback_split_half → state it; cap at SUSPECTED.
  • Cross-version Δ None → div-by-zero when model-A value is 0; note the confound.

Comments

Markdown supported