leakguard-mcp

leakguard-mcp

Squawk for backtests. A local-first MCP server that static-analyzes agent-generated Python code and flags lookahead bias & data leakage before the backtest runs.

Works for any time-series ML code — quant trading (crypto, equities, forex, futures), demand forecasting, energy, weather, IoT sensors — wherever a wrong .shift() or a global normalization silently poisons your results.

• Pure source analysis (libcst) — your code never leaves the machine, no API calls.
• Heuristic, not a proof: severity tiers (🔴 error / 🟡 warning), like an aviation squawk.
• Framework-agnostic: pandas / numpy / polars, any stack.
• MCP-native (stdio): works directly inside Claude Code, Cursor, and any MCP-compatible agent.
• All 10 rules ship free — no license, no tiers, no phone-home.

---

Why this exists

AI agents (Claude Code, Cursor) write feature engineering and strategy code faster than humans can review it. But they introduce lookahead bias at scale — subtle time-boundary errors that backtest perfectly and fail catastrophically in live trading or production:

# Agent writes this — looks fine, is catastrophically wrong
df['momentum'] = df['close'].shift(-5)   # uses FUTURE prices as a feature
df['vol_norm'] = (df['close'] - df['close'].mean()) / df['close'].std()  # leaks future mean

leakguard catches these in the same agent loop — before the backtest runs:

LG001 🔴 line 2: Future shift used as feature — shift(-5) references 5 bars ahead.
  Fix: df['momentum'] = df['close'].shift(5)   (lag, not lead)

LG003 🔴 line 3: Global-fit normalization — mean/std computed over the full series
  before any train/test split, leaking future statistics into the past.
  Fix: df['vol_norm'] = (df['close'] - df['close'].expanding().mean()) / df['close'].expanding().std()

The agent reads the finding + fix snippet and self-corrects in one turn. No human review needed.

---

Install

Requirements

• Python 3.11+
• uv (recommended) or pip

From PyPI

pip install leakguard-mcp

From source

git clone https://github.com/doazvjettu/leakguard-mcp
cd leakguard-mcp
uv sync

---

Setup with Claude Code

Add to your MCP config (~/.claude/claude_desktop_config.json or .claude/settings.json in your project):

{
  "mcpServers": {
    "leakguard": {
      "command": "python",
      "args": ["-m", "leakguard.server"]
    }
  }
}

Or if installed via uv:

{
  "mcpServers": {
    "leakguard": {
      "command": "uv",
      "args": ["run", "python", "-m", "leakguard.server"]
    }
  }
}

Restart Claude Code. leakguard's tools are now available to the agent.

Setup with Cursor

Add the same block under mcpServers in your Cursor MCP settings file.

---

MCP Tools

Tool	Description
lint_code(code)	Analyze a code string, return findings
lint_file(path)	Analyze a file on disk
lint_paths(glob)	Analyze all matching files
list_rules()	List all rules with severities
explain_rule(rule_id)	Full rationale + fix patterns for a rule

---

CLI

The same scanner is available as a CLI — handy for a pre-commit hook or CI step (exits non-zero when leakage is found):

uv run leakguard path/to/strategy.py
# or, installed:  leakguard path/to/strategy.py

It prints each finding with its severity, line/col, and a concrete fix snippet — the same output shown in the demo above.

---

Rules

All 10 rules active, no tiers:

ID	Severity	Pattern
LG001	🔴	Future shift as feature: shift(-n) / diff(-n) / pct_change(-n)
LG002	🔴	Centered windows: rolling(center=True)
LG003	🔴	Global-fit scaling: StandardScaler().fit(full_df) / hand-rolled mean-std before split
LG004	🔴	Shuffled time-series split: train_test_split default, KFold, cross_val_score(cv=n)
LG005	🔴	Label leakage: future-derived target column reused in features
LG006	🟡	Whole-history aggregates as features: .max() / .mean() over full series
LG007	🔴	Backfill imputation: bfill() / fillna(method='bfill')
LG008	🔴	Forward asof-joins: merge_asof(direction='forward'/'nearest')
LG009	🟡	Resample label/closed mismatch on bar timestamps
LG010	🟡	groupby().transform()/agg() spanning train/test boundary

Each finding includes a concrete fix snippet so the calling agent can self-correct immediately.

---

Benchmark

Measured on two labeled corpora, 49 snippets total. Reproduce with uv run python -m benchmark.run.

Honesty note: the trading corpus was written by the tool's author — treat its numbers as regression fixtures, not independent validation. The general-ML corpus is one arm's length removed in domain (author-composed reproductions of widely documented leakage anti-patterns, not a downloaded public dataset). The corpus deliberately includes adversarial snippets the scanner is known to miss; they are counted against it.

Trading corpus — 39 snippets (23 leaky, 16 clean + hard negatives):

Rule	Precision	Recall	TP	FP	FN
LG001	75%	100%	6	2	0
LG002	100%	100%	5	0	0
LG003	75%	100%	3	1	0
LG004	100%	100%	4	0	0
LG005	100%	100%	5	0	0
LG006	100%	100%	5	0	0
LG007	100%	100%	5	0	0
LG008	100%	100%	2	0	0
LG009	75%	100%	3	1	0
LG010	100%	100%	2	0	0
Overall	91%	100%	40	4	0

General-ML corpus — 10 snippets (LG003/LG004/LG010): Precision 88%, Recall 100% (TP 7 / FP 1 / FN 0).

Combined: Precision 90.4%, Recall 100% (TP 47 / FP 5 / FN 0).

Recall is 100% on this corpus — every adversarial miss exposed has since been fixed (constant propagation, hand-rolled normalization, cv=<int>, drop-based selection). Leak shapes not yet in the corpus are still missed — see Known Limitations below.

Known false positives (clean code that gets flagged)

• LG001: a forward-return label built with a negative shift and used only as y — pure AST cannot distinguish a target column from a feature.
• LG003: fit inside a helper defined above the split call site — line-order heuristic confuses definition order with execution order.
• LG004: shuffled splits on genuinely non-temporal data — no datetime-index inference yet.
• LG009: resampling for reporting/plotting rather than features — intent is invisible to static analysis.

Known false negatives (leak shapes not yet covered)

• LG004: cross_val_score(...) with cv omitted (defaults to KFold).
• LG005: taint through df.loc[:, 'col'] = ... or df.assign(col=...).
• All rules: values flowing through function calls, dicts, or non-constant variables — no cross-function dataflow.

These sets are pinned in tests/test_benchmark.py: any new miss or silent fix fails the suite until docs and corpus are updated to match.

---

Develop

uv sync --extra dev
uv run pytest                              # 96 tests
uv run python -m leakguard.server          # stdio MCP server
uv run leakguard demo/strategy_leaky.py    # CLI on the demo file
uv run python -m benchmark.run             # precision/recall tables + FP/FN lists

The scanner core lives in leakguard/core/ (pure, no MCP imports); server.py and cli.py are thin wrappers. Each rule has a fixture pair under tests/fixtures/.

---

Limitations (v1)

• Heuristic static analysis — catches ~90% of common patterns, not 100%.
• Single-file only — no cross-module taint tracking.
• Python only (pandas / numpy / polars).
• No runtime execution — cannot catch patterns that only emerge at runtime.

Contributions welcome: new corpus snippets (especially real bugs you've hit) strengthen the benchmark more than new rules do.