opendeviation-eval-metrics

Open Deviation Bar Evaluation Metrics

Machine-readable reference + computation scripts for state-of-the-art metrics evaluating open deviation bar (ODB, brim-to-brim price-based sampling) data.

Cross-reference: Project-level experiment catalogue at signal-archaeology skill in the project's research repo — contains 10 BHR-validated experiments with auditable SQL.

Self-Evolving Skill: This skill improves through use. If instructions are wrong, parameters drifted, or a workaround was needed — fix this file immediately, don't defer. Only update for real, reproducible issues.

When to Use This Skill

Use this skill when:

• Evaluating ML model performance on open deviation bar data
• Computing Sharpe ratios with non-IID bar sequences
• Running Walk-Forward Optimization metric analysis
• Calculating PSR or MinTRL statistical tests
• Generating evaluation reports from fold results

No trial-count corrections (operator rule, 2026-10-15): never compute, report, rank by or cite the Deflated Sharpe Ratio, the minimum backtest length (MinBTL) or any other trial-count correction (FWER/FDR procedures such as Holm or Benjamini-Yekutieli, e-LOND and other online multiple testing). Judge parsimony on the model itself, guard selection with registration before runs and with unused data, and report each rule's own statistics (Sharpe, PSR, MinTRL). They were removed from this skill on purpose; do not re-add them. PBO/CSCV is not covered by the rule.

Quick Start

# Compute metrics from predictions + actuals
python scripts/compute_metrics.py --predictions preds.npy --actuals actuals.npy --timestamps ts.npy

# Generate full evaluation report
python scripts/generate_report.py --results folds.jsonl --output report.md

Metric Tiers

Tier	Purpose	Metrics	Compute
Primary (5)	Research decisions	weekly_sharpe, hit_rate, cumulative_pnl, n_bars, positive_sharpe_rate	Per-fold + aggregate
Secondary/Risk (5)	Additional context	max_drawdown, bar_sharpe, return_per_bar, profit_factor, cv_fold_returns	Per-fold
ML Quality (3)	Prediction health	ic, prediction_autocorr, is_collapsed	Per-fold
Diagnostic (4)	Final validation	psr, autocorr_lag1, effective_n, binomial_pvalue	Aggregate only
Extended Risk (5)	Deep risk analysis	var_95, cvar_95, omega_ratio, sortino_ratio, ulcer_index	Per-fold (optional)

Why Open Deviation Bars Need Special Treatment

Open deviation bars violate standard IID assumptions:

1. Variable duration: Bars form based on price movement, not time
2. Autocorrelation: High-volatility periods cluster bars → temporal correlation
3. Non-constant information: More bars during volatility = more information per day

Canonical solution: Daily aggregation via _group_by_day() before Sharpe calculation.

References

Core Reference Files

Topic	Reference File
Sharpe Ratio Calculations	sharpe-formulas.md
Risk Metrics (VaR, Omega, Ulcer)	risk-metrics.md
ML Prediction Quality (IC, Autocorr)	ml-prediction-quality.md
Crypto Market Considerations	crypto-markets.md
Temporal Aggregation Rules	temporal-aggregation.md
JSON Schema for Metrics	metrics-schema.md
Anti-Patterns (Transaction Costs)	anti-patterns.md
SOTA 2025-2026 (SHAP, BOCPD, etc.)	sota-2025-2026.md
Beyond Hit Rate (BHR) Framework	beyond-hit-rate.md
Worked Examples (BTC, EUR/USD)	worked-examples.md
Structured Logging (NDJSON)	structured-logging.md

Related Skills

Skill	Relationship
sharpe-ratio-non-iid-corrections	ρ-corrected PSR, MinTRL, power — full non-IID single-test framework
adaptive-wfo-epoch	Uses weekly_sharpe, psr for WFE calculation

Dependencies

pip install -r requirements.txt
# Or: pip install numpy>=1.24 pandas>=2.0 scipy>=1.10

Key Formulas

Daily-Aggregated Sharpe (Primary Metric)

def weekly_sharpe(pnl: np.ndarray, timestamps: np.ndarray) -> float:
    """Sharpe with daily aggregation for open deviation bars."""
    daily_pnl = _group_by_day(pnl, timestamps)  # Sum PnL per calendar day
    if len(daily_pnl) < 2 or np.std(daily_pnl) == 0:
        return 0.0
    daily_sharpe = np.mean(daily_pnl) / np.std(daily_pnl)
    # For crypto (7-day week): sqrt(7). For equities: sqrt(5)
    return daily_sharpe * np.sqrt(7)  # Crypto default

Information Coefficient (Prediction Quality)

from scipy.stats import spearmanr

def information_coefficient(predictions: np.ndarray, actuals: np.ndarray) -> float:
    """Spearman rank IC - captures magnitude alignment."""
    ic, _ = spearmanr(predictions, actuals)
    return ic  # Range: [-1, 1]. >0.02 acceptable, >0.05 good, >0.10 excellent

Probabilistic Sharpe Ratio (Statistical Validation)

from scipy.stats import norm

def psr(sharpe: float, se: float, benchmark: float = 0.0) -> float:
    """P(true Sharpe > benchmark)."""
    return norm.cdf((sharpe - benchmark) / se)

Annualization Factors

Market	Daily → Weekly	Daily → Annual	Rationale
Crypto (24/7)	sqrt(7) = 2.65	sqrt(365) = 19.1	7 trading days/week
Equity	sqrt(5) = 2.24	sqrt(252) = 15.9	5 trading days/week

NEVER use sqrt(252) for crypto markets.

CRITICAL: Session Filter Changes Annualization

View	Filter	days_per_week	Rationale
Session-filtered (London-NY)	Weekdays 08:00-16:00	sqrt(5)	Trading like equities
All-bars (unfiltered)	None	sqrt(7)	Full 24/7 crypto

Using sqrt(7) for session-filtered data overstates Sharpe by ~18%!

See crypto-markets.md for detailed rationale.

Dual-View Metrics

For comprehensive analysis, compute metrics with BOTH views:

1. Session-filtered (London 08:00 to NY 16:00): Primary strategy evaluation
2. All-bars: Regime detection, data quality diagnostics

Academic References

Concept	Citation
Probabilistic Sharpe Ratio	Bailey & López de Prado (2012)
Sharpe SE with Non-Normality	Mertens (2002)
Statistics of Sharpe Ratios	Lo (2002)
Omega Ratio	Keating & Shadwick (2002)
Ulcer Index	Peter Martin (1987)

Beyond Hit Rate (BHR) Framework

Hit rate is a necessary but insufficient metric. Always supplement with outcome predictability metrics. See beyond-hit-rate.md for the full framework.

Minimum Viable Signal Evaluation

Every signal MUST be evaluated with at least:

1. One sequence structure test: entropy, LZC, or runs test on the W/L sequence
2. One temporal decay test: CUSUM on equity curve or rolling hit rate
3. One regime awareness test: per-session hit rate or HMM decomposition

A signal that passes all three is robust. A signal with only high hit rate is noise.

Outcome Predictability Index (OPI)

OPI = 0.25 * (1 - LZC_norm) + 0.25 * |z_runs| + 0.25 * Var(HR_per_regime) + 0.25 * AUC_meta

Higher OPI = more predictable win/loss timing. A 45% HR signal with OPI=0.8 is more valuable than a 65% HR signal with OPI=0.1.

Decision Framework

Go Criteria (Research)

go_criteria:
  - positive_sharpe_rate > 0.55
  - mean_weekly_sharpe > 0
  - cv_fold_returns < 1.5
  - bhr_sequence_test_passes: true # At least one of: entropy, LZC, runs test significant
  - bhr_cusum_verdict: "ALIVE" # No recent regime break

Publication Criteria

publication_criteria:
  - binomial_pvalue < 0.05
  - psr > 0.85
  - registered_before_run: true # Hypothesis, rule and window fixed before results were seen
  - confirmed_on_unused_data: true # Holdout no run or selection step has touched
  - bhr_lzc_shuffle_z < -2.0 # W/L sequence has genuine structure
  - bhr_alpha_halflife > 200 # Edge persists for 200+ trades

Scripts

Script	Purpose
scripts/compute_metrics.py	Compute all metrics from predictions/actuals
scripts/generate_report.py	Generate Markdown report from fold results
scripts/validate_schema.py	Validate metrics JSON against schema

Remediations (2026-01-19 Multi-Agent Audit)

The following fixes were applied based on a 12-subagent adversarial audit:

Issue	Root Cause	Fix	Source
weekly_sharpe=0	Constant predictions	Model collapse detection + architecture fix	model-expert
IC=None	Zero variance predictions	Return 1.0 for constant (semantically correct)	model-expert
prediction_autocorr=NaN	Division by zero	Guard for std < 1e-10, return 1.0	model-expert
Ulcer Index divide-by-zero	Peak equity = 0	Guard with np.where(peak > 1e-10, ...)	risk-analyst
Omega/Profit Factor unreliable	Too few samples	min_days parameter (default: 5)	robustness-analyst
BiLSTM mean collapse	Architecture too small	hidden_size: 16→48, dropout: 0.5→0.3	model-expert
profit_factor=1.0 (n_bars=0)	Early return wrong value	Return NaN when no data to compute ratio	risk-analyst

Model Collapse Detection

# ALWAYS check for model collapse after prediction
pred_std = np.std(predictions)
if pred_std < 1e-6:
    logger.warning(
        f"Constant predictions detected (std={pred_std:.2e}). "
        "Model collapsed to mean - check architecture."
    )

Recommended BiLSTM Architecture

# BEFORE (causes collapse on open deviation bars)
HIDDEN_SIZE = 16
DROPOUT = 0.5

# AFTER (prevents collapse)
HIDDEN_SIZE = 48  # Triple capacity
DROPOUT = 0.3     # Less aggressive regularization

See reference docs for complete implementation details.

---

Troubleshooting

Issue	Cause	Solution
weekly_sharpe is 0	Constant predictions	Check for model collapse, increase hidden_size
IC returns None	Zero variance in predictions	Model collapsed - check architecture
prediction_autocorr is NaN	Division by zero	Guard for std < 1e-10 in autocorr calculation
Ulcer Index divide error	Peak equity is zero	Add guard: np.where(peak > 1e-10, ...)
profit_factor = 1.0	No bars processed	Return NaN when n_bars is 0
Sharpe inflated 18%	Wrong annualization for data	Use sqrt(5) for session-filtered, sqrt(7) for 24/7
PSR not computed	Missing scipy	Install: pip install scipy
Timestamps not parsed	Wrong format	Ensure Unix timestamps, not datetime strings

Post-Execution Reflection

After this skill completes, check before closing:

1. Did the command succeed? — If not, fix the instruction or error table that caused the failure.
2. Did parameters or output change? — If the underlying tool's interface drifted, update Usage examples and Parameters table to match.
3. Was a workaround needed? — If you had to improvise (different flags, extra steps), update this SKILL.md so the next invocation doesn't need the same workaround.

Only update if the issue is real and reproducible — not speculative.