mcpField
A
ab-test-stats

ab-test-stats

github.com/guia-matthieuData & databases150198 installsUpdated 4w ago
Repository

About ab-test-stats

A Claude skill for analyzing A/B test significance and planning experiments. It provides command-line calculations for significance, sample size, and duration, while leaving metric choices and business decisions to the user.

Key features

  1. 01Significance calculations
  2. 02Sample size estimation
  3. 03Experiment duration estimates
  4. 04Power and MDE concepts

Use cases

  1. 01Assess whether test results are significant
  2. 02Plan sample sizes before a test
  3. 03Estimate test duration from traffic

Install

Copy the command and run it in a terminal.

$npx skills add https://github.com/guia-matthieu/clawfu-skills --skill ab-test-stats

After installing, type/ab-test-statsin Claude Code to confirm it loaded.

Bundled filesscripts/main.pyscripts/requirements.txt

Details

From the project README

A/B Test Statistics Calculator

Calculate statistical significance for A/B tests - know when your results are real, not random chance.

When to Use This Skill

  • Test analysis - Determine if results are statistically significant
  • Sample planning - Calculate required sample size before testing
  • Duration estimation - Know how long to run experiments
  • Power analysis - Ensure tests can detect meaningful differences

What Claude Does vs What You Decide

Claude DoesYou Decide
Structures analysis frameworksMetric definitions
Identifies patterns in dataBusiness interpretation
Creates visualization templatesDashboard design
Suggests optimization areasAction priorities
Calculates statistical measuresDecision thresholds

Dependencies

pip install scipy numpy click

Commands

Check Significance
python scripts/main.py significance --control 1000,50 --variant 1000,65
python scripts/main.py significance --control 5000,250 --variant 5000,300 --confidence 0.99
Calculate Sample Size
python scripts/main.py sample-size --baseline 0.05 --mde 0.02
python scripts/main.py sample-size --baseline 0.10 --mde 0.01 --power 0.90
Estimate Duration
python scripts/main.py duration --traffic 1000 --baseline 0.05 --mde 0.02

Examples

Example 1: Analyze Test Results
# Control: 1000 visitors, 50 conversions (5%)
# Variant: 1000 visitors, 65 conversions (6.5%)
python scripts/main.py significance --control 1000,50 --variant 1000,65

# Output:
# A/B Test Results
# ─────────────────────────
# Control:  5.00% (50/1000)
# Variant:  6.50% (65/1000)
# Lift:     +30.0%
#
# Statistical Analysis
# ─────────────────────────
# p-value:      0.089
# Confidence:   91.1%
# Result:       NOT SIGNIFICANT (need 95%)
#
# Recommendation: Continue test for more data
Example 2: Plan Sample Size
# Baseline 5% conversion, want to detect 20% relative lift (1% absolute)
python scripts/main.py sample-size --baseline 0.05 --mde 0.01

# Output:
# Sample Size Calculator
# ──────────────────────────────
# Baseline conversion: 5.0%
# Minimum detectable effect: 1.0% (20% relative)
# Target conversion: 6.0%
#
# Required per variant: 3,842 visitors
# Total required: 7,684 visitors
#
# At 1000 daily visitors: ~8 days

Key Concepts

TermDefinition
p-valueProbability result is due to chance
Confidence1 - p-value (usually want 95%+)
PowerProbability of detecting real effect (usually 80%)
MDEMinimum Detectable Effect - smallest lift worth detecting
LiftRelative improvement (variant - control) / control

When Results Are Significant

p-valueConfidenceVerdict
< 0.01> 99%Highly Significant ✓
< 0.05> 95%Significant ✓
< 0.10> 90%Marginally Significant
≥ 0.10< 90%Not Significant ✗

Skill Boundaries

What This Skill Does Well
  • Structuring data analysis
  • Identifying patterns and trends
  • Creating visualization frameworks
  • Calculating statistical measures
What This Skill Cannot Do
  • Access your actual data
  • Replace statistical expertise
  • Make business decisions
  • Guarantee prediction accuracy

Related Skills

Skill Metadata

  • Mode: centaur
category: analytics
subcategory: statistics
dependencies: [scipy, numpy]
difficulty: intermediate
time_saved: 3+ hours/week

Comments

Markdown supported