mcpField
T
trustworthy-experiments

trustworthy-experiments

github.com/pmpromptData & databases55350 installsUpdated 6d ago
Repository

About trustworthy-experiments

A product-management skill for planning and evaluating controlled experiments. It focuses on checking whether results are reliable enough to guide decisions, from test design through result validation.

Key features

  1. 01Plan controlled A/B tests
  2. 02Estimate sample size and runtime
  3. 03Select outcome metrics
  4. 04Validate results before decisions
  5. 05Diagnose suspicious findings

Use cases

  1. 01Design an experiment before launch
  2. 02Decide whether a test has enough power
  3. 03Assess results before shipping a change
  4. 04Investigate a potentially invalid result

Install

Copy the command and run it in a terminal.

$npx skills add https://github.com/pmprompt/claude-plugin-product-management --skill trustworthy-experiments

After installing, type/trustworthy-experimentsin Claude Code to confirm it loaded.

Details

From the project README

Domain Context

This skill implements a proven product management framework. The approach combines best practices from industry leaders and is designed for practical application in day-to-day PM work.

Input Requirements

  • Context about your product, feature, or problem
  • Relevant data, research, or constraints (recommended but optional)
  • Clear articulation of what you're trying to achieve

Trustworthy Experiments

What It Is

Trustworthy Experiments is a framework for running controlled experiments (A/B tests) that produce reliable, actionable results. The core insight: most experiments fail, and many "successful" results are actually false positives.

The key shift: Move from "Did the experiment show a positive result?" to "Can I trust this result enough to act on it?"

Ronny Kohavi, who built experimentation platforms at Microsoft, Amazon, and Airbnb, found that:

  • 66-92% of experiments fail to improve the target metric
  • 8% of experiments have invalid results due to sample ratio mismatch alone
  • When the base success rate is 8%, a P-value of 0.05 still means 26% false positive risk

When to Use It

Use Trustworthy Experiments when you need to:

  • Design an A/B test that will produce valid, actionable results
  • Determine sample size and runtime for statistical power
  • Validate experiment results before making ship/no-ship decisions
  • Build an experimentation culture at your company
  • Choose metrics (OEC) that balance short-term gains with long-term value
  • Diagnose why results look suspicious (Twyman's Law)
  • Speed up experimentation without sacrificing validity

When Not to Use It

Don't use controlled experiments when:

  • You don't have enough users — Need tens of thousands minimum
  • The decision is one-time — Can't A/B test mergers or acquisitions
  • There's no real user choice — Employer-mandated software
  • You need immediate decisions — Experiments need time
  • The metric can't be measured — No experiment without observable outcomes

Resources

Book:

  • Trustworthy Online Controlled Experiments by Ronny Kohavi, Diane Tang, and Ya Xu

Comments

Markdown supported