mcpField
E
experiment-design-review

experiment-design-review

github.com/sylphxaiData & databases146 installsUpdated today
Repository

About experiment-design-review

A skill for teams reviewing product experiments and rollout decisions. It produces an experiment specification, instrumentation plan, decision rule, launch checklist, and learning review while accounting for risk and test suitability.

Key features

  1. 01Decision, hypothesis, and segment definition
  2. 02Metric and sample planning
  3. 03Experiment type selection
  4. 04Instrumentation and decision rules
  5. 05Launch checklist and learning review

Use cases

  1. 01Choosing a test approach for a product change
  2. 02Planning metrics and eligibility for an experiment
  3. 03Reviewing whether a change is suitable for testing
  4. 04Preparing rollout and post-test evaluation

Install

Copy the command and run it in a terminal.

$npx skills add https://github.com/sylphxai/skills --skill experiment-design-review

After installing, type/experiment-design-reviewin Claude Code to confirm it loaded.

Bundled filesagents/openai.yamlreferences/experiment-design-patterns.md

Details

From the project README

Experiment Design Review

Use this skill to convert a high-stakes product decision into a concrete, measurable, reviewable operating artifact.

Workflow

  1. Identify the decision, user segment, change, expected mechanism, risk, traffic source, and reversible rollout path.
  2. Read references/experiment-design-patterns.md.
  3. Separate hypothesis, primary metric, guardrails, counter-metrics, sample needs, eligibility, exposure, and analysis window.
  4. Choose experiment type: A/B, holdout, staged rollout, switchback, pre/post, qualitative pilot, or no-test decision.
  5. Produce experiment spec, instrumentation plan, decision rule, launch checklist, and learning review.

When not to use

  • Do not use for generic advice the base model already handles without this skill's specific artifact contract.

Guardrails

  • Do not call an experiment successful from vanity engagement alone.
  • Do not test unsafe, deceptive, discriminatory, privacy-invasive, or legally sensitive variants without qualified review.
  • Do not ignore novelty effects, seasonality, sample pollution, repeated peeking, or segment imbalance.
  • Use product judgment when experiments are underpowered, too risky, or not ethically testable.

Output format

Experiment context:
Decision / hypothesis / segment:

Experiment spec:
| Field | Value |
| --- | --- |
| Variant | <change> |
| Primary metric | <metric/window> |
| Guardrails | <metric/window> |
| Eligibility | <users/sessions/accounts> |
| Decision rule | <ship/iterate/rollback/no decision> |

Risk and instrumentation:
- <risk> -> <event/check/owner>

Learning review:
- <result, caveat, decision, next action>

Comments

Markdown supported