mcpField
A
ab-test-designer

ab-test-designer

github.com/pmpromptData & databases55390 installsUpdated 6d ago
Repository

About ab-test-designer

An A/B test design skill for product teams evaluating features or design changes. It structures hypotheses, metrics, sample-size planning, risks, and analysis criteria before a test runs.

Key features

  1. 01Form testable hypotheses
  2. 02Select success and guardrail metrics
  3. 03Plan sample size and duration
  4. 04Define variants and segments
  5. 05Set risk controls and stop criteria
  6. 06Prepare an analysis plan

Use cases

  1. 01Validate a feature before rollout
  2. 02Compare design alternatives
  3. 03Optimize conversion or key metrics
  4. 04Make data-informed change decisions

Install

Copy the command and run it in a terminal.

$npx skills add https://github.com/pmprompt/claude-plugin-product-management --skill ab-test-designer

After installing, type/ab-test-designerin Claude Code to confirm it loaded.

Details

From the project README

Domain Context

This skill implements a proven product management framework. The approach combines best practices from industry leaders and is designed for practical application in day-to-day PM work.

Input Requirements

  • Context about your product, feature, or problem
  • Relevant data, research, or constraints (recommended but optional)
  • Clear articulation of what you're trying to achieve

A/B Test Designer

When to Use

  • Testing a new feature or design variation
  • Validating a hypothesis before full rollout
  • Optimizing conversion rates or key metrics
  • Choosing between multiple design approaches
  • Need to make a data-driven decision on a change

What This Skill Does

Helps you design rigorous A/B tests with clear hypotheses, success metrics, sample size calculations, and analysis plans.

Instructions

Help me design an A/B test for [feature/change]. Include:

  1. Hypothesis

    • Current situation and metrics
    • Proposed change
    • Expected impact and why

  2. Test Design

    • Primary success metric
    • Secondary metrics
    • Sample size needed
    • Test duration
    • User segments to include/exclude

  3. Variants

    • Control (A): current experience
    • Variant (B): new experience
    • Any additional variants (C, D, etc.)

  4. Risks and Controls

    • Potential negative impacts
    • Guardrail metrics
    • When to stop the test early

  5. Analysis Plan

    • Statistical significance threshold
    • How to handle edge cases
    • Decision criteria

Feature context: [Add context about the change you want to test]

Best Practices

  • Start with a clear, falsifiable hypothesis
  • Choose one primary metric to avoid multiple comparison issues
  • Calculate sample size upfront based on expected effect size
  • Run tests for full weekly cycles to account for day-of-week effects
  • Set a minimum test duration (usually 1-2 weeks)
  • Define success criteria before running the test
  • Monitor guardrail metrics (revenue, errors, performance)

Example

Input: Testing new onboarding flow vs current 3-step process Output: Hypothesis (new 1-step flow will increase co...

Comments

Markdown supported