experiment-brainstorm

experiment-brainstorm

Propose new experiment ideas grounded in the team's past stopped experiments. Read history first; propose based on what actually moved metrics, where guardrails failed, and which tags or projects under-explored.

All API calls go through the bundled helper: ${CLAUDE_PLUGIN_ROOT}/scripts/gb-call. It needs GB_API_KEY — set in your shell, or written to ~/.config/growthbook/.env by /growthbook:gb-setup. If unset or invalid, gb-call's error message points back at /growthbook:gb-setup.

Workflow

1. Pull the experiment list, most recent first.

   gb-call GET '/api/v1/experiments?limit=50&status=stopped&sortBy=dateCreated&sortOrder=desc'
   

   Returns up to 50 experiments per page (limit caps at 100). Keep sortBy=dateCreated&sortOrder=desc — the API's default order is oldest-first, and on an org with more than one page of history an unsorted pull grounds every proposal in ancient experiments.

   If the user scoped the brainstorm ("ideas for checkout", "what should the growth team test next"), narrow the pull with filters instead of discarding results after the fact: &tag=checkout, &projectId=prj_abc123, &owner=<email>, &result=won,lost, &metricId=met_abc123, or &implementationType=feature. tag, owner, result, metricId, and implementationType take comma-separated values (ORed within a param); projectId takes a single project id. Separate params AND together.

   Consider adding &bandits=false when the user wants product-test patterns: bandit results carry per-arm probabilities rather than a winner/loser verdict, so they distort the win-rate tally in step 4.

2. Fetch results for each stopped experiment. Loop over the stopped IDs:

   gb-call GET /api/v1/experiments/<id>/results
   

   Pace the calls — the API is rate-limited at 60 requests per minute. Cap the pull at ~20 experiments unless the user explicitly wants more; that's plenty for pattern-finding and stays well inside the budget.

3. Read the patterns. From the result payloads, identify three things before proposing anything:

   • What's working — themes shared by experiments where the test variation beat the control (which projects, which surfaces, which kind of change).
   • What's stalling — themes shared by losers and inconclusive tests.
   • What's under-explored — projects, tags, or surfaces with few experiments compared to the rest.

4. Compute light aggregate context. Mentally tally — no need to surface a full dashboard:

   • Approximate win rate (won / total settled).
   • Top 3 winners by absolute lift on the primary metric.
   • Top 3 losers by absolute lift.
   • Any experiments flagged for SRM (sample ratio mismatch).
   • Project / tag distribution.

5. Propose 5–7 ideas. Each proposal contains:

   • Hypothesis in one sentence: "If we change X, then Y will improve, because Z."
   • Why this is grounded — one sentence linking it to a specific past experiment (winner to extend, loser to retry differently, gap to fill). Cite the experiment name or ID.
   • Primary metric — pick one. State the type (proportion, mean, ratio, quantile) and why.
   • Expected effect size — order of magnitude only ("comparable to the +3.2% lift on the checkout flow test"), not a precise number.
   • Risk to watch — one guardrail metric or potential regression.

6. Present with structure. Lead with the patterns you saw (1–2 lines each), then the proposals. End by asking the user which to refine — do not start designing or creating experiments inside this skill. Hand off to experiment-design for the one(s) the user picks.

Guardrails

• Stopped experiments only. Filter drafts and running experiments out of your synthesis. If the user asks about the live pipeline, that's a different question — point them at flag-discovery or experiment-design instead.
• result alone does not imply stopped. A result is recorded when an experiment is stopped but is retained if it's later restarted, so result=won can return running experiments. Always keep status=stopped in the query rather than relying on result to scope the history.
• Filtering bandits uses bandits, not type. The list endpoint has no type param — implementationType filters the linked-change kind (feature / visualChange / redirect), which is a different axis from the response's type field (standard / multi-armed-bandit). Sending type= returns a 400.
• Ground every proposal. Cite the specific past experiment(s) you're building on. No proposals based on generic best practices.
• Don't repeat losers without saying why. If a proposal mirrors a recent loser, say so explicitly and explain what's different this time.
• Win rate definition: won / (won + lost + inconclusive). Don't invent another formula.
• Watch for SRM and guardrail issues in the history. If many experiments show SRM failures, mention it and propose at least one idea aimed at improving experiment hygiene rather than another product test.
• Propose, do not create. Never POST to /api/v1/experiments. The user's next step is experiment-design for the proposal they want to pursue.
• Avoid metric-fishing proposals. Each idea has one (occasionally two) goal metric. Don't propose tests with five metrics hoping one moves — that's the "too many goal metrics" footgun. GrowthBook's decision framework supports up to two goal metrics, but more dilutes power and confuses the ship/kill call.
• Rate limit awareness. 50 experiments + 20 result fetches = ~21 calls; well under the 60 rpm cap. If the user wants more depth, ask before fanning out to >40 calls.

Endpoints used

• GET /api/v1/experiments?limit=50&status=stopped&sortBy=dateCreated&sortOrder=desc — list stopped experiments, newest first (limit caps at 100). Optional scoping filters: tag, owner, result, metricId, implementationType (comma-separated values ORed within a param), projectId (single project id), bandits (true/false), archived. Sortable fields: dateCreated, dateUpdated, name.
• GET /api/v1/experiments/{id}/results — full results for one experiment. One call per stopped experiment in scope.

Output template

## Patterns from your last N stopped experiments

- Working: <theme>
- Stalling: <theme>
- Under-explored: <theme>

## Proposed experiments

### 1. <short title>
- Hypothesis: …
- Grounded in: <past experiment + verdict>
- Primary metric: <name> (<type>)
- Expected effect: <order of magnitude>
- Risk: <guardrail>

…

Pick one or two and I'll hand off to `experiment-design` to scope it.