experimentation-and-ab-testing

experimentation-and-ab-testing

The causation engine — manipulate one variable under controlled conditions to learn what actually moves a KPI. This skill designs the test and drafts variants; scheduling-and-queue → WoopSocial publishes them; analytics-and-reporting reads the result.

The POV: evidence, not vibes

Most "testing" on social is vibes — post two things, eyeball the likes, declare a winner, learn nothing. Real experimentation turns guesses into evidence: change one variable, control everything else, set the decision rule before you publish, and run it long and often enough to separate signal from noise. Organic can't give clean statistical significance (small samples, an algorithm in the middle), so you compensate with tighter controls, a ~20%+ effect threshold, guardrail metrics, and 3–5 repetitions — and treat a single viral post as noise, not a strategy.

Read these first

1. brand-profile — voice/format constraints for the variants.
2. goals-and-kpis — the KPI/primary metric the test must move.

The framework: TEST

(Depth: references/the-test-framework.md.)

• T — Target one variable: a clear hypothesis; change ONE element (hook/first-frame/caption/CTA/time/ format), everything else identical; pick the highest-leverage one.
• E — Establish the decision rule first: set the primary metric + win threshold + guardrail before publishing ("B wins if reach +15% and saves/reach not worse"); no post-hoc rationalizing.
• S — Set controls + sample: same platform/format/topic/length/window; run ≥7 days (small accounts 2–4 weeks); judge on a ~20%+ consistent effect (a tie = "test elsewhere").
• T — Tally, repeat, scale: 3–5 paired repetitions before a "best practice"; log every test; scale winners into the playbook (content-recycling), retire the rest.

What to test (highest leverage, in your control)

Hook/first-frame (short video) → posting time (easy) → format → caption/CTA → thumbnail → hashtags — always tied to the KPI; test what's in your control, not algorithm-dependent factors. Run a 30-day sprint with one test always running. Priority list, design template, sprint plan, testing log + worked examples: references/what-to-test-and-recipes.md. Full method + rules: references/experimentation-2026-reality.md.

Honest scope (never violate)

• Organic isn't lab-grade — results are directional; compensate with controls + effect-size + repetition, not p-value theater.
• WoopSocial has no A/B/audience-split surface → organic testing = controlled sequential posts; the agent designs + drafts variants + schedules; the primary metric is read from native analytics (analytics-and-reporting). One true native split exists: YouTube's Test & Compare (YouTube Studio, long-form, not Shorts) — up to 3 titles, thumbnails, or title+thumbnail combos; use it for YouTube title/thumbnail tests instead of sequential posts. (verify-quarterly)
• No p-hacking / HARKing / cherry-picking — decision rule pre-set; a multi-variable change can't be pinned on one element; one post/one day is noise. Never fabricate a result; a tie is valid. (Scope, the loop role + connections: references/scope-and-connections.md.)

Distinct from its siblings (route correctly)

experimentation (this) = manipulate one variable to establish causation · analytics-and-reporting = observe/measure what happened · goals-and-kpis = set the target/primary metric · content-recycling = scale proven winners · viral-reverse-engineering = explain a past post (hindsight) vs testing forward.

Where this connects

Reads first: brand-profile, goals-and-kpis. Variants drafted via: hook-writer, caption-writer, reels-script/tiktok-script, carousel-writer, image-prompt/ideogram/nano-banana, thumbnail-design. Readout: analytics-and-reporting (native analytics). Scale/plan: content-recycling, social-strategy, content-calendar/batch-content-plan, every *-growth skill. Publish variants: scheduling-and-queue → WoopSocial (controlled sequential posts).

Definition of done

A clear hypothesis testing ONE variable tied to a KPI; identical controlled context; a primary metric + win threshold + guardrail set before publishing; duration ≥7 days (2–4 weeks small accounts) and 3–5 paired repetitions; results read from native analytics and judged on a ~20%+ consistent effect (ties acknowledged); winners logged and scaled to content-recycling/strategy; organic limits stated, nothing fabricated or p-hacked, correctly distinguished from analytics-and-reporting and goals-and-kpis.