Run a valid A/B test in five steps: write a real hypothesis ("Because [data], we believe [change] will cause [outcome] for [audience], measured by [metric]"), pick one primary metric plus secondary and guardrail metrics, calculate the sample size before launching from your baseline rate and minimum detectable effect, run for the full pre-committed duration (at least one full week to capture day-of-week effects), then judge at 95% confidence. The single biggest mistake is peeking — stopping the moment a variant hits significance inflates false positives, because early leads regress and audience mix shifts across days.
When to use it
Your agent reaches for this skill when you ask for things like:
What's inside
The playbook covers
Key data
The recommendations are grounded in benchmarks, not opinion:
See it in action
I want to A/B test our homepage headline. We currently say 'The All-in-One Project Management Tool' and want to test something benefit-focused. We get about 15,000 visitors/month and our current signup rate is 3.2%.
Because users report difficulty finding the CTA (heatmaps + feedback), we believe a larger, higher-contrast button will increase CTA clicks 15%+ for new visitors. We'll measure click-through from page view to signup start.
Type: A/B (2 variants, 50/50) Primary: signup rate (baseline 3.2%) Secondary: time on page, scroll depth Guardrail: bounce rate, support tickets Sample: ~31k per variant to detect a 20% lift Duration: run full window, no early calls
FAQ
How long should I run an A/B test?
Long enough to hit your pre-calculated sample size, and never less than one full week so you capture day-of-week variation (two business cycles for B2B). Calculate sample size from your baseline conversion rate and the minimum lift you care about, then divide by daily traffic. Avoid running past 4–8 weeks, where novelty effects fade and outside factors creep in.
My variant hit 95% confidence after 3 days — can I ship it?
No. Checking results early and stopping the moment you see significance is "peeking," and it dramatically inflates your false-positive rate. Early leads regress to the mean and are distorted by day-of-week and audience-mix effects. Run for the full pre-committed duration, or use a proper sequential testing method if you genuinely need to stop early.
What if I don't have enough traffic to reach significance?
You have options: raise your minimum detectable effect so you only test bolder changes, test higher in the funnel where traffic is greater, combine similar pages, reduce the number of variants, or accept a longer run. If none of those work, make the call from qualitative evidence instead of running an underpowered test that can't detect a real effect.
Related skills
Build a tool for this workflow?
Marketers run this skill mid-task — the exact moment your tool matters. Partner tiers put your tool inside the skill their agent runs.
See partner tiers