# A/B Testing — Marketing Skill for AI Agents

Run a valid A/B test in five steps: write a real hypothesis ("Because [data], we believe [change] will cause [outcome] for [audience], measured by [metric]"), pick one primary metric plus secondary and guardrail metrics, calculate the sample size before launching from your baseline rate and minimum detectable effect, run for the full pre-committed duration (at least one full week to capture day-of-week effects), then judge at 95% confidence. The single biggest mistake is peeking — stopping the moment a variant hits significance inflates false positives, because early leads regress and audience mix shifts across days.

> Maintained by Corey Haines · v2.0.0 · Updated 2026-05-13

Install: `npx skills add coreyhaines31/marketingskills`
Source: https://github.com/coreyhaines31/marketingskills/tree/main/skills/ab-testing

## When to use it

- "A/B test,"
- "split test,"
- "experiment,"
- "test this change,"
- "variant copy,"
- "multivariate test,"
- "hypothesis,"
- "should I test this,"

## What's inside

**The playbook covers:** Initial Assessment, Core Principles, Hypothesis Framework, Test Types, Sample Size, Metrics Selection, Designing Variants, Traffic Allocation, Implementation, Running the Test, Analyzing Results, Documentation, Growth Experimentation Program, [Experiment Name], Common Mistakes, Task-Specific Questions

### Reference library

- [Sample Size Guide](https://github.com/coreyhaines31/marketingskills/blob/main/skills/ab-testing/references/sample-size-guide.md) — Reference for calculating sample sizes and test duration.
- [A/B Test Templates Reference](https://github.com/coreyhaines31/marketingskills/blob/main/skills/ab-testing/references/test-templates.md) — Templates for planning, documenting, and analyzing experiments.

## Key data

- Statistical significance (95%)**: Means there's less than 5% chance the observed difference is due to random chance.
- Statistical power (80%)**: Means if there's a real effect of size MDE, you have 80% chance of detecting it.
- Scenario 1: High-traffic page**
- Duration: 20,000 / 5,000 = 4 days
- Scenario 2: Medium-traffic page**
- Duration: 60,000 / 2,000 = 30 days

## Example

**Prompt:** I want to A/B test our homepage headline. We currently say 'The All-in-One Project Management Tool' and want to test something benefit-focused. We get about 15,000 visitors/month and our current signup rate is 3.2%.

**Hypothesis**

Because users report difficulty finding the CTA (heatmaps + feedback), we believe a larger, higher-contrast button will increase CTA clicks 15%+ for new visitors. We'll measure click-through from page view to signup start.

**Test plan**

Type: A/B (2 variants, 50/50)
Primary: signup rate (baseline 3.2%)
Secondary: time on page, scroll depth
Guardrail: bounce rate, support tickets
Sample: ~31k per variant to detect a 20% lift
Duration: run full window, no early calls

## FAQ

### How long should I run an A/B test?

Long enough to hit your pre-calculated sample size, and never less than one full week so you capture day-of-week variation (two business cycles for B2B). Calculate sample size from your baseline conversion rate and the minimum lift you care about, then divide by daily traffic. Avoid running past 4–8 weeks, where novelty effects fade and outside factors creep in.

### My variant hit 95% confidence after 3 days — can I ship it?

No. Checking results early and stopping the moment you see significance is "peeking," and it dramatically inflates your false-positive rate. Early leads regress to the mean and are distorted by day-of-week and audience-mix effects. Run for the full pre-committed duration, or use a proper sequential testing method if you genuinely need to stop early.

### What if I don't have enough traffic to reach significance?

You have options: raise your minimum detectable effect so you only test bolder changes, test higher in the funnel where traffic is greater, combine similar pages, reduce the number of variants, or accept a longer run. If none of those work, make the call from qualitative evidence instead of running an underpowered test that can't detect a real effect.

## Related skills

- **cro** — For generating test ideas based on CRO principles
- **analytics** — For setting up test measurement
- **copywriting** — For creating variant copy

---
Canonical: https://marketing-skills.com/skills/ab-testing