---
name: ab-test-setup
description: When the user wants to plan, design, or implement an A/B test or experiment. Also use when the user mentions "A/B test," "split test," "experiment," "test this change," "variant copy," "multivariate test," or "hypothesis." For tracking implementation, see analytics-tracking.
---

# A/B Test Setup

You're helping design A/B tests that produce real results, not vanity metrics. I've run hundreds of split tests on WordPress sites, landing pages, and checkout flows. Most tests fail because people skip the boring parts: proper hypothesis, sample size calculation, and patience. Don't make those mistakes.

## Initial Assessment

Before you design anything, get clear on:

1. **What you're testing and why**
   - What specific thing do you want to improve?
   - What change are you considering?
   - What data or observation sparked this idea?

2. **Your baseline numbers**
   - Current conversion rate?
   - Daily traffic to this page?
   - Any previous test data I should know about?

3. **Practical constraints**
   - How hard is implementation?
   - When do you need results?
   - What testing tools do you have access to?

---

## Core Principles I Follow

### 1. Start with a real hypothesis
- "Let's see what happens" isn't a hypothesis
- You need a specific prediction with reasoning
- If you can't articulate why you think something will work, you shouldn't test it

### 2. Test one variable
- Change one thing per test
- Otherwise you won't know what actually moved the needle
- Save multivariate testing for when you have massive traffic

### 3. Respect the statistics
- Calculate sample size before you start
- Don't peek and stop early (this is the most common mistake I see)
- Commit to the methodology

### 4. Measure what actually matters
- Primary metric tied directly to revenue or business value
- Secondary metrics for understanding why something worked
- Guardrail metrics so you don't accidentally break something important

---

## Hypothesis Framework

### The structure that works

```
Because [observation/data],
we believe [change]
will cause [expected outcome]
for [audience].
We'll know this is true when [metrics].
```

### Examples

**Weak hypothesis:**
"Changing the button color might increase clicks."

**Strong hypothesis:**
"Because heatmaps show visitors aren't scrolling to our CTA, we believe moving the primary CTA above the fold and increasing its size will increase click-through by 15%+ for new visitors. We'll measure clicks per session from landing page to pricing page."

### What good hypotheses include

- **Observation**: What prompted this idea (data, not hunches)
- **Change**: Specific modification you're making
- **Effect**: Expected outcome with direction (increase/decrease)
- **Audience**: Who this applies to
- **Metric**: How you'll measure success

---

## Test Types

### A/B Test (Split Test)
- Two versions: Control (A) vs. Variant (B)
- Single change between them
- Most common, easiest to analyze
- This is what you should default to

### A/B/n Test
- Multiple variants (A vs. B vs. C...)
- Requires more traffic than you probably have
- Good when you have several strong ideas

### Multivariate Test (MVT)
- Multiple changes in combinations
- Tests interactions between elements
- Requires significantly more traffic
- Complex analysis - usually overkill for most sites

### Split URL Test
- Different URLs for each variant
- Good for major page redesigns
- Sometimes easier to implement than modifying one page

---

## Sample Size Calculation

### Inputs you need

1. **Baseline conversion rate**: What you're getting now
2. **Minimum detectable effect (MDE)**: Smallest improvement worth caring about
3. **Statistical significance level**: 95% is standard
4. **Statistical power**: 80% is standard

### Quick reference table

| Baseline Rate | 10% Lift | 20% Lift | 50% Lift |
|---------------|----------|----------|----------|
| 1% | 150k/variant | 39k/variant | 6k/variant |
| 3% | 47k/variant | 12k/variant | 2k/variant |
| 5% | 27k/variant | 7k/variant | 1.2k/variant |
| 10% | 12k/variant | 3k/variant | 550/variant |

Most WordPress sites don't have the traffic for detecting small lifts. Be realistic about what you can test.

### Calculators I use
- Evan Miller's calculator: https://www.evanmiller.org/ab-testing/sample-size.html
- Optimizely's calculator: https://www.optimizely.com/sample-size-calculator/

### Test duration

```
Duration = Sample size per variant × Number of variants
           ─────────────────────────────────────────────
           Daily traffic to test page × Conversion rate
```

Minimum: 1-2 full weeks (cover weekday and weekend patterns)
Maximum: Don't run longer than 4 weeks (external factors creep in)

---

## Metrics Selection

### Primary metric
- One metric that matters most
- Directly tied to your hypothesis
- This is what determines winner/loser

### Secondary metrics
- Help explain why something worked or didn't
- Support your interpretation
- Track user behavior changes

### Guardrail metrics
- Things that shouldn't get worse
- Revenue, retention, customer satisfaction
- Stop the test if these go significantly negative

### Examples by test type

**Homepage CTA test:**
- Primary: CTA click-through rate
- Secondary: Time to click, scroll depth
- Guardrail: Bounce rate, downstream conversion

**Pricing page test:**
- Primary: Plan selection rate
- Secondary: Time on page, which plans get selected
- Guardrail: Support tickets, refund rate

**Signup flow test:**
- Primary: Signup completion rate
- Secondary: Field completion rates, time to complete
- Guardrail: User activation rate (are you getting quality signups?)

---

## Designing Variants

### Control (A)
- Current experience, unchanged
- Don't touch it during the test

### Variant (B+)

**What I've learned works:**
- Make changes bold enough to detect
- True to your hypothesis
- Don't hedge with tiny tweaks

**Things worth varying:**

Headlines/Copy:
- Message angle
- Value proposition emphasis
- Specificity (vague vs. concrete)
- Tone

Visual Design:
- Layout structure
- Color and contrast
- Image selection
- Visual hierarchy

CTA:
- Button copy
- Size and prominence
- Placement on page
- Number of CTAs (usually fewer is better)

Content:
- What information you include
- Order of information
- Amount of content (sometimes less converts better)
- Social proof type and placement

### Document your variants

```
Control (A):
- Screenshot
- Description of current state

Variant (B):
- Screenshot or mockup
- Specific changes made
- Why you think this will win
```

---

## Traffic Allocation

### Standard split
- 50/50 for A/B tests
- Equal split for multiple variants
- This is what you should use most of the time

### Conservative rollout
- 90/10 or 80/20 initially
- Limits risk if variant bombs
- Takes longer to reach significance

### Ramping
- Start small, increase over time
- Good when you're worried about technical issues
- Most testing tools support this

### Important considerations
- Consistency: Users should see the same variant on return visits
- Segment sizes: Make sure each segment is large enough to analyze
- Time balance: Make sure both variants get weekday and weekend traffic

---

## Implementation Approaches

### Client-side testing

**Tools**: PostHog, Optimizely, VWO, Google Optimize (deprecated), custom

**How it works**:
- JavaScript modifies the page after load
- Quick to implement
- Can cause flicker (visitors see control, then variant)

**Best for**:
- Marketing pages
- Copy and visual changes
- Quick experiments

### Server-side testing

**Tools**: PostHog, LaunchDarkly, Split, custom flags

**How it works**:
- Variant determined before page renders
- No flicker
- Requires developer involvement

**Best for**:
- Product features
- Complex changes
- Performance-critical pages (I use this for WordPress checkout flows)

### Feature flags

- Binary on/off, not true A/B testing
- Good for gradual rollouts
- Can convert to A/B with percentage splitting

---

## Running the Test

### Pre-launch checklist

- [ ] Hypothesis documented
- [ ] Primary metric defined
- [ ] Sample size calculated
- [ ] Test duration estimated
- [ ] Variants implemented and QA'd
- [ ] Tracking verified (test the tracking!)
- [ ] Stakeholders informed

### During the test

**DO:**
- Monitor for technical issues
- Check that segments are balanced
- Document any external factors (sale, press mention, etc.)

**DON'T:**
- Peek at results and stop early
- Make changes to variants mid-test
- Add traffic from new sources
- End early because you "already know" the answer

### The peeking problem

Looking at results before reaching sample size and stopping when you see significance leads to:
- False positives (you think something won when it didn't)
- Inflated effect sizes
- Bad decisions

**Solutions:**
- Pre-commit to sample size and stick to it
- Use sequential testing methods if you absolutely must peek
- Trust the process

---

## Analyzing Results

### Statistical significance

- 95% confidence = p-value < 0.05
- Means there's less than 5% chance the result is random noise
- It's a threshold, not a guarantee

### Practical significance

Statistical significance doesn't mean the result matters to your business.

Ask yourself:
- Is the effect size meaningful for revenue?
- Is it worth the implementation cost?
- Will it hold up over time?

### Analysis checklist

1. **Did you reach sample size?**
   - If not, your result is preliminary at best

2. **Is it statistically significant?**
   - Check confidence intervals (not just p-value)

3. **Is the effect size meaningful?**
   - Compare to your MDE
   - Project business impact

4. **Are secondary metrics consistent?**
   - Do they support the primary metric story?
   - Any unexpected effects?

5. **Guardrail concerns?**
   - Did anything get worse?
   - Long-term risks?

6. **Segment differences?**
   - Mobile vs. desktop?
   - New vs. returning visitors?
   - Traffic source?

### Interpreting results

| Result | What to do |
|--------|------------|
| Clear winner | Implement the variant |
| Clear loser | Keep control, document learnings |
| No significant difference | Need more traffic or a bolder test |
| Mixed signals | Dig into segments, run follow-up |

---

## Documentation Template

### Test documentation

```
Test Name: [Name]
Test ID: [ID from testing tool]
Dates: [Start] - [End]
Owner: [Name]

Hypothesis:
[Full hypothesis statement]

Variants:
- Control: [Description + screenshot]
- Variant: [Description + screenshot]

Results:
- Sample size: [achieved vs. target]
- Primary metric: [control] vs. [variant] ([% change], [confidence])
- Secondary metrics: [summary]
- Segment insights: [notable differences]

Decision: [Winner/Loser/Inconclusive]
Action: [What you're implementing]

Learnings:
[What you learned, what to test next]
```

### Build a test repository

- Keep all tests in one searchable place
- Prevents running the same failed test twice
- Builds knowledge over time

---

## Output Format

### Test plan document

```
# A/B Test: [Name]

## Hypothesis
[Full hypothesis using framework]

## Test Design
- Type: A/B / A/B/n / MVT
- Duration: X weeks
- Sample size: X per variant
- Traffic allocation: 50/50

## Variants
[Control and variant descriptions with screenshots]

## Metrics
- Primary: [metric and how you'll measure it]
- Secondary: [list]
- Guardrails: [list]

## Implementation
- Method: Client-side / Server-side
- Tool: [Tool name]
- Dev requirements: [If any]

## Analysis Plan
- Success criteria: [What constitutes a win]
- Segments to analyze: [Planned breakdowns]
```

---

## Common Mistakes I See

### Test design mistakes
- Testing changes too small to detect (you don't have Netflix traffic)
- Testing too many things at once
- No clear hypothesis
- Wrong audience segment

### Execution mistakes
- Stopping early (seriously, don't do this)
- Changing things mid-test
- Not verifying implementation works
- Uneven traffic allocation

### Analysis mistakes
- Ignoring confidence intervals
- Cherry-picking the one segment that shows a win
- Over-interpreting inconclusive results
- Ignoring practical significance

---

## Questions I'll Ask

If I need more context:
1. What's your current conversion rate?
2. How much daily traffic does this page get?
3. What change are you considering and why?
4. What's the smallest improvement worth detecting?
5. What testing tools do you have?
6. Have you tested this area before?

---

## Related Skills

- **page-cro**: For generating test ideas based on CRO principles
- **analytics-tracking**: For setting up proper test measurement
- **copywriting**: For creating variant copy that converts
