I wasted my first year of A/B testing. Changed button colors. Swapped headline fonts. Tested “Subscribe” vs. “Sign Up.” And after dozens of tests, my conversion rate moved by maybe half a percentage point. Total.
The problem wasn’t the testing. The problem was what I was testing. I was optimizing the garnish while the steak was overcooked.
A/B testing is the most powerful conversion tool available to bloggers. But only when you understand what it is, what it isn’t, and what’s worth testing. Most bloggers either skip testing entirely (flying blind) or test the wrong things (flying in circles). This chapter fixes both.
What A/B Testing Actually Is (and Isn’t)
A/B testing, also called split testing, means showing two versions of something to different visitors and measuring which one performs better. Half your visitors see Version A. The other half see Version B. After enough data, you see which version produced more conversions.
That’s it. It’s a controlled experiment. Nothing more.
What A/B testing is NOT:
It’s not “try something new and see if it feels better.” That’s called guessing. A real A/B test runs both versions simultaneously, splitting traffic equally, so external factors (day of week, traffic source, seasonality) affect both versions equally. If you change your headline on Monday and check your conversion rate on Friday, you haven’t run a test. You’ve changed a thing and hoped for the best.
It’s not just for big companies. I hear this constantly. “I don’t get enough traffic for A/B testing.” You might be right, and I’ll cover minimum traffic requirements shortly. But the testing mindset, forming hypotheses, measuring results, making data-driven decisions, applies at any scale. Even if you can’t run statistically valid split tests, you can still test sequentially and make informed changes.
It’s not a one-time activity. A/B testing is a practice, not a project. You run a test. You implement the winner. You form a new hypothesis. You run another test. The best-converting blogs I’ve worked on have been through 50+ tests over multiple years. Each test builds on the last.
It’s not always the answer. Some changes are so obviously needed that testing them is a waste of time. If your form has no headline and your button says “Submit,” you don’t need an A/B test to tell you that’s bad. Fix the obvious problems first. Test the non-obvious ones.
What to Test on a Blog
Here’s where most bloggers go wrong. They test low-impact elements because those tests are easy to set up. Button color tests are fun. They’re visual. They produce a clear winner. But the impact is usually 2-5%. Meanwhile, the blog has structural problems that testing could solve for 50-200% gains.
Here’s my priority order, from highest potential impact to lowest:
1. Headlines and titles. Your headline is the first (and sometimes only) thing visitors read. A better headline can increase page engagement by 30-80%. Test different angles: curiosity-driven vs. benefit-driven, specific numbers vs. general claims, long vs. short. I once tested “How to Speed Up WordPress” against “How I Got WordPress to Load in 0.8 Seconds.” The specific version converted 67% more email subscribers because it established immediate credibility.
2. CTA copy and offers. What you’re offering and how you describe it has a massive impact. Test the lead magnet itself: does a checklist outperform an ebook? Does a template outperform a video course? Test the CTA copy: “Get the Free Guide” vs. “Send Me the Checklist” vs. “Start Growing My Blog.” These tests routinely produce 20-50% differences.
3. Page layout and structure. Where elements appear on the page matters. Test whether putting your email form above the first H2 outperforms putting it after the second H2. Test a long-form landing page against a short one. Test whether removing the sidebar increases conversions (spoiler: it usually does).
4. Form design. Number of fields, form placement, button size and color, the presence or absence of social proof near the form. These tests typically produce 10-30% differences.
5. Pop-up timing and triggers. Exit-intent vs. 60% scroll-triggered. Immediate vs. delayed. Full-screen vs. slide-in. These tests can produce 15-40% differences when you find the right combination for your audience.
6. Visual elements. Button color, font choices, image vs. no image, video vs. static content. These are the smallest-impact tests, usually 2-10%. They matter once you’ve optimized everything above.
The rule: test the thing that’s closest to the conversion decision first. Headlines are closer to the decision than button colors. CTA copy is closer than font choices. Work from the conversion point outward.
Minimum Traffic Requirements for Valid Tests
This is where most blogging advice gets dishonest. People tell you to “just start testing” without mentioning that you need real traffic for tests to mean anything.
Here’s the math, simplified.
To detect a 20% improvement with 95% statistical confidence, you need roughly 1,000 conversions per variation. If your current conversion rate is 3% and you’re splitting traffic 50/50, you need about 33,000 visitors per variation, or 66,000 total visitors during the test period.
That’s a lot. Most blogs don’t get 66,000 visitors in a month.
But here’s the thing: you don’t always need to detect a 20% improvement. If you’re testing big changes (different headlines, different offers, different layouts), the difference is often 50-100% or more. To detect a 50% improvement at 95% confidence, you need roughly 200 conversions per variation. At a 3% conversion rate, that’s about 6,700 visitors per variation, or 13,400 total.
More manageable. But still not trivial.
My practical guidelines:
5,000+ monthly page views on the page being tested: You can run meaningful tests, but expect to wait 3-4 weeks per test. Focus on testing big changes that are likely to produce large differences.
10,000-50,000 monthly page views: You can test regularly. One test at a time, 2-3 weeks per test. You can detect moderate differences (20-30% improvements).
50,000+ monthly page views: You can run multiple concurrent tests on different pages. Tests complete in 1-2 weeks. You can detect smaller differences (10-15% improvements).
Under 5,000 monthly page views: Traditional A/B testing isn’t reliable at this volume. You’ll need to run tests for months to get valid results, and by then, so many other variables have changed that the data is muddied. Use sequential testing instead (see the end of this chapter).
Whatever your traffic level, never end a test early because one version “looks like it’s winning.” Wait for statistical significance. Early results are misleading more often than not. I’ve watched tests flip winners three times before stabilizing. The version that’s ahead on day 3 is often behind on day 14.
Free and Affordable Testing Tools
Google Optimize was the go-to free A/B testing tool for years. Google shut it down in September 2023. Since then, bloggers have been scrambling for alternatives. Here’s what I recommend:
VWO (free plan up to 50,000 monthly tested visitors). This is my default recommendation for bloggers. The free plan covers enough traffic for most blogs. The visual editor lets you change headlines, button text, and layout without touching code. Setup takes 15-20 minutes for your first test. I use VWO for most client blogs because it’s reliable and the interface is straightforward.
Google Tag Manager + Google Analytics 4. Not a dedicated A/B testing tool, but you can set up basic redirect tests and event tracking. Free, but requires technical comfort. You’ll write a small script in GTM that randomly assigns visitors to a variant and tracks the results in GA4. I use this approach for sites where VWO’s free plan isn’t enough or when clients don’t want another third-party script.
CloudFlare Workers (free tier). If your site is on Cloudflare (which it should be), you can use Workers to split traffic at the CDN level. The test happens before the page even loads, so there’s zero performance impact. It’s the most technically demanding option, but it’s free and fast. I use this for clients who are serious about testing and want zero page speed impact.
WordPress plugins. Nelio A/B Testing and Split Hero both work within WordPress. They’re simpler than external tools but limited in what you can test. Pricing starts around $29/month. Worth it if you want to stay within the WordPress ecosystem and don’t need advanced features.
Manually (the “poor man’s A/B test”). Change something. Run it for two weeks. Record the conversion rate. Change it back to the original. Run for two more weeks. Compare. This isn’t a true A/B test because the two versions aren’t running simultaneously, but it’s better than nothing when your traffic is too low for proper split testing.
Whichever tool you choose, make sure it doesn’t significantly slow down your page. Some testing tools add 200-400ms to page load time because they need to modify the page before displaying it. That speed penalty can negate the conversion gains from the test itself. Always check your page speed before and after adding a testing tool.
Running Your First A/B Test: Step by Step
Forget theory. Here’s exactly how to run your first test.
Step 1: Pick one high-traffic page. Check your analytics. Find the blog post or landing page with the most traffic. That’s where your test will have the fastest, most reliable results.
Step 2: Identify the conversion action. What counts as a conversion on this page? Email signup? Link click? PDF download? You need a single, measurable action. Don’t try to measure multiple things in your first test. One page, one conversion action.
Step 3: Form a hypothesis. Not “let’s see what happens.” A specific prediction. “Changing the email opt-in headline from ‘Subscribe to My Newsletter’ to ‘Get the WordPress Speed Checklist (Free)’ will increase signups because it’s more specific about what the reader gets.” The hypothesis forces you to think about why a change might work, not just what to change.
Step 4: Create the variation. Change only one thing. If you change the headline AND the button color AND the form placement, you won’t know which change caused the result. One change per test. Be disciplined about this.
Step 5: Set up the test. In your testing tool, create the test with a 50/50 traffic split. Set the conversion goal (email signup, click, whatever you defined in Step 2). Make sure tracking is working by testing it yourself before going live.
Step 6: Let it run. Don’t check results daily. Seriously. Set a calendar reminder for 2 weeks from now. Checking results early creates a temptation to end the test before it’s valid. The data will fluctuate wildly in the first few days. Ignore it.
Step 7: Check for statistical significance. After 2-3 weeks (or when you have at least 100 conversions per variation), check the results. Most testing tools calculate statistical significance for you. You want 95% confidence minimum. If you’re not there yet, let it run longer.
Step 8: Implement the winner. Make the winning version permanent. Document what you tested, the hypothesis, and the result. This documentation becomes your testing playbook over time.
Step 9: Form the next hypothesis. Based on what you learned, what should you test next? If a more specific headline won, try making it even more specific. If a different CTA offer won, test variations of that offer.
Statistical Significance Explained Simply
I’m not going to bore you with statistics formulas. Here’s what you need to know in practical terms.
Statistical significance tells you the probability that the difference between your two versions is real and not just random chance. A 95% confidence level means there’s only a 5% chance the result is due to luck.
Why this matters: imagine you flip a coin 10 times and get 7 heads. Is the coin biased? Probably not. With only 10 flips, 7 heads is well within the range of normal randomness. But if you flip it 10,000 times and get 7,000 heads, something is definitely off with that coin.
A/B testing works the same way. Small sample sizes produce noisy results. You need enough data for the signal to emerge from the noise.
Practical rules:
- 100 conversions per variation is the minimum for a rough signal
- 200-400 conversions per variation gives you reliable results for most decisions
- If the difference between versions is less than 5%, you need a very large sample to confirm it’s real
- If the difference is 30%+, you need a smaller sample because the signal is strong
Common mistakes:
“Version B is winning after 2 days!” Probably not. Early leads reverse all the time. I’ve seen Version B ahead by 40% after 3 days, then lose by 10% after 3 weeks. Short-term results are dominated by noise.
“The test has been running for a month and there’s a 3% difference.” If after a month and thousands of visitors, the difference is only 3%, it’s probably not meaningful. The versions are performing roughly the same. Call it a draw and test something bigger.
“I got 95% confidence after 50 total conversions.” Be skeptical. With very small samples, statistical significance can be achieved by chance, especially if the measured difference is large. This is a known issue with frequentist statistics at small sample sizes. If your result seems too good to be true at a small sample, extend the test.
When A/B Testing Isn’t Worth It (And What to Do Instead)
A/B testing is powerful. But it’s not always the right approach. Here’s when you should skip it:
When the answer is obvious. If your opt-in form has no headline, no benefit copy, and a button that says “Submit,” you don’t need a test. You need a rewrite. Test the non-obvious stuff. Fix the obvious stuff immediately.
When traffic is too low. Under 5,000 monthly page views on the page you’d test, traditional A/B testing takes too long to produce valid results. By the time you have enough data, the context has changed. Seasonal patterns, algorithm updates, content changes, all muddying the water.
When the stakes are low. Testing which shade of blue to use on a secondary link isn’t worth the effort. The potential gain is tiny. Focus your testing energy on high-impact elements: headlines, offers, forms, layouts.
When you’re testing opinions instead of conversions. “Which design do you think looks better?” is not A/B testing. Opinions are irrelevant. Only conversion data matters. Your team might hate Version B, but if it converts 40% better, Version B wins. End of discussion.
What to do instead:
Sequential testing (for low-traffic sites). Make a change. Measure the conversion rate for 2-4 weeks. Compare it to the previous 2-4 weeks. This isn’t as rigorous as a simultaneous split test, but it gives you directional data. Account for seasonality and traffic source changes when comparing periods.
Heuristic evaluation. Apply conversion principles (clear headlines, specific benefits, one CTA per page, minimal friction) without testing. If you’re following proven patterns, you’ll be right most of the time. Not every change needs a test. Sometimes experience and common sense are enough.
User testing. Watch 5 real people use your site. The insights you get from watching someone struggle with your form or miss your CTA are more valuable than any A/B test at low traffic volumes. And it’s free.
Copy competitor patterns. If a competitor with much larger traffic is using a specific approach, they’ve probably tested it. I’m not saying to copy their design. But if every successful blog in your niche uses inline forms with specific headlines and high-contrast buttons, that pattern has been validated at scale by someone else’s testing budget.
Pre/post analysis. Make changes in batches (new headline + new CTA + new form design), implement them all at once, and compare the before/after conversion rate. You won’t know which specific change drove the improvement, but you’ll know whether the overall package is better. Once you have more traffic, you can isolate individual elements.
The goal isn’t to A/B test everything. The goal is to make better decisions about your blog’s conversion performance. Testing is one input. Experience, observation, user feedback, and common sense are others. Use all of them.
Chapter Checklist
- [ ] You understand the difference between A/B testing and “changing things and hoping”
- [ ] You’ve identified your highest-traffic page for a first test
- [ ] You know your monthly page views for that page (to set realistic test timelines)
- [ ] You’ve selected a testing tool appropriate for your traffic level
- [ ] Testing tool has been verified to not significantly increase page load time
- [ ] You have a written hypothesis for your first test (not just “let’s see what happens”)
- [ ] Your first test changes exactly one element
- [ ] You understand minimum sample sizes: 100+ conversions per variation
- [ ] You have a calendar reminder to check results after 2 weeks (not daily)
- [ ] You know the threshold: 95% statistical confidence before declaring a winner
- [ ] If traffic is under 5,000 monthly page views, you have an alternative plan (sequential testing, user testing, or heuristic evaluation)
- [ ] You have a document or spreadsheet to record test hypotheses, results, and learnings
Chapter Exercise
Set up and launch your first A/B test (or your first structured sequential test if traffic is low):
- Open your analytics. Find your page with the most traffic over the past 30 days. Write down the page URL and monthly visitor count
- Identify the primary conversion action on that page (email signup, click, download)
- Look at the current conversion rate for that action over the past 30 days
- Write a hypothesis: “I believe changing [specific element] from [current version] to [new version] will increase [conversion action] because [specific reason]”
- If monthly page views are over 5,000: set up a proper A/B test using VWO or your chosen tool. 50/50 split, single element change, conversion goal defined
- If monthly page views are under 5,000: implement the change and record today’s date. You’ll compare the next 30 days against the previous 30 days
- Set a calendar reminder for 14 days from now to check initial results, and 28 days for final results
- When the test concludes, record the result in a spreadsheet: test name, hypothesis, winner, conversion lift percentage, sample size, confidence level
- Form your next hypothesis based on what you learned and repeat
After 3-5 tests, you’ll start developing an intuition for what works with your specific audience. That intuition, informed by data instead of guessing, is what separates bloggers who convert from bloggers who hope.