If your A/B test results always come out ambiguous, the problem is usually the design rather than the idea. Getting the sample size and test duration right alone eliminates much of the “we can’t tell what won” situation. Here are the minimum statistics concepts worth knowing as a marketer.
Calculate sample size before you start
The habit of stopping when results look good makes you mistake chance for performance. Once you set your current conversion rate and the improvement you expect, you can calculate the required sample size up front, and the rule is not to judge results before reaching it.
Figures like “probability to win” that testing tools report are convenient, but the basis differs by tool. Don’t draw conclusions without knowing how the value was calculated; at a minimum, build the habit of having the team check sample size, duration, and conversion count directly.
Run for at least one or two full cycles
Visitors behave differently by day of the week, so running only a few days introduces bias. Include at least one full week, and two cycles if you can.
When writing a hypothesis, don’t keep it vague like “changing the title will improve things.” Include the reason, such as “a title with a specific number will raise click rate.” A hypothesis with a reason becomes the starting point for the next test whichever way the result goes.
- Test only one hypothesis at a time
- Don’t change the traffic split midway
- Avoid periods when promotions or campaigns overlap
The moment you change the design after seeing results, a test stops being evidence and becomes an excuse.
The Story Light editorial team
“Significant” does not mean “important”
Even a statistically significant difference can have a small impact on the business. Conversely, a result that isn’t significant because the sample was too small can still be worth noting for direction. Don’t conclude from a single number; look at the size of the effect and the cost together.
After a test ends, don’t stop at applying the winner; check whether other channels and other audiences give the same result. A single result strengthens a hypothesis but is not a law. Keep only what has been confirmed repeatedly as the team’s principle.
Decision criteria to set before the test
If you set criteria after seeing results, any number can be interpreted plausibly. So before starting a test, write in one line what you will count as success. For example: the primary metric is signup conversion rate, the secondary metric is bounce rate, and you will roll it out if it improves by at least a certain amount.
It’s also important to separate primary and secondary metrics. If you watch many metrics, one of them is bound to look good, and it’s easy to call that single one a win. It’s safer to decide on the primary metric you set in advance and use the rest only to check for side effects. Following this principle alone greatly reduces arguments over after-the-fact interpretation.
It also helps to write down criteria for not rolling out a change. When the improvement is too small to justify the operating burden, leaving things as they are can be the reasonable choice even if one version won. Rolling out a change has costs, too: design, development, copy updates. Factor those in when you decide.
Realistic approaches for low-traffic teams
On a site without many visitors, it’s hard to collect a sample large enough to detect small differences. In that case it’s better to test changes likely to have a big effect, such as the offer itself or the page structure, rather than tiny changes like button color. The larger the effect, the smaller the sample you need.
Here are methods you can use when your sample is limited.
- Narrow tests to pages close to the conversion step to concentrate your sample
- Don’t create more than two variants at a time
- Extend the duration, but don’t stop early to peek at results
- When quantitative results fall short, supplement them with user interviews or session recordings
If a test can’t give you confidence, treat the result not as settled fact but as a clue for the next hypothesis. For smaller teams, the ability to record and carry forward what you learn from tests matters more than the number of tests.
When designing a test, also ask in advance: “Whichever way this comes out, what will we do?” If the next action is the same either way, the test isn’t worth running. For a team with limited time, choosing which tests not to run is also an important call. It’s better to finish the highest-priority tests one at a time.
Summary
Briefly record the winning and losing variants and why you think each turned out that way. More than the success or failure of any single test, the team instinct built as these records accumulate is a far bigger asset in the long run.
Add your first comment to this post