A/B testing

What it means

An A/B test splits traffic in two: one branch sees the original, the other the change. Everything else stays the same, so a difference in the conversion rate can be attributed to that one change.

It only works with enough traffic for the difference not to be noise. At ten orders a month a test proves nothing, and the decision should come from talking to customers instead.

An A/B test is not a comparison of this month with last month. A before-and-after view mixes in the season, a different campaign or a competitor’s outage. In an A/B test both versions run at the same time on the same audience, so outside influences hit both branches equally and web analytics shows the clean difference.

How it works in practice

Test one thing at a time: the headline, the price, the length of the form, where the call to action sits. Change five things at once and a result still arrives, but nobody knows which change produced it.

On a landing page behind paid advertising the payback is largest, because every point of conversion cuts the cost per order directly. Measure at the order, not at the click.

With webshops we make sure the result of a test can also be checked against the figures in the books. The branch that brought more orders but more returns or a deeper discount does not necessarily earn the company more. So the margin per order and the return rate are compared too, not just the order count in the testing tool.

What to watch out for

The expensive mistake is stopping a test the moment one branch looks better. Early days always swing; decide after a pre-agreed number of conversions, not after the first flattering chart.

The second problem is who gets into the test at all. If the tool only measures visitors who accepted cookies, it works with a sample, not with the whole traffic. And if it fires before the visitor has given cookie consent, it may breach privacy rules, which makes it a compliance issue in an inspection, not merely an inaccurate number.