Conversion maths, A/B tests & statistical significance

The cheapest fraud detector you will ever own

Most "our new page won!" stories are noise read as signal. This lesson's maths — one formula and a calculator — tells you before a test starts whether it can possibly answer your question. Five minutes with it will stop you, or an agency, declaring victory after three lucky days. As in Module 1: a rough number beats no number, and here the rough number is spectacularly cheap to get.

First, the ground rule of conversion arithmetic. A conversion rate is conversions ÷ eligible visitors, and it means nothing until you name all three parts: numerator, denominator, time window. Ostara: 91 orders ÷ 6,500 sessions in a typical month = 1.4%. Change any part — orders per user instead of session, a peak month instead of a typical one — and you get a different, equally true number. Most conversion-rate arguments are two people using different denominators.

Why small numbers wobble

Here is the uncomfortable core of the lesson. Suppose a page truly converts at 3%. Show it to 100 people and you expect 3 conversions — but 1 or 6 would be entirely unremarkable, the way 10 coin flips often produce 7 heads. Small samples wobble, and the wobble is routinely bigger than the improvement you're trying to detect. An A/B test is a machine for collecting enough evidence that the wobble can no longer explain the difference. The question is how much is enough — and the answer is almost always "far more than you hoped".

Four terms make the machine work:

• Minimum detectable effect (MDE): the smallest improvement you care about, e.g. +20% relative (3.0% → 3.6%). Halving the MDE roughly quadruples the traffic needed — this is the lever that decides everything.
• Significance (α = 0.05): the convention that you'll accept a result only if there's under a 5% chance of seeing a gap this large when none exists. A noise filter, not proof of importance.
• Power (80%): the probability the test detects a real effect of MDE size. Underpowered tests mostly return false "no difference" — the quiet failure nobody notices.
• Sample size: computed from the three numbers above, before the test starts. Never decided by impatience afterwards.

The worked example — every line shown

How many visitors to detect 3.0% → 3.6% at α = 0.05 and 80% power? The standard two-proportion formula, per arm:

n = (z_α/2 + z_β)² × [p₁(1−p₁) + p₂(1−p₂)] ÷ (p₂ − p₁)²

z_α/2 = 1.96 (95% significance)   z_β = 0.84 (80% power)
p₁ = 0.030 → p₁(1−p₁) = 0.0291
p₂ = 0.036 → p₂(1−p₂) = 0.0347
(p₂ − p₁)² = 0.006² = 0.000036

n = (1.96 + 0.84)² × (0.0291 + 0.0347) ÷ 0.000036
  = 7.85 × 0.0638 ÷ 0.000036
  ≈ 13,900 visitors per arm  →  ≈ 27,800 in total

Call it roughly 13,000–14,000 per arm — online calculators land across that range depending on the approximation, and the difference never changes a decision. You will never do this by hand again; you will recognise when someone hasn't done it at all.

Now watch the MDE lever move:

Subtle improvements are astronomically expensive to prove. Big swings are cheap. Hold that thought.

Now apply your own traffic

Divide the total by monthly eligible visitors and the verdict falls out:

• Ostara — 6,500 sessions/month sitewide: 27,800 ÷ 6,500 ≈ 4.3 months, and that's the best case where every session enters the test; a single page sees a fraction of that. At the sitewide 1.4% baseline the same +20% test needs ~30,400 per arm — over nine months.
• Herzog — ~800 visitors/month: 27,800 ÷ 800 ≈ 35 months. Nearly three years to test one button.
• Storkflow's landing page — ~118 visitors a fortnight: about ten years.

This is not a defect in the cast. It is the honest arithmetic of most small businesses, and it leads to the canonical verdict Tomas records: Herzog does not A/B test. Not because testing is bad — because at his scale the method physically cannot answer within a useful lifetime. Knowing when a method doesn't fit is statistical literacy; forcing it anyway is cargo cult.

Don't call it early — the whole point

Here is a test replayed as a simulation — no cast business has the traffic to have run it, which you now know how to check. Pre-computed sample: 13,900 per arm.

Day 5 — 900 visitors per arm. A: 24 conversions (2.7%). B: 42 (4.7%). The dashboard glows: B up 75%, p = 0.02, significant! The agency wants to ship it.

The class checks: 900 of 13,900 is 6% of the committed sample. They wait.

At full sample — A: 417 of 13,900 (3.0%). B: 446 (3.2%). Lift +7%, p ≈ 0.32. Not significant.

What happened on day 5 was not evidence; it was peeking. Check a running test daily and stop at the first p < 0.05, and you multiply your false-positive rate several times over — you are giving noise thousands of chances to cross the line once. Notice also the tell you can spot without any maths: at small samples, only huge lifts can look significant. A real +75% would mean your original page was a catastrophe; dramatic early winners are almost always noise wearing a medal.

The discipline, fixed before launch: compute the sample, run whole weeks (weekday and weekend visitors differ), read the result once. The early call here would have booked a fictional +75% into next year's forecast; the honest final reading says "if B is better at all, it's by less than our +20% MDE — ship it for qualitative reasons or move on." A non-significant test is not a failed test. It taught you the effect's ceiling.

And keep practical significance in view: with enough traffic a +0.4% lift can be statistically real yet not worth its maintenance; without enough, a real +15% can hide. Significance filters noise. Judgement still decides value.

What to do instead — Plan B is not a consolation prize

If the calculator says months, do what the cast does:

1. Make bigger swings. Don't test button colours; change the offer, the headline promise, the entire page — the +100% row of the table costs 750 per arm. Big changes are the only testable ones at small scale.
2. Before/after with guard metrics. Ship the change, compare whole months, watch a counter-metric (lead quality, refund rate) — Tomas's method, and the M1.L6 hypothesis format you already write: I believe X because Y; I'll watch Z for four weeks.
3. Qualitative evidence. Five user recordings or five customer conversations will find the broken thing faster than 27,800 visitors will prove it — M2 methods never stop paying.
4. Sequential judgement. Decide, observe honestly, keep what earns its keep. That is what Module 1 called hypothesis discipline; it's also what your funnel baseline was for — this module's measured numbers versus that page is the before/after test you've been running all along.

---