M10.L1 · Analytics & Experiments
Numbers that matter: KPIs vs vanity metrics
What you'll be able to doConstruct a three-level KPI tree — business outcome → marketing KPI → leading indicator — for your project, and justify excluding at least three vanity metrics.
Learn
The question that decides your budget
At some point someone who controls money will ask you: "is the marketing working?"
The wrong answer is a follower count. The wrong answer is a reach graph. Lena gave the wrong answer once — the month one LinkedIn post reached about 2,600 accounts against a usual 450, her status line to the founders read "LinkedIn reach up more than fivefold." It was true. And nothing in the business had changed: same 41 trials, same 9 new customers, same MRR. The next month, when reach fell back, she had taught her founders that marketing numbers go up and down for no reason.
This module is where you finally get real measurement — and where you find out whether the leak hypothesis you wrote in your Module 1 funnel baseline was right. This first lesson decides what you'll measure, because tools come after choices, not before.
A metric is a decision waiting to happen
Here is the only test a metric ever needs to pass:
If this number moved sharply, would we do anything differently?
If the answer is yes — cut a budget, change a page, ring a customer — the metric earns a place. If the answer is no, it is decoration, however impressive it looks.
That test also gives you the honest definition of a vanity metric. A vanity metric is not a metric that is big. It is a metric that survives being wrong. Reach could halve tomorrow and Lena would change nothing. Trial starts halving would have her rebuilding the landing page by Friday. Same dashboard, entirely different class of number.
Vanity metrics share three habits: they only ever get reported when they rise, they cost nothing to inflate, and they predict nothing downstream. Impressions, followers, raw pageviews, "reach" — all four fail the decision test for almost every business, which is why they appear in almost every report.
KPIs, and the two directions of time
A KPI is a metric you decided in advance would define success — chosen before the period, not discovered afterwards when you go looking for something that went up. Two to four is the working number. More than that and the word loses its meaning.
KPIs come in two tempos. Lagging indicators tell you what already happened: revenue, new customers, retention. Reliable, and too late to act on. Leading indicators move weeks earlier: trial starts, enquiries, add-to-carts, landing-page sessions. Less certain, but still steerable. You need both — the lagging number to know the truth, the leading number to do something about it in time.
The KPI tree: nothing dangles
A KPI tree is one page connecting three levels:
One business outcome — the thing the business actually wants (new MRR, booked patients, online revenue) → 2–4 marketing KPIs — the funnel numbers that drive it → 1–3 leading indicators per KPI — the earlier numbers that predict each one
The rule that gives the tree its power: nothing dangles. Every metric on the page must hang off the level above it. If you can't say which KPI a number feeds, it doesn't go on the tree — and if it doesn't go on the tree, you don't report it. That single rule deletes most of what marketing dashboards usually contain.
You have met this structure before. Your Module 1 funnel baseline is the middle of your tree: found → acted → bought are your candidate KPIs, and the leak you diagnosed in M1.L6 tells you which one carries this quarter's target.
Goodhart's Law, live at Ostara
There is a trap waiting inside every KPI, and it has a name. Goodhart's Law: when a measure becomes a target, it stops being a good measure — because people, including you, will find the cheapest way to move the number rather than the thing the number stood for.
The cast has a canonical example. In Mateus's first months, the only number Ana asked him for was email signups per month. So he optimised it: the 10%-off pop-up started firing on arrival instead of exit, and signups roughly doubled for two months. Target hit. Only when Module 8's segment split arrived did the cost surface: the discount-seeker cohort clicked at 1.2% against 4.1% for everyone else, and unsubscribed at roughly three times the rate after first purchase. The list grew; the audience didn't. Nobody lied. The measure was simply obeyed instead of the mission.
The defence is a counter-metric: a second number watched specifically to catch your KPI being gamed. Ostara's list-growth target now travels with one — the click rate of each new cohort's first two broadcasts. Lead volume pairs with lead quality; trial starts pair with ICP fit; bookings pair with show-up rate. One counter-metric per tree is the minimum dose.
A related idea you'll meet in the wild: the North Star Metric, a single number chosen to proxy delivered customer value. Useful discipline; dangerous as the only number, for exactly the Goodhart reasons above.
Lena's tree, in full
Outcome: new MRR added per month KPI 1: trial starts/month — current 41, target 55 within two quarters KPI 2: trial→paid rate — current 22%, target 25% Leading indicators: landing-page sessions/week (via
?src=lp) · share of trials starting a real client intake in week one Counter-metric: share of new trials matching the ICP (accountancy practices, 3–25 staff, NL/BE) Refused, in writing: LinkedIn reach · company-page followers (640) · blog page views
Notice what the counter-metric protects: Lena could hit 55 trials by attracting freelance bookkeepers who will never pay — volume up, business unchanged. The ICP share catches it. Notice too that reach appears nowhere, even though it is her biggest number.
Tomas's contrast tree is smaller and just as complete: outcome = new patients across both clinics; KPI = Check-up bookings/month; leading indicator = booking-form views; counter-metric = share of bookings that attend. The vanity metric he strikes off: Instagram followers — 340 of them, dormant since March, predicting nothing.
One more payoff. Writing her tree forced Lena to finally build a retention cohort table — and it confirmed what M1.L5 suspected: the 2.4% churn figure and the 14-month lifetime disagree because early cohorts haven't had time to age, not because either is wrong. Some arguments end the moment someone draws the table.
Now build yours.
Do
Exercise 10.1.1 — Your KPI tree
Build your KPI tree. Name the one business outcome your marketing exists to move, the 2–4 KPIs that drive it, and at least one leading indicator per KPI. Add one counter-metric. Then name three metrics you are explicitly refusing to report, and why.
Write these down — in your plan document, or on the worksheet at the end of this lesson.
| What to write | Guidance |
|---|---|
| One business outcome | 3–15 words, with a noun and a direction — e.g. "increase monthly online revenue". One outcome only |
| Your 2–4 KPIs | For each: its name · its current value (a number, or "unknown" — honest, and a to-do for M10.L2) · a target value · a target date no more than 12 months out |
| Leading indicators | At least one per KPI, each naming which KPI it feeds. Leave no KPI without an indicator — nothing dangles |
| One counter-metric | 5–25 words, and it must name what failure it detects — e.g. "ICP share of trials — catches volume gamed with out-of-profile signups" |
| Three vanity rejects | Exactly three metrics you refuse to report, each with 15–60 words on why. "It's small" is not a reason; "it survives being wrong" is |
When the tree is written, sketch it as a diagram and check two things: no metric dangles, and the stage you named as your leak in M1.L6 is covered by a KPI.
Where this goes: Measurement §1 — KPIs — of your Marketing Plan. Your KPI targets return in the M10.L5 report and the Module 10 Project.
Check
Rubric
Mark your own work against these criteria.
| Criterion | 8–10 | 5–7 | 1–4 |
|---|---|---|---|
| Causal chain | Every indicator plausibly moves its KPI; every KPI plausibly moves the outcome | Chain exists but one link is asserted rather than argued | Metrics listed side by side with no linkage |
| Vanity-free | Nothing on the tree survives the decision test; rejects genuinely tempting | One decorative metric smuggled on | Followers or impressions appear as KPIs |
| Counter-metric quality | Names a specific gaming failure it would catch | Present but generic ("quality") | Missing, or restates a KPI |
| Targets | Dated, and justified from the M1.L6 baseline or explicit estimation logic | Dated but plucked from the air | No targets, or "more" |
Pass: 5+ on every criterion. Distinction: 8+ on all four.
Quiz — 4 questions
1. Herzog Physio wants more booked appointments. Which is the best leading indicator?
- a) Monthly revenue
- b) Check-up bookings requested through the website form ✔
- c) Instagram followers
- d) Sessions completed last quarter
Why: bookings requested predict revenue weeks ahead and can be acted on now. Revenue and completed sessions are lagging — true but too late. Followers fail the decision test entirely.
2. Ostara set "email signups per month" as Mateus's target. He made the discount pop-up more aggressive; signups doubled while the new cohort's click rate and post-purchase retention collapsed. This is an example of:
- a) Attribution error
- b) Goodhart's Law ✔
- c) Statistical insignificance
- d) Survivorship bias
Why: the measure became the target and was obeyed instead of the mission. The number went up; the thing it stood for went down. A counter-metric — new-cohort click rate — is the standing defence.
3. What is the single test that tells you whether a metric belongs on your report?
- a) Whether it has grown this quarter
- b) Whether competitors report it
- c) Whether you would do anything differently if it moved sharply ✔
- d) Whether your analytics tool can chart it
Why: a metric is a decision waiting to happen. If no movement in it would change your actions, it is decoration — whatever its size, direction or chart type.
4. Why does a KPI tree need a counter-metric?
- a) Because analytics tools require an even number of metrics
- b) To have a backup KPI if the first stops growing
- c) Because a targeted measure invites gaming, and a paired metric catches the cheap way of hitting it ✔
- d) To measure competitors as well as yourself
Why: Goodhart's Law is not about dishonesty — it's about optimisation finding the shortest path. The counter-metric watches the thing the shortest path sacrifices: quality, fit, retention.
Advance
One of five. You now have the page that answers "is the marketing working?" before anyone asks — and you know which numbers you're refusing to discuss.
Next: M10.L2 — GA4 without tears. Your tree has "unknown" written in at least one current-value box. The next lesson is about filling those boxes with numbers you can defend — and knowing exactly how those numbers lie.
Mark your own work
| Good | Not yet | |
|---|---|---|
| One outcome | A single business outcome with a direction | Three outcomes, or "grow the brand" |
| Nothing dangles | Every metric hangs off the level above | A metric you like but can't connect |
| Counter-metric bites | It names the failure it detects | A second KPI wearing a costume |
| Rejects are tempting | You refused metrics that flatter you | You refused metrics nobody reports anyway |
Worksheet
THE SCHOOL OF NET MARKETING
Lesson 10.1 — Your KPI tree
THE DECISION TEST
A metric earns its place only if: "if this number moved
sharply, we would do something differently."
A vanity metric is not a metric that is big.
It is a metric that survives being wrong.
MY TREE
BUSINESS OUTCOME (one, with a direction)
____________________________________________
KPI 1 ______________________ now ______ target ______ by ______
leading indicator: ______________________________
KPI 2 ______________________ now ______ target ______ by ______
leading indicator: ______________________________
KPI 3 (optional) ___________ now ______ target ______ by ______
leading indicator: ______________________________
COUNTER-METRIC (what gaming does it catch?)
____________________________________________
REFUSED — three metrics I will not report, and why
1. _______________________ because ____________________
2. _______________________ because ____________________
3. _______________________ because ____________________
SELF-CHECK
☐ One outcome, 2–4 KPIs, nothing dangles
☐ Every KPI has a leading indicator
☐ My counter-metric names the failure it detects
☐ My targets have dates and reasons
☐ My M1.L6 leak stage is covered by a KPI
Next: Lesson 10.2 — GA4 without tears.
theschoolofnetmarketing.com/learn/ga4-without-tears