Would it have happened anyway? Incrementality vs attribution in ad spend

1. Before you start

Two words decide whether an advertising dollar was worth spending, and they are easy to confuse. Attribution is the bookkeeping question: which ad, click, or channel gets credit for a conversion? Incrementality is the harder question underneath it: would that conversion have happened anyway, even with no ad at all? A tiny example. Suppose 100 people who searched your brand name and clicked your paid ad went on to subscribe. Attribution credits the ad with 100 subscriptions. But if 85 of those people would have found you and subscribed regardless — through the free link right below the ad, or because they already meant to — then only 15 are incremental. The ad’s real work was 15, not 100. The other 85 is credit it borrowed from demand that already existed.

Three honest statements before you start:

This is a Decide course. You read a situation, learn the ideas, and make a call. You do not write or run any code.
The companies below, Northwind Media and Cadence Audio, are composite — invented firms built from ordinary, illustrative figures so the reasoning is clean. No number here is a claim about any real company.
This is not a certification. It proves, to you, that you can tell borrowed credit from real lift and defend a spend decision.

You need no arithmetic beyond subtraction and division. The whole difficulty is judgment: telling correlation apart from causation when a dashboard is showing you a very flattering correlation.

2. The Situation

Northwind Media, a composite subscription streaming service, spends $200,000 a month bidding on its own brand name in search so that “Northwind” queries land on a paid ad. The last-click dashboard credits that campaign with 10,000 new subscribers a month — a cost of just $20 each, the best number on the report. Finance wants to know whether that $200,000 is buying growth or buying credit for growth that was coming anyway, and the answer decides whether the budget stays, grows, or gets moved.

The trap is that the most flattering channel on an attribution report is often the one sitting closest to a decision the customer had already made — so it looks like the engine when it may only be the odometer.

3. What you’ll be able to do

After this course you will be able to:

State the counterfactual a spend decision actually turns on — “would this conversion have happened anyway?” — and say why an attribution number cannot answer it.
Explain how last-click and multi-touch attribution over-credit channels that harvest existing demand (brand search, retargeting), and spot that pattern on a dashboard.
Describe how a holdout or geo test measures true lift, and read its result: incremental conversions, incrementality rate, and incremental ROAS.
Make and defend a reallocation call when attribution and incrementality disagree, and name the one result that would change your mind.

4. Prerequisites & time box

Prerequisites: none beyond comfort reading a small table of numbers and the idea that an advertising channel is supposed to cause sales, not just be present near them. No spreadsheet or setup — the Decide hall is read-and-decide in the browser; see the Decide hall’s how-to-read page if this is your first concept course. No prior Decide course is assumed.

Time box: about 21 minutes of reading (measured), plus real thinking time on the call in section 7. That is under the 25-minute cap for a concept course.

Difficulty: 6 / 8 — a manager-level decision: several factors move at once and the naive reading points the wrong way, so you have to reason past it.

Free-tier honesty: no signups, no paid tools, requires_gpu: false. Nothing here costs money to learn.

5. The case & where the numbers come from

Northwind Media is a composite media company: its spend, subscriber, and test figures are in-course illustrative assumptions, chosen for clean arithmetic and clearly labelled as such — not drawn from or claimed about any real firm. The methods — attribution, the counterfactual, holdout and geo experiments, difference-in-differences, incremental ROAS — are standard and cited in section 11. Every subtraction and division below is done inside the course from these stated figures, so you can reproduce each one by hand.

The brand-search campaign, as the dashboard reports it (all figures illustrative):

Item	Figure (illustrative)
Monthly spend on brand search	$200,000
Subscribers last-click attributes to it	10,000
Attributed cost per subscriber	$200,000 ÷ 10,000 = $20
Value of one incremental subscriber (assumed)	$180

Hold the $20. It is the number that misleads. Everything in section 6 is about finding out how many of those 10,000 subscribers the campaign actually caused.

6. The Concepts

The counterfactual

The only honest test of an advertising dollar is a comparison against a world where you did not spend it. That imagined world — what would have happened anyway — is the counterfactual. When Northwind pays to put an ad on searches for “Northwind,” the person typing that query has already named the brand; they are, by definition, demand that already exists. The counterfactual question is: if the paid ad had not been there, would they have subscribed regardless — clicked the free organic result an inch below, or come back later? For most brand-name searchers the honest answer is yes.

This is the difference between correlation and causation, dressed in marketing clothes. The ad click and the subscription are correlated — they happen together, in that order — but correlation is not proof the ad caused the subscription. The subscription may have caused the click (someone who had decided to subscribe went looking for the site) rather than the other way around. A number that counts co-occurrence, not causation, cannot tell you which. Incrementality is just the discipline of insisting on the counterfactual before you claim credit.

How attribution over-credits existing demand

Attribution assigns credit for a conversion to the touchpoints that preceded it. The most common rule, last-click, hands 100% of the credit to the final click before the conversion. Its blind spot is structural: whichever channel sits closest to the moment of conversion collects the credit, whether or not it created the demand. Brand search sits at the very bottom of the funnel — it meets people who have already decided to look you up — so it systematically harvests credit for demand built elsewhere (a show people loved, a friend’s recommendation, an intent formed weeks ago). Multi-touch attribution spreads the credit across more touchpoints but shares the same flaw: it divides up credit among clicks that were observed, and never asks the counterfactual about any of them. Both models answer “who was near the conversion,” not “what changed because of the spend.”

That is why Northwind’s brand-search line looks like its best channel at $20 a subscriber. It is not that the number is miscalculated; it is that the number is answering the wrong question. The flattering channels on almost any attribution report tend to be the demand-harvesting ones (brand search, retargeting), not the demand-creating ones — precisely because harvesting happens last and last-click rewards whoever is last.

Holdout and geo tests isolate true lift

To measure what the spend actually caused, you stop reading the dashboard and run an experiment: you withhold the ads from a comparable group and watch what they do without them. That withheld group is a holdout — a live, measured version of the counterfactual. The cleanest form randomizes which users see ads; a practical and common form for media budgets is a geo test, splitting matched markets and comparing them (a difference-in-differences read: the gap between the group that got ads and the group that did not).

Northwind runs one. It picks 20 matched metros — similar in size, subscriber base, and trend — and splits them into two equal groups of 10, holding everything else constant for one month:

Group (10 metros each)	Brand-search ads	New subscribers
Treatment	on	5,000
Holdout	off	4,250

The holdout’s 4,250 is the counterfactual, now measured rather than imagined: that many people subscribed in comparable markets with no brand-search ads at all. So in the treatment group, the ads’ true contribution is the difference: 5,000 − 4,250 = 750 incremental subscribers. The other 4,250 in the treatment group would have subscribed anyway. Because the two halves are matched and equal, running ads across all 20 metros produces roughly 750 × 2 = 1,500 incremental subscribers — out of the 10,000 that last-click attribution had claimed.

Reading lift: iROAS and the limits

Now put the two numbers side by side. The incrementality rate is the share of attributed conversions that were truly caused: 1,500 ÷ 10,000 = 15%. Everything else was borrowed credit.

Attributed cost per subscriber: $200,000 ÷ 10,000 = $20.
Incremental cost per subscriber: $200,000 ÷ 1,500 ≈ $133.

Incremental ROAS (iROAS) compares incremental revenue to spend. At the assumed $180 per incremental subscriber, incremental revenue is 1,500 × $180 = $270,000, so iROAS = $270,000 ÷ $200,000 = 1.35. The attributed figure, crediting all 10,000, would have read 10,000 × $180 ÷ $200,000 = 9.0. Same campaign, same month — a 9.0 story and a 1.35 reality. The call turns on the 1.35, because that is the only one measured against a world without the spend.

The limits keep you honest, in both directions:

A geo test measures the tested conditions, not a universal truth. Change the creative, the season, the competitive pressure, or the saturation level and the lift can change. Incrementality is a reading, not a constant.
Matched markets and no spillover are assumptions. If holdout metros still saw the brand another way (national TV, word of mouth crossing metro lines), the measured gap understates or distorts the true lift.
A small measured lift is not the same as zero. 15% incremental at a 1.35 iROAS is thin, not worthless. The honest move on a result like this is usually to reallocate toward demand creation, not to switch the channel off and assume the 15% vanishes for free.

The lesson is blunt: an attributed number tells you where a conversion was recorded; only a holdout tells you whether the spend changed anything. When they disagree, trust the one built on a counterfactual.

7. Your Call

You have seen how the counterfactual, attribution’s over-crediting, and a holdout test decide Northwind’s brand-search budget. Now a different decision lands on your desk.

Cadence Audio is a composite podcast and audio-streaming network with a fixed monthly acquisition budget of $150,000 that cannot grow this quarter. Today $100,000 of it runs a social retargeting campaign — ads shown to people who already visited Cadence’s site or began a signup — and the remaining $50,000 sits in untested prospecting to people who have never heard of Cadence. Last-click attributes 8,000 signups to retargeting, a cost of $12.50 each, the headline “best” channel. Under pressure to grow, the head of growth wants to move the untested $50,000 into retargeting too.

Before deciding, Cadence runs a geo holdout on retargeting across matched markets, one month, every other factor held constant (all figures illustrative):

Group	Retargeting	Signups
Treatment	on	4,000
Holdout	off	3,700

How this differs from the taught case (the transfer): this is a different company and sector (Cadence Audio, a podcast/audio network, not Northwind’s video streaming), the figures are different so the arithmetic must be redone rather than recalled, it is a different kind of decision (reallocate a fixed budget between two channels, not keep-or-trim a single one), and it adds a constraint the taught case did not have (the total budget is fixed and the two uses compete for it). The core concept is the same: incrementality — would the conversion have happened anyway — versus what attribution credits.

8. Self-check

Before you write the memo, make sure you can say each of these in one line:

What is the counterfactual for a given campaign, and why can no attribution number answer it?
Why do brand search and retargeting tend to look like the best channels on a last-click report?
How does a holdout or geo test turn “would it have happened anyway?” into a measured number, and what is the incrementality rate you read from it?
What is the one result — from a clean test — that would change your reallocation call?

If any is fuzzy, reread section 6: the counterfactual, attribution’s over-crediting, the holdout, and reading the lift are the whole course.

9. Stretch

Push the thinking further on your own:

Back at Northwind: brand search came in at 15% incremental. Would you expect generic (non-brand) search — bidding on “streaming service,” where searchers have not named you — to be more or less incremental, and why? (The genuinely harder one: think about where in the funnel the channel meets the customer, and what that implies for which channels a holdout will flatter or punish.)
Cadence’s holdout showed a small positive lift. Design the follow-up test that would tell you how far you can cut retargeting before the lift disappears — what would you vary, and what result would mark the floor?
A colleague says “if we just switch from last-click to a multi-touch model, we’ll fix this.” Would that measure incrementality? Say precisely what multi-touch does and does not change.

10. Ship it — your decision memo

Write a one-page memo to Cadence’s leadership. State the call (reallocate a portion of the $100,000 retargeting budget toward prospecting, and test the prospecting before scaling it — do not add the untested $50,000 to retargeting). Show the reasoning in two or three lines (last-click credited retargeting with 8,000 signups at $12.50, but a matched-market holdout put the incremental effect near 600 signups at roughly $167 each; with a fixed budget the next dollar belongs where it is more likely to cause a signup). Name what you rejected (pouring more into retargeting on its flattering attributed cost) and why. Name the one thing that would change your mind (a clean test showing retargeting’s lift is far larger than 7.5%, or prospecting testing worse than retargeting’s true incremental cost). Keep it to a single page leadership grasps in two minutes. This memo is your own argued claim — not a credential.

11. Sources

Northwind Media and Cadence Audio, and every figure attached to them, are composite and illustrative — constructed for clean teaching arithmetic, not drawn from or claimed about any real company. The methods used to reason about them are standard marketing measurement and causal inference; references below.

Concept / claim	Source (publisher)	URL	Accessed
Counterfactual / potential-outcomes reasoning (“would it have happened anyway”)	Wikipedia — Rubin causal model	https://en.wikipedia.org/wiki/Rubin_causal_model	2026-07-19
Counterfactual thinking, plain sense	Wikipedia — Counterfactual thinking	https://en.wikipedia.org/wiki/Counterfactual_thinking	2026-07-19
Correlation is not causation	Wikipedia — Correlation does not imply causation	https://en.wikipedia.org/wiki/Correlation_does_not_imply_causation	2026-07-19
Attribution / conversion crediting in digital marketing	Wikipedia — Conversion marketing	https://en.wikipedia.org/wiki/Conversion_marketing	2026-07-19
Holdout / controlled experiment for lift	Wikipedia — A/B testing	https://en.wikipedia.org/wiki/A/B_testing	2026-07-19
Randomized controlled experiment as the causal gold standard	Wikipedia — Randomized controlled trial	https://en.wikipedia.org/wiki/Randomized_controlled_trial	2026-07-19
Geo test read as difference between groups	Wikipedia — Difference in differences	https://en.wikipedia.org/wiki/Difference_in_differences	2026-07-19
Measured lift as an average treatment effect	Wikipedia — Average treatment effect	https://en.wikipedia.org/wiki/Average_treatment_effect	2026-07-19
Return on marketing/ad spend as a ratio of return to spend	Wikipedia — Return on marketing investment	https://en.wikipedia.org/wiki/Return_on_marketing_investment	2026-07-19

Next up

Finished this call? Continue the Marketing track:

One price, unknown demand: pricing a perishable room block by expected value → · Browse all courses