16 parts · 2 checkpoints · 2 free / 14 member parts →
Continues from: Creative Testing: High CTR but No Orders
What disappears without the spend?
Imagine an independent store serving commuters in the US Northeast with a 20oz stainless-steel tumbler. Branded Search has a $100 daily budget for 14 days, spends $1,260, and receives $4,440 of platform-attributed conversion value, or 3.52 ROAS. Reported orders do not automatically mean new orders. Fix the control and contamination boundaries before choosing a formal holdout, a light scale-down observation, or directional evidence only.
Platform reports show how much credit ads received, but not what the business would lose without them. First separate three outcomes: formal holdout, light scale-down observation, or directional evidence only when the sample is too weak.
Attribution assigns credit. Incrementality asks whether the business truly gained something new. The larger the spend, the more these questions must be separated; with weak samples, say possible, not proven.
Holdout thinking gives budget reviews a correction layer: if the platform says it won, would the business truly miss it?
- Ready: Formal holdout
- Observe first: Light scale-down
- Cannot prove: Directional only
What would disappear without spend?
How do you measure ad incrementality? Start with what would disappear without spend.
Before changing spend, answer three questions: were attributed orders truly new, is a holdout workable, and what should you check when platform ROAS is high but total orders stay flat? The article ties test/control design, contamination logging, and Shopify business results to one budget decision.
This lesson answers
Separate credit from new value
Incrementality testing asks: what would the business lose without this spend?
Fix test group, control group, time window, business outcome, exclusions, and contamination boundary first. Platform attribution can record touchpoint credit, but it cannot replace the counterfactual comparison of what happens without spend.
Retain groups, spend change, region or audience, orders, profit, leakage, and contamination records. With an insufficient sample, report directional evidence only; do not turn correlation into causal proof. Choose a formal holdout, a light scale-down observation, or no conclusion yet.
Show that the design can answer the counterfactual before taking its result to a budget meeting.
Understand lift in plain language before opening the terms
Start with one sentence: if this layer receives less or no spend, how many orders, new customers, or contribution profit would the business lose? Only a comparable region or audience kept away from the budget change gives you a chance to answer it.
Try it first
Branded terms have strong CTR, CVR, and ROAS while taking more spend.
- First test
- Start with branded-demand holdout, geo scale-down, or bid reduction and watch total orders and new customers.
- Do not do
- Do not use branded ROAS as proof for acquisition spend.
Returning customers, visitors, cart audiences, and email audiences drive a large part of platform-attributed revenue.
- First test
- Isolate remarketing or returning-customer layers and read new-customer mix, non-paid orders, and repeat-purchase margin.
- Do not do
- Do not keep cutting cold traffic just because remarketing CPA is low.
PMax beats other campaigns but brand, Shopping, remarketing, and new-customer shares are unclear.
- First test
- Start with brand controls, Search/Shopping baselines, and new-customer quality before judging PMax lift.
- Do not do
- Do not treat blended automation ROAS as a single proof of incrementality.
Orders are sparse, tracking is unstable, or profit definitions are unclear, making any isolation falsely precise.
- First test
- First repair order, UTM, profit, new-customer, and refund definitions, then run a small observation.
- Do not do
- Do not pause spend just to say an experiment was done.
Free account to read · Parts 3-16 · 14 parts
Members: full exercises and records
This part includes
- Name the counterfactual before reading ROAS
- Separate credit from incremental value
- Report direction when the sample is small
1 checklist
An account is free; sign in to read this part.
Checking your membership…
FAQ
- What does ad incrementality testing mean?
It asks whether an order was newly created by ads or would have happened anyway. Platform attribution shows who received credit. Incrementality testing checks whether Shopify total orders, new customers, and contribution profit would fall if this spend was reduced or removed.
- What is the difference between attribution and incrementality?
Attribution assigns credit inside Meta, Google, GA4, or Shopify reports. Incrementality asks what would disappear without the spend. Branded search, remarketing, returning customers, and blended PMax traffic can look strong in attribution while adding less new demand.
- What is a holdout?
A holdout keeps one region, audience, time window, or demand layer away from the ad change, then compares it with an exposed group. The point is not a prettier report; it is to see whether ads created extra orders.
What to learn next
Return to attribution reconciliation and budget pacing before choosing the next incrementality step.
Incrementality asks whether advertising created new demand. Check attribution definitions, business totals, and comparable scope before choosing whether to protect, reduce, or retest the layer.
Return to the topic Hub: Advertising analysis paths