Skip to content
Ecomwith

Incrementality Testing: Which Orders Are Truly New?

  • Intermediate
  • 55 minutes
  • Free account to read

Continues from: Creative Testing: High CTR but No Orders

What disappears without the spend?

Imagine an independent store serving commuters in the US Northeast with a 20oz stainless-steel tumbler. Branded Search has a $100 daily budget for 14 days, spends $1,260, and receives $4,440 of platform-attributed conversion value, or 3.52 ROAS. Reported orders do not automatically mean new orders. Fix the control and contamination boundaries before choosing a formal holdout, a light scale-down observation, or directional evidence only.

Platform reports show how much credit ads received, but not what the business would lose without them. First separate three outcomes: formal holdout, light scale-down observation, or directional evidence only when the sample is too weak.

Attribution assigns credit. Incrementality asks whether the business truly gained something new. The larger the spend, the more these questions must be separated; with weak samples, say possible, not proven.

Holdout thinking gives budget reviews a correction layer: if the platform says it won, would the business truly miss it?

  • Ready: Formal holdout
  • Observe first: Light scale-down
  • Cannot prove: Directional only

What would disappear without spend?

How do you measure ad incrementality? Start with what would disappear without spend.

Before changing spend, answer three questions: were attributed orders truly new, is a holdout workable, and what should you check when platform ROAS is high but total orders stay flat? The article ties test/control design, contamination logging, and Shopify business results to one budget decision.

Separate credit from new value

Incrementality testing asks: what would the business lose without this spend?

Fix test group, control group, time window, business outcome, exclusions, and contamination boundary first. Platform attribution can record touchpoint credit, but it cannot replace the counterfactual comparison of what happens without spend.

Retain groups, spend change, region or audience, orders, profit, leakage, and contamination records. With an insufficient sample, report directional evidence only; do not turn correlation into causal proof. Choose a formal holdout, a light scale-down observation, or no conclusion yet.

Show that the design can answer the counterfactual before taking its result to a budget meeting.

Understand lift in plain language before opening the terms

Start with one sentence: if this layer receives less or no spend, how many orders, new customers, or contribution profit would the business lose? Only a comparable region or audience kept away from the budget change gives you a chance to answer it.

Exercise · pick one

Try it first

Free account to read · Parts 3-16 · 14 parts

Members: full exercises and records

This part includes

  • Name the counterfactual before reading ROAS
  • Separate credit from incremental value
  • Report direction when the sample is small

1 checklist

An account is free; sign in to read this part.

Checking your membership…

What you finished

FAQ

What does ad incrementality testing mean?

It asks whether an order was newly created by ads or would have happened anyway. Platform attribution shows who received credit. Incrementality testing checks whether Shopify total orders, new customers, and contribution profit would fall if this spend was reduced or removed.

What is the difference between attribution and incrementality?

Attribution assigns credit inside Meta, Google, GA4, or Shopify reports. Incrementality asks what would disappear without the spend. Branded search, remarketing, returning customers, and blended PMax traffic can look strong in attribution while adding less new demand.

What is a holdout?

A holdout keeps one region, audience, time window, or demand layer away from the ad change, then compares it with an exposed group. The point is not a prettier report; it is to see whether ads created extra orders.

What to learn next

Return to attribution reconciliation and budget pacing before choosing the next incrementality step.

Incrementality asks whether advertising created new demand. Check attribution definitions, business totals, and comparable scope before choosing whether to protect, reduce, or retest the layer.

Return to the topic Hub: Advertising analysis paths

Sources

Edition notes

  • Author Ranfeng Wei
  • Published
  • Updated
  • Reviewed
  • Next review

This update: Restored the FAQ and tidied the source list.