Start with the hypothesis, primary metric, sample, variable, and counter-signal gates before trying more versions. Low-sample pages can improve, but report the result within its evidence boundary.
22 parts · 1 checkpoint · 2 free / 20 member parts →
Key point In one line
Do not turn random noise into a win
- Output: Experiment discipline gate
- Pass rule: Can test, observe, or defer
- Next lesson: Monthly CRO roadmap
How to run Shopify A/B tests: set sample, primary metric, and stop rules first
This is for Shopify teams prioritizing experiments, working with small samples, or unsure when to stop. Decide whether the problem needs formal A/B testing, directional validation, qualitative validation, or direct repair before reading a result.
This lesson answers
Classroom case: a +20% lift on 500 visits
- Visits per arm
- 500
- Arm A add-to-carts
- 40
- Arm B add-to-carts
- 48
- Extra add-to-carts
- 8
Arm A gets 40 add-to-carts from 500 visits; arm B gets 48: 8.0% versus 9.6%, a 20% relative lift and only 8 extra add-to-carts. With 500 visits per arm, those 8 are more likely noise or a same-period campaign difference than proof that the page version won.
Low sample can improve the page, but it is not formal A/B proof
Put these five rules before the test. They do not make testing heavier; they stop teams from packaging random movement, campaign changes, tracking drift, or a few orders as experiment wins.
Low sample can improve the page, but it is not formal A/B proof
Low sample
- Rule
- Low sample can guide optimization, but it is not a formal A/B conclusion.
- Allowed
- Use controlled repair, directional observation, qualitative validation, or wait for a stronger sample window.
- Do not
- Do not turn a few orders into statistical significance, certainty, or proof that the variant is better.
No peeking
- Rule
- Readout date, runtime, and stop rule must be written before launch.
- Allowed
- Fixed-horizon default: monitor tracking, allocation and severe guardrail harm, not temporary ordinary significance as a win. A preselected sequential design follows its own stopping rules.
- Do not
- Do not stop or move the window because B leads today, is already significant, or looks like a winner.
Plain statistics
- Rule
- MDE, power, and baseline CVR must be explainable in business language.
- Allowed
- MDE is the smallest change you want to detect, power is the chance of seeing a real change, and baseline CVR is the original conversion level.
- Do not
- Do not run a formal test if the team cannot explain these terms plainly.
Clear counter-evidence
- Rule
- Before reading results, check whether another change caused the movement.
- Allowed
- Check traffic source, discount, inventory, payment issues, page speed, and tracking-definition changes first.
- Do not
- Do not attribute the result to the page variant when ads, campaign, stock, payment, or tracking changed.
Freeze versions
- Rule
- Page version, event definitions, audience, promotion, and shipping rules must be reviewable.
- Allowed
- Keep experiment ID, variant marker, theme/component version, GA4 events, Shopify order guardrails, and change log.
- Do not
- Do not change page, discount, or event definitions mid-run and then read it as a clean experiment.
Not every improvement deserves the label A/B test
Define A/B test, primary metric, guardrails, sample, window, and counter-signal first. Without these, uplift may just be noise.
Do not turn random noise into a win
- Must write
- Which evidence, which friction, and which step should change.
- Bad signal
- Only says "increase conversion rate."
- Action
- Send back to the friction evidence queue.
- Must write
- One primary metric and at most two guardrails, locked before launch.
- Bad signal
- Explains results with whichever metric rises.
- Action
- Define metrics first or do not test.
- Must write
- Baseline, expected sample, runtime, and readout timing.
- Bad signal
- Peeks daily and calls winners ad hoc.
- Action
- If sample is weak, downgrade to directional validation.
- Must write
- Prefer one change to isolate a mechanism; for a package comparison, predefine stable treatment arms and interpret only their overall effect.
- Bad signal
- Changes several elements but claims one element caused the effect, or keeps changing the variant mid-run.
- Action
- Split tests to isolate elements. A fixed package can use concurrent randomized A/B for the overall effect; a nonrandom before/after rollout remains observational.
- Must write
- What disproves the hypothesis and when to roll back.
- Bad signal
- Writes success but no failure boundary.
- Action
- Add rollback rules and learning notes.
Pro · Parts 3-22 · 20 parts
Members: full exercises and records
This part includes
- Check the gate before calling it a test
- Translate relative lift into counts
- Write the stop rule and counter-evidence
1 self-check question
Pro $20 / monthYearly $16 / month
Choose a plan, sign in with your email or Google account, and complete payment. You'll have instant access to all member benefits.
- Unlock every premium lesson
- Advanced tools and member resources
- Member-requested courses and ongoing updates
- Pro community and priority support
Checking your membership…
What's the difference between monthly and yearly?
Yearly plans save you 20% with annual payment. Monthly plans offer more flexibility for short-term needs.
FAQ
- How many days should an A/B test run before I read the result?
Do not judge by days alone. Check whether the sample covers a full business cycle: weekdays, weekend, ad budget rhythm, email sends, inventory, and shipping variation. Many small stores need at least one full cycle; if each version has only a few dozen orders, a 20% uplift in seven days may be only a few random orders.
- Can I stop a test early when uplift looks high?
Usually no. Early stopping amplifies random swings, especially when sample size is small, orders are few, or ads and campaigns are still changing. Stop early only when guardrails clearly break, such as payment failure, refunds, support complaints, AOV, or margin getting worse.
Experiment discipline must reconnect to evidence and review cadence
- Behavior evidence: turn recordings into testable hypotheses: When the experiment starts from a vague feeling, return to heatmaps, recordings, feedback, and funnel evidence first.