Measurement

How to run an A/B test on Shopify without crowning a false winner

Peeking at an A/B test ten times turns a 1% false alarm rate into 5%, Evan Miller showed. A fair test fixes its end date first, runs whole weeks and finishes before Black Friday.

SEA Survey Team · 5 min read

In short

  • Set the size and end date before the start, run whole weeks, and do not stop when B pulls ahead.
  • Read the range, not only the lift, and check guard numbers such as average order value before rollout.
  • Finish store tests before Black Friday, keep the winner live through Cyber Monday, and test emails and ads during the sale.
  • Shopify Rollouts tests a theme or checkout on the Grow plan or higher, while Klaviyo and Meta Ads Manager test emails and ads.

How to run a fair test

Evan Miller, whose free sample size calculator many teams use, showed that peeking at an A/B test ten times turns a 1% false alarm rate into 5%. Results swing a lot in the first days, so a store that checks every day and stops at the first good result crowns winners that are not real.

What is A/B testing? covers what a test is, how much traffic it needs and where a good idea comes from. Once the idea is written down, the run and the read decide whether the winner is real. A winner that goes live now stays there through Black Friday, and peak week is the wrong time to find a broken variant.

A fair test sets its size and end date first, from a sample size calculator, and writes the date down. It runs in whole weeks, because weekend shoppers are not weekday shoppers, so even if the calculator says 10 days, the test runs 14.

Nothing changes during the test: No price change, no new banner on one version, no switch in traffic share. If something must change, the test starts again.

The split needs a check too. A 50/50 test that ends up 55/45 has a problem, such as a redirect or a cached page, to fix before trusting the result. One test per page at a time keeps results clean, since two tests on the same page blur into each other.

How to read the result

At 95% confidence, a gap this large would show up less than 5% of the time if the change did nothing. It does not mean B has a 95% chance of being better.

The American Statistical Association puts it plainly: A p value does not measure the probability that the data were produced by random chance alone.

The range says more than the lift. A result like "B: +18%, likely between +2% and +34%" says B is probably better and that the size is uncertain, so plans go by the low end.

One test result drawn as a range from +2% to +34% on a scale from 0% to +40%, with the most likely lift of +18% marked in the middle and the low end of +2% marked as the number to plan around
B is probably better, but the real lift could be anywhere from +2% to +34%, so plan around the low end.

Guard numbers come next. A pop up that lifts email signups can lower orders, and a free shipping line that lifts orders can lower average order value. A win on the main number with a loss on a guard number needs a second look before rollout.

Slices count only where they were planned before the test, such as mobile and desktop. Cut the result into ten groups afterwards and one will show a "winner" by chance.

Novelty fades. A new look can draw clicks for a week from returning visitors who are only curious, so the main number gets a second check a month after rollout.

No winner is still a result. It tells you the change does not matter much to shoppers, so the simpler version stays and the next idea comes from your list of reasons.

A/B testing around Black Friday

Over the 2025 BFCM weekend, Shopify merchants sold a record $14.6 billion, 27% more than the year before. That week brings a different crowd, with more first time visitors, deal hunters and gift buyers, and a version that wins with them may lose in January.

A timeline from today to Black Friday, Cyber Monday and January. A store test that fits ends before the sale and its winner stays live through BFCM week. A store test that needs 4 weeks with 3 weeks left would run into the sale, so it is kept for January. Email and ad tests run during BFCM week
Store tests finish before the sale and keep the winner live through Cyber Monday, while emails and ads are tested during it.

A test that runs from mid November into December mixes two crowds and answers neither. A store test that cannot finish before the sale waits for January, so if the calculator says 4 weeks and Black Friday is 3 weeks away, the idea goes on the January list.

The winner of an earlier test stays live through Cyber Monday. Emails and ads carry the testing during the sale, because each send or campaign is its own test, finished within hours or days, before the crowd changes.

Tools for A/B testing on Shopify

Store, email and ad tests each run in their own tool. Shopify Rollouts sits in the admin under Markets, then Rollouts, where an Experiment compares a new theme, checkout or accounts setup against your current one, split 50/50 by default.

Theme tests in Rollouts report conversion rate, add to cart rate, reached checkout rate and bounce rate. Experiments run on the Grow plan or higher, and count only visitors who gave cookie consent.

Klaviyo's campaign A/B test can try subject lines, content or send times on part of your list, pick a winner by open rate, click rate or placed order rate, and send it to the rest. Click rate or placed order rate is the safer pick, because Apple Mail opens emails on its own and Klaviyo counts those opens in open rate tests.

Meta Ads Manager's A/B test compares ad images, text, audiences or placements, and nobody sees both versions. Meta recommends at least 7 days, and a test can run for up to 30.

Google Optimize is no longer an option: Google shut it down on September 30, 2023, so any guide that sends you there is out of date.

Your first test, step by step

Put together, the loop is short: Ask buyers, test one change, then ask again.

Three cards in a loop. Ask buyers, with two weeks of Thank you page responses and the doubt they name most. Test one change, with a set size and end date, no peeking and a read of the range and guard numbers. Ask again, to check that the doubt shows up less often, then back to the list of reasons
Ask buyers what almost stopped them, test the doubt they name most, then ask again to see if it went away.
  1. Add "What almost stopped you from buying today?" to the Thank you page with a post purchase survey app, and collect two weeks of responses.
  2. Pick the doubt buyers name most and write the idea down in one sentence.
  3. Work out the size and the end date, and if it is more than 6 weeks away, make the change bolder or test a busier page.
  4. Build version B with one change, check it on a phone, and run it to the end date without peeking for a winner.
  5. Read the main number, the range and the guard numbers, roll out or drop, and keep asking the question to see if the doubt fades.

SEA Survey  is a post purchase survey app for Shopify with a Free plan, and code BFCMSEA30 takes 30% off any paid plan for 3 months, until Dec 7. Added before Black Friday, the question turns your busiest week of buyers into a January test list.

SEA Survey

Listed as SEA Post Purchase Survey Quiz, 4.9 stars from 190 reviews, with the Built for Shopify badge.

30% off any paid plan for 3 monthsStarter $13.30 instead of $19, Growth $34.30 instead of $49

BFCMSEA30Last day: Dec 7

  • Why now: Black Friday week brings more buyers than any other. Ask them what almost stopped them, and you start January with a test list in their own words.
  • On the Free plan: 30 responses a month, with attribution, NPS and CSAT surveys.
  • What a paid plan adds: Starter counts 500 responses a month. Growth has unlimited responses, A/B testing for surveys and conditional logic, plus the Marketing attribution dashboard with revenue and LTV by the channel buyers named.
  • How to claim: Install SEA Survey from the Shopify App Store, open Plans, paste the code and pick Starter or Growth. Both start with a 7 day free trial.

FAQ

How long should an A/B test run?

Until it reaches the number of visits you worked out before the start, and for at least two full weeks. Stopping early because one version looks ahead is the most common way to pick a false winner.

What does statistical significance mean in an A/B test?

It means the gap between A and B would rarely appear if the change did nothing. At 95% confidence, that is less than 5% of the time, and it does not tell you how big the real lift is, so the range matters too.

Can a small Shopify store run A/B tests?

Yes, with bold changes on busy pages, or with emails and ads, where each send reaches many people at once. For small tweaks on a quiet page, asking buyers what held them back teaches you more in the same time.

Black Friday is Nov 27. Start collecting answers now.