A/B Testing, step by step
The discipline is in the design, not the analysis.
- State one hypothesisWhat change, on what metric, by how much.
- Choose the primary metricOne. Guardrail metrics may be watched, not optimised.
- Calculate the sample sizeFrom the baseline rate and the effect you care about detecting.
- Randomise properlyBy user, not by session. Assign once and keep it.
- Run the full durationAt least one full weekly cycle. Do not peek and stop.
- Analyse onceAgainst the pre-declared metric.
Common errors
- Stopping when it looks significant
- Testing many metrics and reporting the winner
- Too small a sample to detect the effect
- Ignoring novelty effects
Peeking and stopping when p < 0.05 raises the real false-positive rate far above 5%. Fix the duration in advance.
Questions about the A/B Testing poster
- What’s on the A/B Testing poster?
- 6 numbered steps, with a full worked run-through. The discipline is in the design, not the analysis. State one hypothesis — What change, on what metric, by how much.; Choose the primary metric — One. Guardrail metrics may be watched, not optimised.; Calculate the sample size — From the baseline rate and the effect you care about dete…; Randomise properly — By user, not by session. Assign once and keep it.; Run the full duration — At least one full weekly cycle. Do not peek and stop.; Analyse once — Against the pre-declared metric.. Peeking and stopping when p < 0.05 raises the real false-positive rate far above 5%. Fix the duration in advance.
- Who is the A/B Testing poster for?
- A/B Testing belongs to the Business & Management section rather than to a school year, because business is not something one grade owns. Anyone learning analytics can pin it up — a beginner, a student mid-course, or someone revising years later.
- When should you use the A/B Testing poster?
- Fix the sample size and the duration before starting. Stopping early inflates false positives badly. A wall chart earns its place by being glanceable from where the work is happening, so A/B Testing belongs on the wall where that business work actually happens, within glancing distance, rather than filed away.
- What other posters go with A/B Testing?
- Cohort Analysis, Analysis Traps and Reading a Relationship sit alongside A/B Testing in the Business & Management section. Printed together they make a wall rather than a single sheet, which is how a reference set actually gets used.Cohort AnalysisAnalysis TrapsReading a Relationship
- Is the A/B Testing poster free to download and print?
- Yes. A/B Testing downloads as a free PDF with no account, no email and no watermark, like everything else in the Business & Management section. Print as many copies as you like for a home, a classroom, a library or a tutoring group; reselling the file is the only thing the licence rules out.Read the licence
- What size does the A/B Testing poster print at?
- A/B Testing is a vector PDF laid out for US Letter, and prints on A4 with Fit to page — the same file, no separate download. Because every mark on it is drawn rather than photographed, it stays sharp enlarged to A3, A2 or A1 at a copy shop. Colour carries emphasis only, so a greyscale print of A/B Testing loses nothing.Printing guide
Related posters
Charts that sit alongside A/B Testing on the same wall.
Browse every Business & Management poster, or start from the full catalogue.