Reading an A/B test: telling a real lift from noise

Data: both parts use synthetic data, generated locally by data/make_data.py (seeded, reproducible). Nothing is downloaded; no signup, no GPU. Synthetic is essential: to score whether a read-out tells real effects from noise you must know which experiments truly had an effect — which you only do when you set it.

Before you start

New to this? Two minutes of vocabulary, because the whole project turns on it.

An A/B test shows a control version to some users and a variant to others, and compares a metric — here, the conversion rate (fraction of users who convert).

Lift — how much better the variant did. Absolute lift is the difference in rates (0.118 − 0.101 = 0.017); relative lift expresses it as a percentage of control (~16%).
Statistical significance — is the difference more than random noise would produce? A two-proportion z-test turns the two counts into a p-value: the probability of seeing a gap this big if the variant were truly identical to control. Small p (< 0.05) → unlikely to be noise.
Confidence interval (CI) — the plausible range for the true lift, not just a single number. A 95% CI that excludes 0 agrees with “significant,” and its width is the honesty a point estimate hides.

A tiny example — flip two fair coins 10 times each; one will usually come up ahead. That doesn’t mean it’s a better coin. “Variant converted higher” is that coin flip. The test asks whether the gap is bigger than chance would give you.

What you’ll learn: to read out an experiment honestly (lift, significance, CI), to avoid the classic traps (significance ≠ importance; peeking), and to tell real effects from noise across many experiments.

New to Python? Section 4 (“Where to write and run code”) starts from zero.

1. The Brief

You’ll read out one A/B test end to end — the lift, a two-proportion z-test, and a 95% confidence interval — and conclude, honestly, that a +16.6% relative lift with p = 0.00003 is a real effect, while stating the interval rather than a bare number. Then, in Part B, you’ll face a backlog of 200 finished experiments, half of them pure noise, and classify which had a real effect — scoring 0.88 accuracy with a proper significance test where “the variant did better” manages only 0.68, because in a null experiment the variant wins about half the time by chance.

Reading an experiment is the most common quantitative decision a product or growth team makes, and it’s the one most often botched — shipping noise as a win, calling a trivial-but-significant lift important, or peeking until the p-value dips under 0.05. The skill isn’t the formula (it’s four lines); it’s the discipline around it: separate the effect size from its significance, report the interval, and remember that with enough tests, chance alone produces “winners.” This is what turns “the number went up” into a decision you can defend.

2. Difficulty & time

Difficulty 4 / 10. It sits above the difficulty-3 builds (#silent-row-loss, #idempotent-loads, #benford-fraud): it brings in statistical inference (a z-test, a confidence interval) and the judgment around it, and Part B is a genuine transfer from reading one test to classifying many under noise. It stays below the difficulty-5 group (#reorder-point-trap, #accuracy-is-lying, #hidden-communities): the core is one technique (the two-proportion test) and Part A’s path is fully specified. It’s distinct from #benford-fraud, the other data-analytics project (that screens digit distributions; this is experiment inference). Anchored to the rubric’s 3–4 band and those named ledger projects (G10).

Time is separate from difficulty: the scripts run in a fraction of a second (measured Part A wall-clock 0.33s). Part A is about 45 minutes; Part B is 2 hours of your own work.

3. What you’ll be able to do after

Read out an A/B test: absolute and relative lift, a two-proportion z-test with a p-value, and a 95% confidence interval — using only numpy and the standard library.
Explain why “significant” and “important” are different, and why a confidence interval beats a point estimate.
Recognise the peeking / multiple-comparisons trap: with many tests, chance produces apparent winners, and a raw “variant did better” rule flags a lot of noise.
Classify real vs null effects across many experiments and measure how well you did.

The finished result

By the end of Part A you’ll have read out one experiment the honest way — lift, significance, and the interval a point estimate hides:

control 0.1012  vs  variant 0.1179   ->  +16.6% relative lift
two-proportion z-test:  z = 4.15,  p = 0.00003   (significant)
95% confidence interval for the lift:  [+0.0089, +0.0246]   (excludes 0)

A real, positive effect — stated as a range, not a bare number. In Part B that same test tells 200 experiments apart at 0.88 accuracy, where the tempting “the variant won” rule manages only 0.68.

4. Prereqs & time box

You can write a function and use a pandas DataFrame. No statistics course is assumed — the one test (a two-proportion z-test) and the confidence interval are built up from scratch, and the normal distribution comes from the standard library (statistics.NormalDist), so there’s nothing to install beyond pandas.

Where to write and run code

Two ways to run this project — pick either:

In your browser (no install). If the project page shows a ▶ Run button, click it to run Part A right here — nothing to set up. Part A uses only pandas and Python’s built-in statistics, both of which run in the browser.
On your computer. The one-time setup — installing Python, downloading the code from the course site (no GitHub, no git), opening a terminal, creating a virtual environment, installing the requirements, and running Part A step by step — is covered once in Start Here. Come back here once python -c "import pandas" runs without error. Part B (writing your own solution) needs this local setup.

Time box

Part A — Guided Build: 30–60 minutes. Hard cap 90.
Part B — Your Turn: 1.5–2.5 hours. Hard cap 4.

5. Setup & data

Both datasets are synthetic, generated by one seeded command (byte-for-byte reproducible, git-ignored, never committed):

python data/make_data.py

That writes data/ab_experiment.csv (Part A — one experiment’s control/variant counts) and data/experiments.csv (Part B — 200 experiments). Which Part B experiments truly had an effect is documented in data/make_data.py and used by the grader — not handed to you.

6. Part A — Guided Build

Run the full script and read along:

python data/make_data.py            # once
python part_a/read_ab_test.py

Prefer the browser? If this page has a ▶ Run button, click it instead of the commands above — Part A runs in your browser with nothing to install. The STEP output and every checkpoint below are identical either way.

The full, commented script is in part_a/read_ab_test.py.

Step 1 — Rates and lift.

Checkpoint: control ≈ 0.1012, variant ≈ 0.1179; absolute lift +0.0168, relative lift +16.6%.

Step 2 — Two-proportion z-test.

Checkpoint: z ≈ 4.15, two-sided p ≈ 0.00003 — significant at α = 0.05. With 12k users per arm, a 1.7-point gap is far outside what noise would produce.

Step 3 — 95% confidence interval for the lift.

Checkpoint: CI ≈ [+0.0089, +0.0246] — excludes 0 (consistent with significant), and its width is the honest uncertainty a point estimate hides.

Step 4 — The read-out, and the traps. State the lift with its CI and p-value, and note the two ways this goes wrong: significance ≠ importance (huge n makes trivial lifts “significant”), and peeking (deciding the moment p < 0.05 inflates false positives).

Why this and not that

Why a significance test, not just “variant is higher”? Because a higher rate is what noise looks like half the time. The test asks the only question that matters — “is this bigger than chance would give me?” — and Part B shows that skipping it flags noise as wins.
Why report a confidence interval, not just the point lift? “+1.7 points” sounds exact; it isn’t. The CI [+0.9, +2.5] says the true lift could be modest or sizeable, and that range changes decisions (is it worth the engineering cost?). A point estimate hides the risk.
Why does significance ≠ importance? With millions of users, a 0.01-point lift can have p < 0.001 and be commercially meaningless. Always read the effect size and its CI alongside the p-value; a tiny significant lift is a real but possibly worthless effect.

Functions you’ll meet (and where to read more)

Function	What it does here	Docs
`pd.read_csv`	Load the experiment counts	read_csv
`statistics.NormalDist().cdf`	Turn a z-score into a p-value (no scipy)	NormalDist
Concept	Two-proportion z-test	Two-proportion z-test
Concept	Confidence intervals and A/B testing	A/B testing (Wikipedia)

If something looks off

My Part B accuracy is stuck around 0.68 — did I do something wrong? That’s the tell-tale of the raw “variant did better” rule, which counts a coin-flip win as a real effect. Add the significance test (flag only when p < 0.05 and the variant is higher) and it jumps to ~0.88.
A checkpoint number doesn’t match. The data is seeded, so the lift, z, p, and CI are exact. Re-run python data/make_data.py (you may have skipped or edited it), then re-run Part A.
Setup problems (Python not found, pip, FileNotFoundError, re-activating the venv): see the troubleshooting list in Start Here.

7. Part B — Your Turn

Part B is the optional “your turn” half — a guided fill-in-the-blank, not a blank page. You get a working starter (part_b/starter.py) that already handles the easy version; your job is to finish the # TODO it marks out. More open than Part A’s step-by-step, but you are never staring at an empty file. If you finished Part A and want to prove the skill, this is where it happens. New to Python? It’s completely fine to stop after Part A and come back later.

Different scale, the real discipline. data/experiments.csv holds 200 finished experiments (control and variant counts each). About half had a genuine effect; the rest were pure noise (the variant was identical to control). You don’t know which.

Your job: classify each experiment as real effect (1) or noise (0). Same core skill as Part A (the two-proportion test), now applied at scale where the raw “variant did better” rule is actively misleading.

Acceptance criteria (checkable)

Fill in part_b/starter.py, then:

python part_b/starter.py            # writes part_b/out/
pytest part_b/test_solution.py -q

If pytest reports everything as skipped, it just means the output files don’t exist yet — run python part_b/starter.py first (once your solution fills in the TODOs), then re-run.

Your solution writes into part_b/out/:

predictions.csv — experiment_id, predicted_effect (1 or 0).
report.json — experiments, flagged_real, accuracy, achieved_metric.

You’re done when the tests pass. The grader knows which experiments truly had an effect (from the seed) and scores your classification accuracy — it requires ≥ 0.80. The reference (a two-proportion z-test per experiment) scores 0.88; the raw “variant did better” rule scores only 0.68, because in a null experiment the variant wins about half the time.

Constraints — what must be true, not how to get there

Decide per experiment from its counts, not from the labels in make_data.py.
Use significance, not a raw rate comparison — require the gap to be unlikely under noise (and in the variant’s favour), as in Part A.
Your reported accuracy (if you fill it) must match your predictions (the grader recomputes it).

Hints

Hint 1 — reuse Part A

You already wrote the two-proportion z-test in Part A. Wrap it in a function and call it on each row of the 200 experiments.

Hint 2 — the decision rule

Flag a real effect when the test is significant (p < 0.05) AND the variant’s rate is above control’s. Requiring the direction avoids counting a significant drop as a “win.”

Hint 3 — why the raw rule fails (still not the code)

In a null experiment the variant beats control roughly half the time by chance, so “variant higher” alone is right on the real ones but wrong on ~half the nulls — about 0.68 accuracy. The z-test throws out those coin-flip wins, which is the whole point of significance.

8. Self-check

You don’t need the answer key. You’re done when:

You can state the Part A result in one sentence with its lift, CI, and p-value.
You can explain why “variant did better” is not enough, using the null-experiment coin flip.
Your Part B accuracy clears the floor, and it’s well above the raw-comparison ~0.68.
You can name the two traps (significance ≠ importance; peeking) and why they matter.

9. Stretch

Power and sample size. For a target lift, compute the sample size needed to detect it reliably, and show which of the Part B nulls-that-looked-real were simply underpowered.
Multiple comparisons. Run 200 experiments where NONE has an effect and count how many hit p < 0.05 anyway (~5%). Apply a correction (Bonferroni or Benjamini–Hochberg) and show it controls the false positives.
(Genuinely hard) Peeking, simulated. Simulate an experiment where you check the p-value every day and stop the moment it dips below 0.05. Show the false-positive rate balloons far above 5%, and that a fixed-horizon or sequential test fixes it.

10. Ship it

Put this in a portfolio repo and it counts. In that repo’s README:

Lead with the one-liner a PM gets: “‘The variant won’ is a coin flip half the time — here’s how a proper test told 200 real experiments from noise at 0.88 accuracy where the naive rule got 0.68.”
Show the Part A read-out (lift, CI, p) and the Part B accuracy vs the raw-comparison baseline.
State the discipline in one sentence: report the effect size and its interval, require significance before you call a win, and remember chance manufactures winners at scale.

Keep make_data.py and the tests in the repo so anyone can reproduce your numbers from zero.

11. Sources

Rendered from SOURCES.md.

Field	Value
Datasets	Synthetic — one A/B test (Part A); 200 experiments (Part B)
Data type	Synthetic (seeded, reproducible)
Generated by	`data/make_data.py` — `numpy.random.default_rng` (seeds 20260718 / 20260719)
Source / license	Not applicable — synthetic, generated in-repo; no third-party rights
Access / verification date	2026-07-18
Redistribution	Not applicable — generator committed, CSV output never committed
robots.txt / ToS	Not applicable — nothing is scraped or fetched

The data is synthetic and teaches the mechanic (reading out an experiment and telling signal from noise); conversions are drawn from known rates so the true effects are known, documented in data/make_data.py. It describes no real product.

Next up

Finished this one? Continue the Data Analytics track:

Maverick Spend: find the purchases that bypassed the contracts → · Browse all courses