Outlier Spotting: one bad reading wrecks the mean, not the median

Before you start

New to this? The mean is the everyday average: add everything up, divide by how many. The median is the middle value when you sort them — half are below it, half above. They usually land close together, but one freak value can pull them wildly apart. Say five houses on a street are worth £200k, £210k, £220k, £230k, and one is a £5,000,000 mansion. The mean house is £1.17M — a figure no house on the street is near. The median is £220k — the actual middle house. The mansion dragged the mean; it barely touched the median. This project teaches you to spot when that has happened, report the number you can defend, and flag the odd value out with a simple rule — whether it’s a broken sensor or a real millionaire.

1. The Brief

A cold-storage freezer logs its temperature every 15 minutes. At the end of the day you average the 96 readings to answer one question — how cold was the aisle? — and pandas tells you the freezer ran at +86 °C. It didn’t: one reading is 9999, the datalogger’s code for a probe that briefly lost contact, and that single non-temperature is heavy enough to drag the mean of 96 readings up by more than a hundred degrees. This is the outlier trap, and it shows up far past sensors: average salary, average delivery time, average basket size — any mean quoted over data with one fat-finger or one genuine extreme is a number waiting to mislead. You’ll see the mean break, see the median hold, find the bad point with a rule you can reuse anywhere, and report a central value you can stand behind.

2. Difficulty & time

Difficulty 2/10. One main tool (pandas), one file, one concept — mean-vs-median under an extreme value — with Part A’s path fully specified. It sits below benford-fraud (3), silent-row-loss (3), and cohort-retention (3): each of those has more moving parts (a digit-distribution test, a join to validate, a cohort matrix to pivot), whereas this turns on a single idea — a robust summary resists one bad value — applied cleanly. It is a peer of averaging-averages (2), another “the obvious average is the wrong one” judgment skill, and sits above missing-not-random (1), which needs no rule beyond noticing a gap. What keeps it off the floor of the scale is the judgment call in Part B: an outlier can be an error to delete or a real value to keep, and telling them apart is the actual skill.

3. What you’ll be able to do after

Look at a mean and a median that disagree and say which one an extreme value has corrupted, and in which direction.
Flag suspect points with the IQR-fence rule (Q1 - 1.5·IQR, Q3 + 1.5·IQR) and read what it’s telling you.
Decide, with a stated reason, whether a flagged outlier should be removed (a physically impossible sensor error) or kept but not allowed to drive the headline (a genuine extreme).
Report a defensible central value — the median, or a mean trimmed of a justified error — and say why you trust it.

4. Prereqs & time box

Part A: ~30 minutes. Part B: ~1.5 hours. Both are hard caps — if Part A runs long, you’re overthinking it. You need to read a CSV with pandas and call a couple of column methods; nothing more.

For where to write and run Python (install, virtual environment, per-OS terminal, running a script step by step), see Start Here guide. No account and no GPU — requires_signup is empty.

5. Setup & data

The data is synthetic, generated locally by data/make_data.py from a fixed seed (byte-identical every run) — see Sources. Generate it once:

python data/make_data.py

That writes data/readings_a.csv (Part A — a freezer probe’s day of readings, one of them the 9999 fault code) and data/incomes_b.csv (Part B — a household-income survey with a few real high earners). Nothing is downloaded; there is no external source. Synthetic is the right call here: the lesson is whether your robust summary recovers the true central value, and only generated data lets you know that truth in advance — and it lets us inject a broken-sensor value and a millionaire without touching anyone’s real data.

6. Part A — Guided Build

Prefer the browser? If this project shows a ▶ Run button, you can run Part A right on the page — no setup. The commands below are for running on your own computer.

▶ Try it now — run the real Part A right here in your browser. This one button makes the data and runs Part A for you — you do not need to install anything, and you do not need to type any of the python … commands anywhere on this page (those are only for running on your own computer instead). First run loads Python + this project’s libraries: usually ~10–20 s, a little longer for machine-learning projects.

Run the guided script (or paste it cell-by-cell — each # %% is one step):

python part_a/analyze_readings.py

It walks six steps and writes part_a/readings_flagged_a.csv. The shape of the argument:

STEP 1. Load the 96 readings (one every 15 min for 24 h). Checkpoint: 96 rows, columns timestamp and temp_c.
STEP 2. Take the naive answer — the mean of every reading. Checkpoint: about +86.26 °C. A freezer at +86 °C is impossible; that absurdity is the tell.
STEP 3. Take the robust answer — the median. Checkpoint: about -18.0 °C — the number that matches the freezer you can walk up to and touch.
STEP 4. Find the culprit with the IQR fences. Checkpoint: Q1≈-18.50, Q3≈-17.70, IQR≈0.80, fences ≈ [-19.70, -16.50]; exactly one reading flagged — the 9999. A physical sanity check (a freezer lives roughly -30 to +5 °C) points at the same value.
STEP 5. Report a defensible central value: the median (-18.0), or the mean after dropping the flagged point (-18.08). They agree to within half a degree; an assert proves it.
STEP 6. Save readings_flagged_a.csv with an is_outlier column you can eyeball later.

Why this and not that

Why not just df['temp_c'].mean()? Because the mean gives every reading equal pull, so one value of 9999 outweighs ninety-five real ones. The median asks a different, sturdier question — “what’s the middle reading?” — which one bad point can’t move to +86.
Why not delete every large value? Because “large” isn’t “wrong.” Delete a point when you can justify it (an impossible value equal to a known fault code), not because it’s inconvenient. The median needs no deletion at all — it was robust from the start, which is why it’s the safer default when you’re unsure.
Why the IQR fences and not “±3 standard deviations”? The standard deviation is itself wrecked by the outlier (the 9999 inflates it enormously), so a fence built from it can miss the very point it should catch. Quartiles are robust, so fences built from them aren’t fooled.

Functions you’ll meet (and where to read more)

Function	What it does here
`pandas.read_csv`	Load `readings_a.csv` into a DataFrame (STEP 1)
`Series.mean`	The naive average that the 9999 wrecks (STEP 2)
`Series.median`	The robust middle value (STEP 3)
`Series.quantile`	Q1 and Q3 for the IQR fences (STEP 4)

If something looks off

If your mean looks sane (near -18), your data didn’t load the 9999 row — check the row count is 96 and re-run python data/make_data.py.
If the IQR fences flag many readings, you may have built them from the mean/standard deviation instead of the quartiles — the standard deviation is itself corrupted by the outlier.
A +86 °C freezer is the point of this project, not a bug — read your numbers against the checkpoints above before assuming you broke something.

7. Part B — Your Turn

Part B is the optional “your turn” half — a guided fill-in-the-blank, not a blank page. You get a working starter (part_b/starter.py) that already handles the easy version; your job is to finish the # TODOs it marks out. More open than Part A’s step-by-step, but you are never staring at an empty file. If you finished Part A and want to prove the skill, this is where it happens. New to Python? It’s completely fine to stop after Part A and come back later.

Same skill, new setting — and a twist. A ward survey’s incomes_b.csv gives ~200 households’ annual income. A handful of genuine high earners (a business owner, a pro athlete) sit far above everyone else. The starter prints the naive answer — the mean as the “typical” income, and zero households flagged — which is wrong twice over: the mean is inflated by those few, and nobody is flagged. The # TODOs are the Part A moves, transferred: report the median as the typical figure, and flag the extremes with IQR fences. But note the judgment shift — unlike Part A’s 9999, these incomes are real. You do not delete them; you report a robust figure and count how many are outliers.

Acceptance criteria (checkable)

Your part_b/starter.py writes part_b/out/report.json with:

typical_income — the robust “typical” household income (the median, whole pounds),
n_outliers — how many households fall outside the IQR fences.

Then:

python part_b/starter.py
pytest part_b/test_solution.py -q

You’re done when the tests pass: typical_income lands within 0.5% of the true median (the reference hits it exactly) and n_outliers is exact. The naive mean is about 51% too high as a “typical” figure, and flagging nothing fails the outlier count.

Constraints — what must be true of your solution, not how to get there

The typical figure must be robust to the high earners — a median, or a defensible trimmed mean, not the plain mean.
Do not delete the high earners. They are real households; count them as outliers, keep them in the data.
Use the same fence rule as Part A (Q1 - 1.5·IQR, Q3 + 1.5·IQR) so the outlier count is reproducible, not eyeballed.

Hints

Hint 1 — the typical figure

The mean is dragged up by a few very high incomes. The median — the middle household when you sort — is not. Swap income.mean() for income.median() for typical_income.

Hint 2 — flagging with IQR fences

Same rule as Part A STEP 4: q1, q3 = income.quantile(0.25), income.quantile(0.75), iqr = q3 - q1, then fences at q1 - 1.5*iqr and q3 + 1.5*iqr. Count the incomes outside them.

Hint 3 — the shape of the answer

n_outliers = int(((income < low_fence) | (income > high_fence)).sum()). It should come out to a handful — the genuine high earners plus a few in the long upper tail — not zero and not half the survey.

8. Self-check

Your typical_income should be far below the mean (the high earners inflate the mean, not the median) and should look like an ordinary household — sanity-check it against the bulk of the survey, not the top of it.
Re-running python data/make_data.py never changes the data (it’s seeded), so your numbers shouldn’t move between runs.
If you can state, in one sentence, when an outlier should be deleted and when it should be kept but flagged, you’ve got the transferable judgment — the specific numbers are secondary.

9. Stretch

Add a trimmed mean to Part B: drop the top and bottom 5% of incomes and average the rest. Compare it to the median — how close do the two robust summaries land, and which would you quote?
Report where the fences fell and which households they caught in Part B, not just the count — a flagged list is far more useful to a survey team than a bare number.
(Genuinely hard.) The IQR fence is symmetric, but income is right-skewed — the upper tail is long even with no errors, so the fence flags ordinary well-off households as “outliers.” Work out why a symmetric rule over-flags skewed data, then try flagging on log(income) instead and explain what changes and why that’s a more honest question to ask.

10. Ship it

Put this in a repo README so it counts as a portfolio piece: one sentence on the trap (“one 9999 sensor fault turned a -18 °C freezer into a reported +86 °C”), the mean and median side by side, and the one-line rule (a robust summary resists a single extreme; flag it with IQR fences). Add a short note on Part B showing you transferred the same idea to a different domain (incomes, not temperatures) and made the harder call — that the extremes there are real and must be kept, not deleted. That judgment — same skill, opposite decision about the outlier — is what an interviewer is actually checking for.

11. Sources

See SOURCES.md. The data is synthetic, generated by data/make_data.py from a fixed seed; there is no external source, and nothing is downloaded or committed.

Next up

Finished this one? Continue the Data Analytics track:

Percent-Change Traps: why +20% then -20% doesn’t get you back → · Browse all courses