Beat the Baseline: an honest test score, and the dumb model it has to beat

Before you start

New to this? Two words you need. A train/test split means you hide some of your rows before training: you fit the model on the rows it’s allowed to see (the training set) and then measure it on the rows you hid (the test set). Scoring a model on the very rows it learned from is like grading students on the exact questions you gave them as homework — of course they score well; it tells you nothing about a new question. A baseline is the dumbest possible “model”: for a yes/no outcome it just guesses whichever answer is more common; for a number it always predicts the average. A real model is only worth anything if it beats that. This project teaches the two habits that make any model number trustworthy: measure it on held-out data, and report it next to the baseline it beats. Miss either one and “92% accuracy” can be worthless.

1. The Brief

Someone hands you a model and says “it predicts which students pass — 80% accurate!” Two sentences later you should have two questions: scored on what data? and how good is 80%, really? You’ll build exactly this situation and answer both. You’ll see the “80%” evaporate to an honest ~74% the moment you score on held-out rows instead of the training rows — and then watch even that number shrink in meaning when you notice a brain-dead “everyone passes” baseline already scores 62%. This is the difference between a number that sounds good and a number that is good, and it is the first thing that separates someone who can use a model from someone who just ran one. The same trap sinks real projects: a fraud model reported at 99% accuracy is useless if 99% of transactions are legitimate and it just labels everything “fine”.

2. Difficulty & time

Difficulty 2/10. One idea (a trustworthy number = held-out + beats a baseline), a trivial model you don’t have to understand, and Part A’s path fully specified. It sits below feature-scaling (4) and review-sentiment (4) — both add a real modelling decision (a scaling pipeline; a text vectoriser and its vocabulary) on top of the evaluation — and below benford-fraud (3), which needs a statistical conformance test and its own screening logic. It is a touch above the very floor because it introduces scikit-learn’s fit/predict shape and two metrics; but the model itself is deliberately not the point, so the moving parts stay small. It’s a judgment skill — which number do I trust — not a syntax drill, which is what keeps it honest at a 2.

3. What you’ll be able to do after

Split data into train and test, fit on train, and explain in one sentence why scoring on the training rows gives a dishonest number.
Build the right dumb baseline for the task — majority-class for a yes/no outcome, mean for a number — and report a model’s score next to it, not alone.
Look at a reported accuracy (or R²) and say what’s missing before you’d trust it.
Recognise the same two habits under a different surface: a classification task and a regression task need the same discipline, different tools.

4. Prereqs & time box

Part A: ~30 minutes. Part B: ~2 hours. Both are hard caps. You need to read a CSV with pandas and call a function; you do not need to understand how logistic regression or linear regression work inside — they’re black boxes here on purpose, because the lesson is the evaluation around the model, not the model.

For where to write and run Python (install, virtual environment, per-OS terminal, running a script step by step), see Start Here guide. No account and no GPU — requires_signup is empty. Install the dependencies with pip install -r requirements.txt.

5. Setup & data

Both datasets are synthetic, generated locally by data/make_data.py from a fixed seed (byte-identical every run) — see Sources. Generate them once:

python data/make_data.py

That writes data/students_a.csv (Part A — a 350-row course roster) and data/cars_b.csv (Part B — a 150-row used-car inventory). Nothing is downloaded. Synthetic is the right call here: the whole point is to check an honest number against a known truth, and only generated data lets the grader know the truth in advance. Both files deliberately carry junk columns (commute time, gym hours, photo counts) that have nothing to do with the outcome — that junk is part of the lesson, not an accident (it’s what lets a model overfit and inflate the training score).

6. Part A — Guided Build

Run the guided script (or paste it cell-by-cell — each # %% is one step):

python part_a/train_and_score.py

It walks six steps and writes part_a/scoreboard_a.csv. The shape of the argument:

STEP 1. Load the 350-student roster. Checkpoint: 21 feature columns and a pass rate of about 0.620 — meaning “guess pass for everyone” is already right 62% of the time. That 0.620 is the bar.
STEP 2. The dishonest number: fit on all the rows, then score on those same rows. Checkpoint: about 0.794. This is the “~80%!” people quote. The model has already seen every row, so of course it looks good.
STEP 3. The honest number: split off 25% as a test set, fit on the other 75%, score on the held-out test rows. Checkpoint: about 0.739. The ~5-6 point drop from STEP 2 is the size of the lie.
STEP 4. The baseline: DummyClassifier(strategy="most_frequent") — always guess the majority class. Checkpoint: about 0.625 (the test-set pass rate).
STEP 5. The scoreboard, side by side: 0.794 (never report), 0.739 (report this), 0.625 (…next to this). The only defensible claim is “73.9% on held-out data, beating a 62.5% baseline” — an 11-point real lift.
STEP 6. Save scoreboard_a.csv.

Why this and not that

Why not just report the 0.794? Because it was measured on the training rows. It answers “how well does the model remember data it has seen?” — a question nobody cares about. The only useful question is how it does on students it has not seen, and that’s the test set.
Why compare to a baseline at all — isn’t 73.9% obviously good? Not until you know the floor. Here the floor is 62.5%, so the model bought you ~11 points. If the pass rate had been 95%, that same 73.9% would be a disaster — worse than guessing “pass” every time. A score only has meaning next to its baseline.
Why is the model itself a black box here? Because swapping LogisticRegression for any other classifier would not change a single lesson on this page. The evaluation is the skill; the model is a detail.

Functions you’ll meet (and where to read more)

Function	What it does here
`pandas.read_csv`	Load the roster into a DataFrame (STEP 1)
`train_test_split`	Hold out 25% of rows as an unseen test set (STEP 3)
`LogisticRegression`	The (black-box) classifier we fit and score (STEP 2–3)
`DummyClassifier`	The majority-class baseline to beat (STEP 4)
`accuracy_score`	Fraction of predictions that are correct

For the bigger picture, scikit-learn’s own common pitfalls page opens with exactly this mistake.

If something looks off

If your STEP 3 (test) number is higher than STEP 2 (train), re-run python data/make_data.py — you may be on stale or hand-edited data. On the shipped data the train number is the higher one.
If the baseline in STEP 4 isn’t ~0.625, check you fit the DummyClassifier on the training data and scored it on the test data, like the real model.
A model that barely beats the baseline is not a bug — it’s the honest result, and noticing it is the whole point of this project.

7. Part B — Your Turn

Part B is the optional “your turn” half — a guided fill-in-the-blank, not a blank page. You get a working starter (part_b/starter.py) that already sets up the split and the baseline; your job is to finish the one # TODO. More open than Part A’s step-by-step, but you are never staring at an empty file. If you finished Part A and want to prove the skill, this is where it happens. New to this? It’s completely fine to stop after Part A and come back later.

Same discipline, new task. cars_b.csv is a used-car dealer’s inventory; you predict each car’s price — a number, not a yes/no. That one change moves everything: you fit a regressor, you score with R² and mean absolute error instead of accuracy, and the dumb baseline is no longer “guess the majority class” — for a number it’s “always predict the average price.” The starter does exactly that lazy thing (it predicts the mean, so it is the baseline) and reports its numbers. Your # TODO: fit a real model on the training cars and score it on the held-out test cars, so your error drops well below the baseline’s.

Acceptance criteria (checkable)

Your part_b/starter.py writes part_b/out/report.json with:

test_r2 — your model’s R² on the held-out test cars,
test_mae — your model’s mean absolute error (in dollars) on the test cars,
baseline_mae — the mean absolute error of always predicting the average training price.

Use the split the starter already sets up — train_test_split(..., test_size=0.25, random_state=42) — so your held-out number is reproducible and gradeable. Then:

python part_b/starter.py
pytest part_b/test_solution.py -q

You’re done when the tests pass: baseline_mae is correct, your test_mae is comfortably below it (a real model roughly a third of the baseline error), and your honest test_r2 matches the held-out truth (about 0.91; the reference achieves 0.9094 and the published bar is 0.50). The mean-predictor starter scores R² ≈ 0 and ties the baseline — it fails.

Constraints — what must be true of your solution, not how to get there

Fit the model on the training cars only, and report test_r2 / test_mae on the held-out test cars — never the rows you trained on.
The baseline must be the mean predictor (average training price for every test car), not a majority class — the outcome is a number.
Beat the baseline honestly: a lower test_mae because the model learned real structure, not because you scored it on the training rows.

Hints

Hint 1 — what changes from Part A

The outcome is a number, so a classifier and accuracy_score don’t apply. You want a regressor and regression metrics. The evaluation habit is identical to Part A: fit on train, score on the held-out test, compare to a baseline.

Hint 2 — the real model is one line

from sklearn.linear_model import LinearRegression, then model = LinearRegression().fit(X_train, y_train) and test_pred = model.predict(X_test). Replace the starter’s “predict the mean” line with those.

Hint 3 — the baseline and the metrics

The baseline is already computed for you (baseline_mae). Score your model with r2_score(y_test, test_pred) and mean_absolute_error(y_test, test_pred). Your MAE should land near $1,500 against a baseline near $5,100 — that gap is the point.

8. Self-check

Your honest test accuracy (Part A) and test R² (Part B) should both be lower than the number you’d get scoring on the training rows — if a “test” score is higher, you’re probably scoring on data the model already saw.
Your model should clearly beat its baseline (accuracy above the majority rate; MAE well below the mean-predictor MAE). If it only ties the baseline, it has learned nothing usable.
Re-running python data/make_data.py never changes the data (it’s seeded), so your numbers shouldn’t move between runs.
If you can state, in one sentence, why a training-set score lies and why a score means nothing without its baseline, you have the transferable skill — the specific numbers are secondary.

9. Stretch

In Part A, swap LogisticRegression for a RandomForestClassifier (default settings) and compare the train-scored and honest-test numbers. The gap between them (the overfitting) is usually larger — a vivid reminder that the flashier the model, the more the training-set score can lie.
In Part B, add a second baseline: the DummyRegressor with strategy="median". Does predicting the median beat predicting the mean here, and why might it on skewed prices?
(Genuinely hard.) A single train/test split is itself a noisy estimate — change random_state in Part A and watch the honest number bounce by several points. Replace the one split with 5-fold cross-validation (cross_val_score), report the mean and spread, and say which single-split numbers on this page you’d now quote with a range instead of a point.

10. Ship it

Put this in a repo README so it counts as a portfolio piece: the three-number scoreboard from Part A (train-scored vs honest-test vs baseline) with one line on why only two of them are worth reporting, and the Part B result (a regression model’s held-out MAE next to the mean-baseline MAE). The sentence an interviewer is listening for: “I never trust a model score unless it’s on held-out data and it beats the right dumb baseline.” Showing you applied the same rule to a classification task and a regression task — with the correct, different baseline for each — is what proves it’s a habit, not a memorised recipe.

11. Sources

See SOURCES.md. Both datasets are synthetic, generated by data/make_data.py from a fixed seed; there is no external source, and nothing is downloaded or committed.

▶ Run Part A in your browser

▶ Try it now — run the real Part A right here in your browser. This one button makes the data and runs Part A for you — you do not need to install anything, and you do not need to type any of the python … commands anywhere on this page (those are only for running on your own computer instead). First run loads Python + this project’s libraries: usually ~10–20 s, a little longer for machine-learning projects.

Next up

Finished this one? Continue the AI & Machine Learning track:

Feature scaling: why one big-numbered column can wreck a distance-based model → · Browse all courses