Data contracts: catch the bad rows before they reach your table

Data: both parts use synthetic data, generated locally by data/make_data.py (seeded, reproducible). Nothing is downloaded; no signup, no GPU. Synthetic is essential: to score a validator you must know exactly which rows are bad, which you only do for data whose defects you injected yourself.

Before you start

New to this? Two minutes of vocabulary, because the whole project turns on it.

Data arrives from somewhere upstream — a nightly export, an API feed — and it is not always clean. If you load it blindly, one bad row corrupts every count and join downstream, and you find out days later when a report looks wrong.

Data contract — the set of rules an incoming batch must satisfy to be allowed in: which columns exist, their types, allowed value ranges, which fields can’t be null, which must be unique, and how fields must relate to each other.
Validation — checking each rule and finding the rows that break it.
Quarantine — instead of dropping bad rows silently or letting them through, you set them aside (with the reason they failed) so a human can look, while the clean rows load.
Cross-field rule — a rule about how columns relate, e.g. line_total must equal qty × unit_price. A row can have every field individually valid and still be wrong.

A tiny example — an order row with amount = -5 or status = "returned" (not an allowed status) breaks the contract; so does a payment where qty = 3, unit_price = 10, but line_total = 40. A contract turns “the data looks fine” into a checkable yes/no per row.

What you’ll learn: to express a batch’s rules as code, to find and quarantine the rows that break them (with reasons), and to catch the cross-field errors that single-column checks miss.

New to Python? Section 4 (“Where to write and run code”) starts from zero.

1. The Brief

You’ll take a nightly orders export riddled with the usual upstream defects — missing and duplicated ids, negative amounts, statuses that aren’t allowed, unparseable dates — and write a contract that finds all 68 bad rows out of 1,000, quarantines them with the reason each failed, and lets the clean 932 through. Then, in Part B, you’ll validate a payments batch whose contract includes a cross-field rule (line_total = qty × unit_price), catching rows that every single-column check would wave through — scoring an F1 of 1.0 against the hidden truth.

Every data pipeline that’s survived contact with reality has a validation layer, because “trust the upstream” is how tables quietly rot. The skill isn’t just writing if checks — it’s thinking in terms of a contract (what must be true of every row), separating “reject loudly with a reason” from “drop silently,” and remembering that the nastiest errors are the cross-field ones where each column looks fine on its own. A validator that says only “bad” without why just moves the debugging downstream; a good one is the difference between a pipeline you trust and one you babysit.

2. Difficulty & time

Difficulty 3 / 10. It’s about level with the other single-mechanic builds #silent-row-loss, #idempotent-loads, and #benford-fraud (all difficulty 3): one clear idea (a contract is a set of per-row rules), one tool (pandas), and a fully specified Part A — with a genuine step up in Part B, the cross-field rule that single-column checks can’t catch. It’s below the difficulty-5 group (#reorder-point-trap, #accuracy-is-lying, #hidden-communities), which chain a simulation or a modelling pipeline; here the reasoning is direct. It’s a different job from #idempotent-loads (that one is about loads that don’t duplicate on re-run; this is about rejecting rows that break the rules). Anchored to the rubric’s level-3 band and those named ledger projects (G10).

Time is separate from difficulty: the script runs in under a second (measured Part A wall-clock 0.35s). Part A is about 45 minutes; Part B is 2 hours of your own work.

3. What you’ll be able to do after

Write a data contract as code: null, type, range, allowed-value, uniqueness, and cross-field checks over a batch.
Split a batch into clean and quarantined rows, and attach to each quarantined row the reason(s) it failed.
Catch cross-field violations (rows valid column-by-column but inconsistent across columns) that single-column checks miss.
Score a validator honestly (precision/recall/F1) against known-bad rows.

The finished result

By the end of Part A you’ll have split a dirty 1,000-row export into what loads and what gets held back — each held-back row tagged with why:

1,000 rows received
  ->  932 clean rows load
  ->   68 quarantined  (null id 12, duplicate id 20, negative amount 15, bad status 13, bad date 9;
                        some rows break more than one rule, so the total is 68, not the sum)

Every quarantined row carries its reason — the difference between a pile someone can fix and a mystery. That’s what a contract buys you.

4. Prereqs & time box

You can write a function and use a pandas DataFrame (filtering rows by a condition). No data-engineering background is assumed; the checks are ordinary pandas expressions, explained as they appear.

Where to write and run code

Two ways to run this project — pick either:

In your browser (no install). If the project page shows a ▶ Run button, click it to run Part A right here — nothing to set up. Part A uses only pandas, which runs in the browser.
On your computer. The one-time setup — installing Python, downloading the code from the course site (no GitHub, no git), opening a terminal, creating a virtual environment, installing the requirements, and running Part A step by step — is covered once in Start Here. Come back here once python -c "import pandas" runs without error. Part B (writing your own solution) needs this local setup.

Time box

Part A — Guided Build: 30–60 minutes. Hard cap 90.
Part B — Your Turn: 1.5–2.5 hours. Hard cap 4.

5. Setup & data

Both datasets are synthetic, generated by one seeded command (byte-for-byte reproducible, git-ignored, never committed):

python data/make_data.py

That writes data/orders_batch.csv (Part A — a dirty orders export) and data/payments_batch.csv (Part B — a payments export with a cross-field defect). Which rows are bad is defined by the contract and documented in data/make_data.py; Part B’s labels are used by the grader, not handed to you.

6. Part A — Guided Build

Run the full script and read along:

python data/make_data.py            # once
python part_a/validate_orders.py

Prefer the browser? If this page has a ▶ Run button, click it instead of the commands above — Part A runs in your browser with nothing to install. The STEP output and every checkpoint below are identical either way.

▶ Try it now — run the real Part A right here in your browser. This one button makes the data and runs Part A for you — you do not need to install anything, and you do not need to type any of the python … commands anywhere on this page (those are only for running on your own computer instead). First run loads Python + this project’s libraries: usually ~10–20 s, a little longer for machine-learning projects.

The full, commented script is in part_a/validate_orders.py. The orders contract: order_id present and unique; amount ≥ 0; status in {PENDING, SHIPPED, DELIVERED, CANCELLED}; order_date a parseable YYYY-MM-DD.

Step 1 — Load the batch (1,000 rows).

Step 2 — Check each rule — each returns the rows that fail it.

Checkpoint: 12 null id, 20 duplicated id (both rows of each duplicate pair), 15 negative amount, 13 disallowed status, 9 unparseable date.

Step 3 — A row is bad if it fails ANY rule; union them.

Checkpoint: 68 bad rows (fewer than the sum above — some rows break more than one rule) and 932 clean. Only the 932 should load.

Step 4 — Write the split, attaching to each quarantined row the reason(s) it failed.

Checkpoint: clean_orders.csv has 932 rows; quarantined_orders.csv has 68, each tagged with why. The reason is what makes a quarantine actionable instead of a mystery.

Why this and not that

Why quarantine, not drop? Silently dropping bad rows makes your totals wrong in a way nobody can see — the data just quietly shrinks. Quarantining keeps the count honest (clean + quarantined = received) and gives someone a pile to fix. “Reject loudly” beats “drop quietly” every time.
Why record the reason per row? “Row 417 is bad” sends someone hunting; “row 417: status not allowed” is a fix. A validator that only emits a count has moved the debugging downstream to whoever’s on call.
Why check uniqueness on both members of a duplicate pair? When an id appears twice you usually can’t tell which is the real one, so both are suspect — flag both and let a human decide, rather than silently keeping the first.

Functions you’ll meet (and where to read more)

Function	What it does here	Docs
`Series.isna`	Find null ids / unparseable values	isna
`Series.duplicated(keep=False)`	Flag every row of a duplicate-key pair	duplicated
`Series.isin`	Check a value is in the allowed set	isin
`pd.to_datetime(..., errors="coerce")`	Turn unparseable dates into NaT you can detect	to_datetime
`pd.to_numeric(..., errors="coerce")`	Range-check a column safely even if it has junk	to_numeric

If something looks off

The per-rule counts add up to more than 68 — is that a bug? No. A single row can break more than one rule (say, a bad status and an unparseable date), so the union of bad rows (68) is smaller than the sum of the per-rule counts. That overlap is expected, not double-counting.
A checkpoint number doesn’t match. The data is seeded, so the counts are exact. Re-run python data/make_data.py (you may have skipped or edited it), then re-run Part A.
Setup problems (Python not found, pip, FileNotFoundError, re-activating the venv): see the troubleshooting list in Start Here.

7. Part B — Your Turn

Part B is the optional “your turn” half — a guided fill-in-the-blank, not a blank page. You get a working starter (part_b/starter.py) that already handles the easy version; your job is to finish the # TODO it marks out. More open than Part A’s step-by-step, but you are never staring at an empty file. If you finished Part A and want to prove the skill, this is where it happens. New to Python? It’s completely fine to stop after Part A and come back later.

Different batch, a rule single-column checks can’t handle. data/payments_batch.csv has a contract: payment_id present and unique; qty > 0; method in {card, ach, wire, paypal}; and — the new one — line_total must equal qty × unit_price.

Your job: flag every row that breaks the contract. Same core skill as Part A (validate against a contract), now including a cross-field rule where a row can be valid in every column yet still be wrong.

Acceptance criteria (checkable)

Fill in part_b/starter.py (the single-column rules are started for you; the cross-field one is the TODO), then:

python part_b/starter.py            # writes part_b/out/
pytest part_b/test_solution.py -q

If pytest reports everything as skipped, it just means the output files don’t exist yet — run python part_b/starter.py first (once your solution fills in the TODOs), then re-run.

Your solution writes into part_b/out/:

flagged.csv — row (the indices you flag as breaking the contract).
report.json — rows, flagged, f1, achieved_metric.

You’re done when the tests pass. The grader knows the true bad rows (recovered from the seed) and scores your flags with F1 ≥ 0.90. The reference catches every rule type and scores 1.00; a validator that implements the single-column rules but skips the cross-field one lands at ~0.89 and fails — those inconsistent-total rows are exactly what it misses.

Constraints — what must be true, not how to get there

Flag from the rules, not from the labels in make_data.py.
Include the cross-field rule — line_total == qty × unit_price (allow a cent of rounding slack). It’s the difference between passing and failing.
Your reported f1 (if you fill it) must match your flags (the grader recomputes it).

Hints

Hint 1 — the single-column rules

Null/unique on payment_id, qty > 0, and method in {...} are the same shapes as Part A (isna, duplicated(keep=False), a numeric comparison, isin). The starter already has these.

Hint 2 — the cross-field rule

Compute qty * unit_price, round to the cent, and compare to line_total. Flag rows where they differ by more than ~0.01 (rounding slack). This can’t be done one column at a time — it needs two columns together.

Hint 3 — why the floor is set where it is (still not the code)

The inconsistent-total rows are a big chunk of the bad rows. Skip that rule and your recall drops enough to fall under the 0.90 F1 floor even if every other check is perfect — which is the whole point: the cross-field errors are the ones that quietly get through.

8. Self-check

You don’t need the answer key. You’re done when:

Your Part A split adds up: clean + quarantined = rows received, and each quarantined row has a reason.
Your Part B F1 clears the floor, and you can point to the rule that mattered most (cross-field).
You can explain why quarantining beats silently dropping, in one sentence.
Re-running reproduces your numbers (fixed seed).

9. Stretch

Column-level report. Summarise violations by rule and by column (“3% of rows failed the status rule”), the kind of dashboard a data team watches over time.
Contract as config. Move the rules out of code into a small schema (a dict or YAML: column → type, range, allowed values) and write a generic validator that reads it — closer to how tools like pandera or Great Expectations work.
(Genuinely hard) Distributional checks. Add rules that aren’t per-row: today’s batch has a null-rate or a mean amount wildly different from the last 30 days (“freshness” / “drift” checks). Decide how to flag a whole batch as suspect, not just rows.

10. Ship it

Put this in a portfolio repo and it counts. In that repo’s README:

Lead with the one-liner a data team gets: “One nightly export had 68 bad rows in 1,000 — here’s the contract that caught them (with reasons) before they corrupted the table.”
Show the violations-by-rule breakdown and a couple of quarantined rows with their reasons, including a cross-field failure.
State the principle in one sentence: validate against a contract, quarantine with a reason, and never trust single-column checks to catch cross-field errors.

Keep make_data.py and the tests in the repo so anyone can reproduce your numbers from zero.

11. Sources

Rendered from SOURCES.md.

Field	Value
Datasets	Synthetic — orders batch (Part A); payments batch (Part B)
Data type	Synthetic (seeded, reproducible)
Generated by	`data/make_data.py` — `numpy.random.default_rng` (seeds 20260718 / 8080)
Source / license	Not applicable — synthetic, generated in-repo; no third-party rights
Access / verification date	2026-07-18
Redistribution	Not applicable — generator committed, CSV output never committed
robots.txt / ToS	Not applicable — nothing is scraped or fetched

The data is synthetic and teaches the mechanic (validating a batch against a contract); the defects are injected deliberately and documented in data/make_data.py. It does not describe any real company’s data.

Next up

Finished this one? Continue the Data Engineering track:

Idempotent loads: why re-running your pipeline shouldn’t double your data → · Browse all courses