Property-Based Testing: let Hypothesis find the input you didn't think of

The data in this project is synthetic — a few example test-cases generated locally by data/make_data.py, plus code (a function under test, buggy variants, and a property oracle). That is deliberate: the subject is a testing technique, so the value is a function whose bugs are known and gradeable, not a dataset. No signup, no downloads, no GPU.

Before you start

New to property-based testing? Start here.

Most tests are example-based: you pick a few inputs and assert the expected output. assert merge([(1,3),(2,4)]) == [(1,4)]. The trouble is you only test the inputs you thought of — and bugs live in the inputs you didn’t (empty lists, touching endpoints, a value contained in another, unsorted input).

Property-based testing flips it: instead of examples, you state a property — something true for every valid input — and the library generates hundreds of inputs trying to break it. Hypothesis is the Python library for this. Two things make it powerful:

Generation — you describe the shape of inputs (lists of pairs of integers) with a strategy, and Hypothesis searches that space.
Shrinking — when it finds a failure, it automatically reduces it to the smallest input that still fails, so you debug [(0,0),(0,0)] instead of a 40-interval mess.

The running example: merge_intervals — combine overlapping or touching ranges like [(1,3),(2,5),(8,10)] → [(1,5),(8,10)]. A good property: the output is sorted, non-overlapping, and covers exactly the same points as the input. If any generated input breaks that, there’s a bug.

What you’ll learn: write a property, watch Hypothesis find and shrink a counterexample an example test missed, then use it to expose a whole battery of buggy implementations. Section 3 lists exactly what you’ll be able to do.

New to Python too? See Start Here — it starts from zero, and shows how to run this in your browser with nothing to install.

1. The Brief

A merge_intervals function passes every unit test in the repo, and it’s still wrong. You’ll see why example tests give false confidence, then write property-based tests that state what must always be true and let Hypothesis hunt down the inputs that break it — first on one buggy function, then across a set of them, each with a different subtle bug.

Property-based testing (from Haskell’s QuickCheck; Hypothesis is its Python heir) is one of the highest-leverage testing techniques there is: a handful of properties can cover input space that would take hundreds of hand-written examples, and shrinking hands you a minimal repro for free. It’s the standard tool for testing parsers, serializers, data transforms, and anything with invariants.

2. Difficulty & time

Difficulty 6 / 10. A new library and a genuine shift in how you think about tests — from “outputs for chosen inputs” to “invariants over all inputs” — plus writing strategies and reasoning about what property actually pins the behaviour down. Calibrated against the ledger (G10): a peer of vendor-dedupe (6), attrition-leakage (6), and fraud-rings (6) — each needs a real technique and a judgment a beginner gets wrong first time (here: a weak property passes buggy code, so the property itself must be strong enough). It sits below recommend-from-scratch (7). The mental model — and getting a property that’s both true and tight — is what pushes it past the 4–5 band.

Time is separate from difficulty: Part A is ~90 minutes (Hypothesis takes some absorbing); Part B is 2–3 hours. The searches run in a second or two.

3. What you’ll be able to do after

Write a Hypothesis property test — a @given strategy plus an invariant that must hold for all inputs.
Explain why example tests miss bugs — and what class of bug property tests catch.
Read a shrunk counterexample — and use the minimal failing input to locate the bug.
Design a property strong enough to matter — one that a subtly-wrong implementation actually violates.

The finished result

By the end of Part A you’ll have watched a plausible function pass every example test, then a single property test find and shrink the exact input that breaks it:

all 6 example tests pass: True
Hypothesis found a failing input: [(0, 0), (0, 0)]
minimal counterexample: [(0, 0), (0, 0)]
  the function returned [(0, 0), (0, 0)], which is not a valid merge

Six hand-picked examples said “ship it”; the property test found the touching-interval bug and shrank it to two zero-width intervals. In Part B that same property is turned loose on a battery of five buggy variants and exposes every one (1.0), where the fixed-example starter catches only 0.4 (2 of 5).

4. Prereqs & time box

You can write a function and a basic pytest test. No Hypothesis experience needed — Part A introduces strategies, @given, and shrinking. No knowledge of the interval algorithm required; the property, not the algorithm, is the point.

Where to write and run code

Two ways to run this project — pick either:

In your browser (no install). If the project page shows a ▶ Run button, click it to run Part A right here — nothing to set up. Part A uses pandas, numpy, and Hypothesis, which run in the browser.
On your computer. The one-time setup — installing Python, downloading the code from the course site (no GitHub, no git), opening a terminal, creating a virtual environment, installing the requirements, and running Part A step by step — is covered once in Start Here. Come back here once python -c "import hypothesis" runs without error. Part B (writing your own solution) needs this local setup.

Time box

Part A — Guided Build: 60–90 minutes. Hard cap.
Part B — Your Turn: 2–4 hours. Hard cap.

5. Setup & data

The data is synthetic and generated locally — see the banner above and SOURCES.md for full provenance. One command creates the example cases:

python data/make_data.py

That writes data/intervals_a.csv (git-ignored; regenerate anytime, byte-for-byte identical because the seed is fixed): 6 example interval lists (case_id, start, end) — the hand-picked unit tests Part A shows passing on buggy code. The function under test, its buggy variants, and the property oracle live in part_b/impls.py.

6. Part A — Guided Build

You’ll test a buggy merge_intervals two ways. Run the whole thing, or (better) one # %% cell at a time — see Start Here:

python part_a/find_bugs.py

Prefer the browser? If this page has a ▶ Run button, click it instead of the command above — Part A runs in your browser with nothing to install. The STEP output and every checkpoint below are identical either way.

The full, commented script is in part_a/find_bugs.py. The spine:

Step 1 — Example tests. Run the function on the 6 hand-picked cases.

Checkpoint: all 6 pass. On these inputs the function looks correct.

Step 2 — State a property and search. is_valid(x, merge(x)) must hold for all x; Hypothesis searches.

Checkpoint: Hypothesis finds a failing input the examples never covered.

Step 3 — Shrinking. Hypothesis reduces it to the minimal failing case.

Checkpoint: the counterexample shrinks to [(0,0),(0,0)] — two touching intervals left unmerged. The bug: the merge uses < where it needs <=.

Step 4 — Record it. Writes part_a/found_a.json with the counterexample.

Checkpoint: example tests said “ship it”; the property test found the break. That’s the whole lesson in one run.

Why this and not that

Why example tests missed it. The bug only shows on touching intervals ([(0,0),(0,0)]), and no hand-written example happened to include that shape. Example tests cover the inputs you imagine; property tests cover the ones you don’t.
Why the property must be strong. A weak property (e.g. “the output is a list of pairs”) passes the buggy code too. The property has to actually pin the behaviour: sorted, disjoint, and union-preserving. Getting the property tight is the real skill.
Why shrinking matters. A random failing input might have 30 intervals; you’d waste an hour bisecting it. Hypothesis hands you the two-interval essence, so the bug is obvious.

Functions you’ll meet (and where to read more)

Function	What it does here	Docs
`@given(strategy)`	Feed generated inputs into a test	given
`st.lists`, `st.integers`, `st.tuples`	Describe the input space (lists of interval pairs)	strategies
`hypothesis.find(strategy, pred)`	Return the smallest input satisfying a predicate	find
`@settings(derandomize=True)`	Make the search deterministic/reproducible	settings

If something looks off

Part A says it “found a failing input” — did I break something? No: that is exactly what Part A is meant to do. The demo merge_intervals has a deliberate bug, and Hypothesis’s whole job is to fail it — the shrunk counterexample [(0,0),(0,0)] (two touching intervals left unmerged) is the deliverable, not a sign you did anything wrong. Nothing crashes: the script prints the counterexample and exits cleanly. Because the search is seeded (derandomize=True), you’ll get exactly [(0,0),(0,0)] every run; if you later write your own property test without that setting, the reported example may look different but will always expose the same touching-interval bug.
A checkpoint number doesn’t match. The data is seeded, so the numbers are exact. Re-run python data/make_data.py (if present), then re-run Part A.
Setup problems (Python not found, pip, FileNotFoundError, re-activating the venv): see the troubleshooting list in Start Here.

7. Part B — Your Turn

Part B is the optional “your turn” half — a guided fill-in-the-blank, not a blank page. You get a working starter (part_b/starter.py) that already handles the easy version; your job is to finish the # TODO it marks out. More open than Part A’s step-by-step, but you are never staring at an empty file. If you finished Part A and want to prove the skill, this is where it happens. New to Python? It’s completely fine to stop after Part A and come back later.

Same core skill, a whole rogues’ gallery. part_b/impls.py holds a battery of buggy merge_intervals variants in MUTANTS, each with a different subtle bug, plus the correct merge_intervals and the is_valid property oracle. Expose every one: for each mutant, find an input where its output fails is_valid. The starter tries a few fixed examples and catches only the obvious mutant — the rest need inputs no one writes by hand.

The transfer: in Part A you found one bug in one function; here you must characterise a set of implementations, and your property has to be general enough that Hypothesis finds a counterexample for each subtle bug (touching endpoints, a contained interval, unsorted input, a unit-gap over-merge).

Acceptance criteria (checkable)

Your solution writes into part_b/out/:

counterexamples.json — a map {mutant_name: input_intervals}, one bug-exposing input per mutant (e.g. {"strict": [[0,0],[0,0]]}).
report.json — with keys mutants, exposed, achieved_metric (you can leave achieved_metric null; the grader scores you).

Then:

python part_b/starter.py
pytest part_b/test_solution.py -q

If pytest reports everything as skipped, it just means the output files don’t exist yet — run python part_b/starter.py first (once your solution fills in the TODOs), then re-run.

You’re done when the tests pass: your inputs expose ≥ 85% of the mutants — for each, the mutant’s output fails is_valid while the correct function’s passes (the reference exposes 100%). Fixed examples expose ~40% and fail.

Constraints — what must be true of your solution, not how to get there

Provide a bug-exposing input for each mutant — the mutant’s output must be invalid on it.
Find the inputs by property search (Hypothesis), not by reading each mutant and hand-crafting.
Each input must be a legal interval list (the correct function is valid on it).
Don’t weaken the property to make mutants “pass” — the property is the spec.

Hints

Hint 1 — reuse Part A's property

is_valid(x, impl(x)) is the property. For a given mutant, ask Hypothesis for an x where it’s False: find(lists, lambda x: not is_valid(x, mutant(x))). That’s the counterexample.

Hint 2 — loop over the battery

impls.MUTANTS is a dict of {name: function}. Run the same search for each and collect the counterexample into your {name: input} map.

Hint 3 — make it deterministic (still not the code)

Pass settings(derandomize=True, database=None) to find so you get the same counterexamples every run, and store each as a list of [start, end] pairs in counterexamples.json.

8. Self-check

You don’t need the answer key. You’re done when all of these hold:

pytest reports ≥ 0.85 exposed (aim for 1.0 — one good property finds them all).
Each counterexample is small (Hypothesis shrinks them) and, when you print mutant(x) next to merge_intervals(x), the difference is visible.
Re-running is stable — with derandomize=True the same counterexamples appear each time.

And the tell-tale: if a mutant survives (you can’t expose it), your property is too weak — it accepts that mutant’s wrong output. Strengthen the invariant, don’t special-case the mutant.

9. Stretch

Round-trip properties. Add a function that splits a merged set back into unit intervals and assert split then merge is the identity — a classic round-trip property.
Metamorphic properties. Assert that merging is idempotent (merge(merge(x)) == merge(x)) and order-invariant (shuffling the input doesn’t change the output) — properties that need no reference oracle at all.
(Genuinely hard) Stateful testing. Use Hypothesis’s RuleBasedStateMachine to test a small interval-set data structure across sequences of add/remove operations, checking the invariant after every step.

10. Ship it

Put this in a portfolio repo and it counts. In that repo’s README:

State the finding in one sentence: “a merge_intervals that passed every unit test failed a property test on [(0,0),(0,0)] — property-based testing caught the touching-interval bug.”
Show an example test (green) next to the property test (red, with the shrunk counterexample).
Include the property and one paragraph on why an invariant over all inputs beats a list of examples. That paragraph is what signals you understand property-based testing, not just Hypothesis syntax.

Keep make_data.py, impls.py, and the tests in the repo so anyone can reproduce your findings from zero.

11. Sources

Rendered from SOURCES.md.

Field	Value
Dataset	Synthetic example interval-merge test-cases (6 cases, 17 intervals)
Type	Synthetic
Generated by	`data/make_data.py` (this repo), seed `20260736`
Source URL	Not applicable — generated, not fetched
License	Not applicable — synthetic, no third-party rights
Access / generation date	2026-07-18
Redistribution	Permitted — synthetic; output regenerated from the seeded script, never committed
robots.txt checked	Not applicable — no scraping
ToS URL	Not applicable — no external source

Synthetic data teaches the mechanic; it does not prove a finding about the real world. The example cases and buggy variants were built on purpose — exposing them demonstrates the technique, not a claim about any real codebase.