Is this review positive? Reading sentiment from text

Data: both parts use synthetic data, generated locally by data/make_data.py (seeded, reproducible). Nothing is downloaded and nothing is scraped from any store or review site; no signup, no GPU. Synthetic is essential here: to score a sentiment classifier — and to build in the exact ways it fails (negation, class imbalance) — you must know the true sentiment of every review, which you only do when you generate it.

Before you start

New to this? Two minutes of vocabulary, because the whole project turns on it.

Sentiment analysis is deciding whether a piece of text is positive or negative — here, whether a product review is a thumbs-up or a thumbs-down. It’s the “hello world” of turning messy text into a prediction.

Text classification — training a model to sort text into categories (here, two: positive / negative). You give it example reviews with their labels and it learns the pattern.
Bag of words — the simplest way to feed text to a model: ignore word order, just count which words appear. “great product” and “product great” look identical to it.
TF-IDF — a smarter word count. It scores a word up when it’s frequent in this review but down when it’s common across all reviews (so “the” barely counts). One number per word per review.
Unigram vs bigram — a unigram feature is one word (“great”); a bigram is a pair of adjacent words (“not great”). This single distinction is the hinge of the whole project.

A tiny example — take “the blender is not great.” To a bag of single words this contains “great,” so it leans positive — exactly backwards. The fix is to also look at the pair “not great,” which the model can learn means the opposite of “great.” That is the entire difference between Part A’s model and Part B’s.

What you’ll learn: to build a text classifier, to evaluate it honestly (not just one accuracy number, but per-class scores and the words it keys on), and to recognise the two places a simple bag-of-words model breaks — negation and class imbalance — and fix them.

New to Python? Section 4 (“Where to write and run code”) starts from zero and can run Part A in your browser.

1. The Brief

You’ll train an ordinary text classifier — TF-IDF features into a logistic regression — on a catalogue of product reviews, and watch it score a respectable 0.848 accuracy on clean test reviews. Then you’ll refuse to trust that one number: you’ll read the confusion matrix and per-class scores, list the exact words the model keys on, and hand it 60 reviews that use negation (“not great,” “not flimsy”) — where the same model collapses to 0.033 accuracy, worse than a coin flip, because it reads the sentiment word and cannot see the “not” in front of it. Then, in Part B, you’ll face a harder, real-shaped corpus — mostly positive (imbalanced) and full of negation — where that naive approach manages only 0.57 macro-F1, and you’ll fix it to ~0.95 with two deliberate changes.

Sentiment classification is the front door to applied NLP, and this is the honest version of it: not “call .fit() and read the accuracy,” but what did the model actually learn, and where will it embarrass you in production? The answer — a single-word model is blind to negation, and accuracy lies when one class is rare — is the difference between a demo that scores well on a slide and a classifier you’d trust to route real complaints.

2. Difficulty & time

Difficulty 4 / 10. It sits above the difficulty-3 builds benford-fraud and cohort-retention: those each teach one clean data-handling mechanic with a single tool, whereas this chains a small machine-learning pipeline (a vectorizer, a model, a train/test discipline) and turns on a genuinely non-obvious idea — why a bag-of-words model can’t read negation, and how to interrogate what it learned rather than trust its accuracy. It’s about level with ab-test-readout (difficulty 4): both bring a real second concept beyond wrangling and a Part B that is a true transfer, not a re-run. It stays below accuracy-is-lying (difficulty 5), which assumes you have already trained a classifier and layers on a fuller scikit-learn pipeline, a leaky-feature judgment on real data, and threshold tuning off a precision–recall curve; here Part A’s path is fully specified, it’s one vectorizer and one model, and no prior-classifier experience is assumed. Anchored to the rubric’s 3–4 band and those named ledger projects (G10).

Time is separate from difficulty: the script runs in about a second and a half (measured Part A wall-clock 1.56s). Part A is about 45 minutes of reading and running; Part B is 2–3 hours of your own work.

3. What you’ll be able to do after

Build a text classifier from scratch: TF-IDF features into a linear model, fit on training reviews and scored on held-out ones — using only pandas and scikit-learn.
Evaluate it honestly: read a confusion matrix and per-class precision/recall, and list the top words driving each class, instead of trusting a single accuracy number.
Explain, with the model’s own weights, why a single-word bag-of-words model cannot read negation — and demonstrate the collapse on a negation probe.
Handle a harder corpus that is class-imbalanced and negation-heavy: switch the metric to macro-F1, add word-pair features, and weight the rare class — and know why each change matters.

The finished result

By the end of Part A you’ll have trained a classifier, evaluated it past its flattering accuracy, and caught it failing on negation in a way you can explain:

accuracy on clean test reviews: 0.848        looks like a decent classifier

confusion matrix          pred neg   pred pos
   true negative             200         33
   true positive              37        190     per-class recall: neg 0.858 | pos 0.837

most POSITIVE words: durable, sturdy, superb, excellent, sleek, crisp, roomy, comfortable
most NEGATIVE words: unreliable, frustrating, awful, useless, scratchy, shoddy, tinny, overpriced
weight of the word "not": -0.534   (ranked #42 of 83 words — a weak, unreliable signal)

accuracy on the 60 negation reviews: 0.033   the SAME model, on "not great" / "not flimsy"

The model reads plain sentiment words well, but it has one weight for “not” that has to serve both “not great” and “not terrible,” so on negation it inverts almost every review — the exact weakness Part B makes you fix.

4. Prereqs & time box

You can write a function and use a pandas DataFrame. You do not need to have trained a model before — the train/fit/predict steps are built up and explained as we go, and every scikit-learn function is linked in the “Functions you’ll meet” table in section 6. No maths beyond “a bigger weight means the model trusts that word more.”

Where to write and run code

Two ways to run this project — pick either:

In your browser (no install). If the project page shows a ▶ Run button, click it to run Part A right here — nothing to set up. Part A uses only pandas and scikit-learn, both of which run in the browser.
On your computer. The one-time setup — installing Python, downloading the code from the course site (no GitHub, no git), opening a terminal, creating a virtual environment, installing the requirements — is covered once in Start Here. Come back here once python -c "import sklearn, pandas" runs without error. Part B (writing your own solution and running pytest) needs this local setup.

Time box

Part A — Guided Build: 30–60 minutes. Hard cap 90.
Part B — Your Turn: 1.5–3 hours. Hard cap 4.

5. Setup & data

All the data is synthetic, generated by one seeded command (byte-for-byte reproducible, git-ignored, never committed):

python data/make_data.py

That writes five CSVs into data/:

Part A: reviews_train.csv and reviews_test.csv — a balanced catalogue of labelled reviews (1,400 train / 460 test); and negation_probe.csv — 60 all-negation hard cases.
Part B: reviews_hard_train.csv — a labelled, harder corpus (1,700 reviews, mostly positive, negation-heavy); and reviews_hard_test.csv — 640 reviews with the label column removed on purpose. Which sentiment each test review truly has is regenerated from the seed by the grader, not handed to you.

The generative story — a mid-size online store’s product reviews, with negation, mixed “looks-cheap-but-works-great” contrast reviews, and (in Part B) realistic class imbalance — and every injected defect are documented in data/make_data.py. Synthetic data teaches the mechanic; it does not prove a finding about any real store.

6. Part A — Guided Build

Run the whole thing, or (better) one # %% cell at a time — see Start Here:

python data/make_data.py               # once, writes the review CSVs
python part_a/build_classifier.py

Prefer the browser? If this page has a ▶ Run button, click it instead of the commands above — Part A runs in your browser with nothing to install. The STEP output and every checkpoint below are identical either way.

The full, commented script is in part_a/build_classifier.py. Every scikit-learn function used is linked in the “Functions you’ll meet” table at the end of this section.

Step 1 — Load the labelled reviews. Each row is one review’s text plus the thumbs-up/down the shopper gave it. We keep a separate test file the model never trains on.

Checkpoint: 1,400 training reviews and 460 test reviews, split roughly 50/50 positive/negative.

Step 2 — TF-IDF features, then a linear model. TfidfVectorizer turns each review into a row of word scores; LogisticRegression learns a weight per word. Fit the vectorizer on the training text only, then transform the test text with that fixed vocabulary — fitting on everything would leak test words into training.

Checkpoint: a vocabulary of 83 distinct words; the training matrix is 1,400 × 83.

Step 3 — Score it the usual way: accuracy on the clean test set.

Checkpoint: accuracy 0.848. Looks like a decent classifier — which is exactly when you should stop trusting a single number.

Step 4 — Look past accuracy: a confusion matrix and per-class scores. One number hides where the mistakes are.

Checkpoint: confusion matrix [[200, 33], [37, 190]] (rows = true neg/pos, cols = predicted); 70 of 460 reviews misclassified; per-class recall 0.858 (negative) and 0.837 (positive). The errors are not random — they’re the negation and contrast reviews.

Step 5 — What did the model actually learn? Read its biggest word weights.

Checkpoint: the top words are pure sentiment words (positive: durable, sturdy, superb, excellent…; negative: unreliable, frustrating, awful, useless…), while the word “not” carries a weight of just −0.534, ranked #42 of 83 — a weak, unreliable signal. The model reads words in isolation, so it effectively cannot see negation.

Step 6 — The trap. Feed it 60 reviews that are all negation (“not great,” “not flimsy”).

Checkpoint: accuracy on the negation probe 0.033 — the model inverts almost every one. Clean accuracy looked fine; on negation the same model collapses. That is the problem Part B hands you, on a harder corpus.

Why this and not that

Why train a model, instead of counting words from a “positive words” list? A hand-built lexicon can’t weigh words (is “sleek” worth as much as “excellent”?), can’t pick up your catalogue’s own vocabulary, and still can’t handle negation. Learning weights from labelled reviews fixes the first two immediately — and this project is about seeing, and then fixing, the third.
Why read per-class scores, not just accuracy? Accuracy is one blended number; it can look healthy while the model quietly fails one class. That becomes decisive in Part B, where the reviews are mostly positive and a lazy “everything’s positive” model posts high accuracy while catching almost no complaints — which the per-class recall exposes at a glance.
Why fit the vectorizer on the training set only? If you fit TF-IDF on all the reviews, the test set’s words shape the vocabulary and their frequencies inform the scores — the model has peeked at the exam. Fit on train, transform test, and your accuracy is honest.

Functions you’ll meet (and where to read more)

Function	What it does here	Docs
`pd.read_csv`	Load the review CSVs	read_csv
`TfidfVectorizer`	Turn review text into per-word TF-IDF features	TfidfVectorizer
`LogisticRegression`	Learn a weight per word; predict positive/negative	LogisticRegression
`confusion_matrix`	The 2×2 of true vs predicted labels	confusion_matrix
`classification_report`	Per-class precision, recall, F1 in one table	classification_report
`precision_recall_fscore_support`	The same numbers, as arrays to save	precision_recall_fscore_support
Concept	TF-IDF, explained	tf–idf (Wikipedia)
Concept	The bag-of-words model	Bag-of-words (Wikipedia)
Going deeper	scikit-learn’s text feature-extraction guide	Text feature extraction

If something looks off

The negation probe scores ~0.03 — did I break the model? No — that near-zero score is correct and expected, and it’s the whole lesson. Every probe review negates its sentiment word (“not great” = a complaint, “not flimsy” = praise). A single-word model sees only “great” or “flimsy” and answers backwards, so it gets almost all 60 wrong. A model that’s “worse than guessing” on a specific pattern is telling you it’s blind to that pattern — read it as a finding, not a bug.
A checkpoint number doesn’t match. The data is seeded, so accuracy, the confusion matrix, the word list, and the probe score are all exact. Re-run python data/make_data.py (you may have skipped or edited it), then re-run Part A.
Setup problems (Python not found, pip, FileNotFoundError, re-activating the venv): see the troubleshooting list in Start Here.

7. Part B — Your Turn

Part B is the optional “your turn” half — a guided fill-in-the-blank, not a blank page. You get a working starter (part_b/starter.py) that already handles the easy version; your job is to finish the # TODO it marks out. More open than Part A’s step-by-step, but you are never staring at an empty file. If you finished Part A and want to prove the skill, this is where it happens. New to Python? It’s completely fine to stop after Part A and come back later.

A harder corpus, the real shape of the job. data/reviews_hard_train.csv holds 1,700 labelled reviews and data/reviews_hard_test.csv holds 640 more with the labels hidden. This corpus is mostly positive (about 72% — like a real review stream) and negation-heavy (about 40% of reviews negate their sentiment word). You train on the labelled file and predict the hidden one.

Same core skill as Part A (TF-IDF text classification), now where Part A’s naive recipe actively fails: single-word features read “not great” as praise, and an unweighted model leans to the positive majority and misses complaints. Your job is to fix both.

Acceptance criteria (checkable)

Fill in part_b/starter.py (it starts as the naive Part A recipe, which fails on purpose), then:

python part_b/starter.py               # writes part_b/out/
pytest part_b/test_solution.py -q

If pytest reports everything as skipped, it just means the output files don’t exist yet — run python part_b/starter.py first (once your solution fills in the TODOs), then re-run.

Your solution writes into part_b/out/:

predictions.csv — review_id, predicted_label (“positive” or “negative”).
report.json — reviews, flagged_negative, macro_f1, achieved_metric.

You’re done when the tests pass. The grader knows the true labels (from the seed) and scores your macro-F1 — the average of the two classes’ F1, so getting the rare complaints right counts as much as the easy positives. It requires ≥ 0.82. For reference, the naive Part A recipe scores only 0.57 here; the reference solution reaches ~0.95.

Constraints — what must be true, not how to get there

Predict each test review from its text, not from any label (the test file has none, and you must not read the hidden labels out of make_data.py).
Score yourself on macro-F1, not accuracy — on this imbalanced corpus a model that leans positive can post high accuracy while failing the rare class. Hold out part of the training file as a validation slice to estimate your macro-F1 before you submit.
Your reported achieved_metric (if you fill it) must match your predictions — the grader recomputes it.

Hints

Hint 1 — reuse Part A, but change the metric you trust

The pipeline is the same as Part A: TfidfVectorizer → LogisticRegression → predict. What must change first is how you judge it. Accuracy is misleading when ~72% of reviews are positive; split a validation slice off the training file and measure macro-F1 on it, so you can tell whether a change actually helped the rare class.

Hint 2 — two knobs, and you need both

There are exactly two levers here. One lets the vectorizer see word pairs so “not great” becomes its own feature, distinct from “great.” The other tells the model that the classes are imbalanced so the rare negatives are not drowned out. Look at the parameters of TfidfVectorizer and LogisticRegression.

Hint 3 — why one lever isn't enough (still not the code)

Measured on this corpus: word pairs alone reach ~0.71 macro-F1 (it can now read negation, but still misses the rare class); class weighting alone reaches ~0.50 (it cares about the rare class, but still reads “not great” as praise). Only both together — ngram_range=(1, 2) and class_weight="balanced" — clear the floor, at ~0.95. Each fixes a different failure.

8. Self-check

You don’t need the answer key. You’re done when:

You can state, in one sentence, why the Part A model scored 0.848 on clean reviews but 0.033 on the negation probe.
You can point at the model’s weight for “not” and explain, from that one number, why a single-word model can’t do negation.
Your Part B macro-F1 clears 0.82 on the hidden test set, and you can name which lever fixes negation and which fixes imbalance.
You checked macro-F1, not accuracy, and you can say what your model’s per-class recall is — especially on the rare negative class.

9. Stretch

Read the bigrams the fix learned. Pull the reference model’s top negative features and confirm pairs like “not great” and “not comfortable” now carry negative weight — the model has literally learned the negation you taught it to see.
How far do n-grams help? Try ngram_range=(1, 3) and tune min_df. Does adding trigrams keep raising macro-F1, or does it start overfitting the vocabulary? Report the curve.
(Genuinely hard) Domain shift. Extend make_data.py to emit a small out-of-domain corpus — different products and a partly different vocabulary — and score your Part B model on it. Watch macro-F1 fall, and explain why: a bag of words keyed to this catalogue’s words doesn’t transfer to another, and what you’d change (shared embeddings, more data) to fix it.

10. Ship it

Put this in a portfolio repo and it counts. In that repo’s README:

Lead with the one-liner a hiring manager gets: “A single-word sentiment model scored 85% — and got ‘not great’ exactly backwards. Here’s how I found that with per-class scores and the model’s own weights, and fixed it to 0.95 macro-F1 on a harder, imbalanced corpus.”
Show the Part A evidence: the confusion matrix next to the accuracy, the top-words list, and the negation probe’s collapse.
State the discipline in one sentence: don’t trust one accuracy number — read per-class scores and the model’s weights, and know that a single-word model is blind to negation and that accuracy lies when a class is rare.

Keep make_data.py and the tests in the repo so anyone can reproduce your numbers from zero.

11. Sources

Rendered from SOURCES.md.

Field	Value
Datasets	Synthetic — balanced review corpus + negation probe (Part A); imbalanced, negation-heavy corpus (Part B)
Data type	Synthetic (seeded, reproducible)
Generated by	`data/make_data.py` — `numpy.random.default_rng` (seeds 20260718–20260722)
Source / license	Not applicable — synthetic, generated in-repo; nothing scraped, no third-party rights
Access / verification date	2026-07-18
Redistribution	Not applicable — generator committed, CSV output never committed
robots.txt / ToS	Not applicable — nothing is scraped or fetched

The data is synthetic and teaches the mechanic (building and honestly evaluating a text classifier, and the negation / imbalance failure modes); the true sentiment of every review is known because it was set by the generator, and documented in data/make_data.py. It describes no real product.