Classification and Metrics: When Accuracy Lies

Finance's classification problems — default, fraud, crash, fill-or-no-fill — share a property that breaks naive evaluation: the interesting class is rare. This lesson builds the classification toolkit with that imbalance at the center, because that's where interviews aim.

Logistic regression: the workhorse

Linear regression on a 0/1 target predicts outside $[0,1]$; logistic regression fixes this by passing a linear score through the sigmoid:

$$P(y = 1 \mid x) = \frac{1}{1 + e^{-x^\top\beta}}$$

Equivalently, the log-odds are linear: $\ln\frac{p}{1-p} = x^\top\beta$ — each unit of a feature shifts the odds by a constant factor $e^{\beta_i}$, which is how credit-scoring coefficients are actually read. Fit by maximizing likelihood (minimizing log-loss); regularize exactly as in the regression lesson. It outputs probabilities, not labels — hold that thought.

Why accuracy lies

Predict "no default" for everyone in a portfolio with a 1% default rate: 99% accuracy, zero usefulness. With imbalanced classes, accuracy measures the base rate, not the model. The honest ledger is the confusion matrix, and from it:

• Precision = of those flagged, how many were real? (Cost of false alarms.)
• Recall = of the real ones, how many did we catch? (Cost of misses.)

They trade off through the decision threshold: flag more → recall up, precision down. There is no model-only answer to "where's the right threshold?" — it's set by the costs: in fraud, a false alarm costs a review; a miss costs the loss. Expected-cost-per-threshold is the calculation, and framing it that way in an interview converts an ML question into the decision-theory answer they wanted.

This is also the rare-disease Bayes question from the probability course wearing ML clothes: with a 1% base rate, even a great classifier's flags are mostly false alarms. Base rates dominate; anyone who quotes precision without the prevalence is hiding the ball.

Threshold-free evaluation: ROC and AUC

The ROC curve traces (false-positive rate, true-positive rate) across all thresholds; AUC — the area under it — has a clean meaning: the probability a random positive scores higher than a random negative. 0.5 is coin-flipping; 1.0 is perfect separation. AUC is imbalance-robust and threshold-free, which is why it's the default screening metric — but it says nothing about costs, and with extreme imbalance the precision-recall curve is the more truthful picture (a model can have AUC 0.95 and still be useless at any workable precision).

Calibration: do the probabilities mean anything?

A model is calibrated if events it calls "30%" happen 30% of the time. Separation (AUC) and calibration are different virtues: a model can rank perfectly and be wildly miscalibrated. In finance you usually need the probability itself — for expected loss, for Kelly-style sizing, for pricing — so calibration matters more here than in most ML domains. Check with a reliability plot; fix with Platt scaling or isotonic regression. "AUC for ranking, calibration for sizing" is a sentence that earns its keep.

The interview version

"Your fraud model is 99% accurate. Good model?" — Base rate first; then the confusion matrix; then "at what precision and recall, and what do the two errors cost?" "AUC is 0.75 — deploy?" — Depends on prevalence and costs; look at precision at realistic alert volumes, and check calibration if the scores feed a sizing decision. The pattern across all of these: refuse the single-number summary, name the trade-off, attach the costs. That reflex — not any formula — is what the question is screening for.