Unsupervised Learning: Clustering and PCA in Markets

Supervised learning needs labels; markets mostly don't hand them out. Unsupervised methods — clustering and dimensionality reduction — find structure without a target, which makes them both genuinely useful in finance and uniquely easy to fool yourself with. Both halves are interview material.

K-means: the default clustering

Choose $k$ centers, assign each point to its nearest center, move each center to its cluster's mean, repeat until stable. Cheap, scales well, and its assumptions bind: clusters are round-ish, similar-sized, defined by Euclidean distance — so standardize features first (a returns column in decimals next to a volume column in millions means volume is the distance).

Choosing $k$: the objective (within-cluster variance) always improves with more clusters, so look for the elbow where improvement flattens, or use silhouette scores — while admitting these are heuristics. Modern initialization (k-means++) mostly fixes the bad-local-optimum problem; mention it and move on.

Market uses: regime detection (cluster days by vol, correlation, breadth — calm/stressed/crisis emerge without labels), client/flow segmentation, grouping stocks by return co-movement to propose pairs candidates for the stat-arb pipeline (cluster first, then test cointegration properly — clustering only nominates).

Hierarchical clustering builds a merge tree (dendrogram) instead of forcing one $k$ — the natural fit when the truth is nested, like sector → industry → sub-industry, and the basis of portfolio methods like hierarchical risk parity.

PCA: finding the axes that matter

PCA re-derives the linear-algebra lesson as a learning method: the covariance matrix's top eigenvectors are the directions of greatest variance, and projecting onto the top few compresses hundreds of correlated series into a handful of drivers.

For markets, the top components of equity returns are strikingly interpretable: the first is the market (broad co-movement, often 30–50% of variance), the next few resemble sectors or styles. Uses that come up in interviews:

• Statistical factor models: hedge the top components instead of named factors — the purely data-driven cousin of the CAPM lesson's approach.
• Yield curves: three components — level, slope, curvature — explain ~99% of curve moves; the canonical PCA success story and a favorite follow-up.
• Covariance denoising: with $n$ assets and not-many-more observations, sample covariance eigenvalues are contaminated by noise (random-matrix theory gives the noise band); keeping the components above it and shrinking the rest yields the cleaned matrices real optimizers run on.

Caveats that separate practitioners from tourists: components are linear combinations with no guaranteed meaning (interpretability is empirical luck), signs are arbitrary, and the decomposition is unstable — top components rotate across estimation windows, so a hedge built on last year's PCA drifts.

The validation problem

The uncomfortable core of unsupervised work: without labels, there is no accuracy. K-means will happily "find" three clusters in uniform noise; PCA always produces components. The professional's tests are stability (do clusters/components persist across time windows and bootstraps?) and economic validation — does the regime label actually predict something out-of-sample? Does the pairs candidate cointegrate? An unsupervised structure is a hypothesis generator, not a result.

The interview version

"How would you detect market regimes without labeling them yourself?" — Features (rolling vol, average correlation, dispersion), standardize, cluster; validate by stability across windows and by whether regime membership predicts anything (vol tomorrow, strategy P&L) out-of-sample. "Your PCA factor explains 40% of variance — is it the market?" — Check loadings (roughly uniform and positive?), correlate the component with the index; and concede the sign and composition drift. The disciplined hedging of claims is, as everywhere in ML-for-markets, the actual answer.