Neural Networks: What They Actually Do

Strip the mythology and a neural network is a composition of linear regressions with bends. This lesson builds that picture precisely enough to answer interview questions, and ends where the ML-in-markets lesson began: knowing when not to use one.

The architecture

One layer computes $h = g(Wx + b)$ — a linear map, then an elementwise nonlinearity $g$. Stack layers and you compose functions:

$$\hat y = W_3\, g(W_2\, g(W_1 x + b_1) + b_2) + b_3$$

Without the nonlinearity, the stack collapses to a single linear map ($W_3W_2W_1$) — the bends are the entire point, and stating that is the standard "why do we need activations?" answer. The default activation is ReLU, $g(z) = \max(0, z)$: cheap, and its gradient doesn't vanish for active units (sigmoid/tanh saturate, which is what stalled deep nets historically). Universal approximation says wide-enough networks can fit any reasonable function — reassuring and nearly useless, since it says nothing about learnability or generalization; citing it with that caveat reads as sophistication.

Training: gradient descent + backpropagation

Pick a loss (squared error; cross-entropy for classification), then descend its gradient: $\theta \leftarrow \theta - \eta \nabla L$. Backpropagation is not a separate learning principle — it's the chain rule, organized to compute all gradients in one backward sweep at the cost of roughly one forward pass. In practice: stochastic gradient descent on mini-batches (noisy gradients, cheap steps — and the noise helps escape bad regions), with Adam as the default optimizer and the learning rate as the hyperparameter that actually matters.

The failure modes worth naming: vanishing/exploding gradients in deep stacks (mitigations: ReLU, careful initialization, residual connections, normalization layers) and the ease of memorizing noise — a big net can fit randomly labeled data perfectly, the cleanest demonstration that capacity without validation discipline is a liability. Regularization here means dropout, weight decay, and early stopping — the bias-variance lesson with new knobs.

Where deep learning earns its keep in finance

The differentiated answer, per the earlier lessons: neural nets win where features must be learned from raw structure —

• Text: news, filings, transcripts → sentiment and event signals (transformer models; the one domain where finance genuinely uses frontier architectures).
• Sequences at scale: order-book dynamics, execution modeling — where the input is thousands of raw events, not fifty engineered features.
• Derivatives approximation: networks trained to mimic slow pricing models give ~10⁴× speedups for risk scenarios — not alpha, but heavily used.

For 50-feature tabular alpha with a few thousand effective observations, gradient boosting or ridge remains the better tool: less variance, more interpretability, easier hedging. The interview isn't testing enthusiasm for deep learning; it's testing whether you can decline it with reasons.

The interview version

"Explain backprop to me in two sentences." — Forward pass computes the output and caches intermediates; backward pass applies the chain rule from the loss down, reusing those caches so all parameter gradients cost about one extra pass. "Your net gets 0.55 accuracy predicting direction — excited?" — Only after walk-forward validation, cost adjustment, a check against a logistic baseline, and confirmation the labels aren't leaky; a linear model within noise of the net means the net is overhead. Same skepticism as everywhere in this course — now aimed at the shiniest tool.