Reinforcement Learning and Optimal Execution

Reinforcement learning is the branch of ML for sequential decisions under uncertainty — which describes trading exactly, and explains both the perennial excitement about RL in finance and the thinness of its production track record. This lesson gives the framework, the one algorithm worth knowing, and the judgment layer.

The framework: MDPs

A Markov decision process is the Markov chain (probability course) plus choices and rewards: states $s$, actions $a$, transition probabilities $P(s' \mid s, a)$, rewards $r(s, a)$. A policy $\pi$ maps states to actions; the goal is maximizing expected (discounted) cumulative reward. The central objects are value functions:

$$Q(s, a) = \text{expected total reward from taking } a \text{ in } s \text{, then acting optimally}$$

satisfying the Bellman equation — the value of now equals reward plus discounted value of the best next: $Q(s,a) = E[r + \gamma \max_{a'} Q(s', a')]$. Recognize it: this is the dynamic-programming lesson's recurrence, and the American-option lesson's max(exercise, continue) is a Bellman equation. When the model ($P$, $r$) is known, "RL" is just DP — backward induction. RL proper begins when the model is unknown and must be learned from experience.

Q-learning in one paragraph

Interact with the environment; after each transition $(s, a, r, s')$, nudge your estimate toward the Bellman target:

$$Q(s,a) \mathrel{+}= \alpha \left[ r + \gamma \max_{a'} Q(s', a') - Q(s,a) \right]$$

— a stochastic approximation that converges (tabular case, enough exploration) to the optimal $Q$, from which the policy is "act greedily." The exploration–exploitation trade-off (ε-greedy: mostly act greedily, sometimes explore) is the secretary problem's tension industrialized. Deep RL replaces the table with a neural network (the network lesson's function approximation) — powerful, famously unstable, and data-hungry.

Where it actually fits in trading

The honest map, which is the interview answer:

• Optimal execution — the best fit. The Almgren–Chriss schedule (execution lesson) is the known-model DP solution; RL versions learn from fills how impact actually behaves and adapt the schedule to book state. Bounded episodes, dense feedback, simulable environment: the preconditions RL needs, and execution has them.
• Market making — inventory-and-spread control as an MDP (state: inventory, queue position, signals; actions: quote placements). Active research, some production claims; the adverse-selection environment is adversarial and shifting, which strains the stationarity RL quietly assumes.
• Portfolio-level "learn to trade from returns" — the weakest case: sample sizes are tiny relative to state spaces, the environment is non-stationary (the ML-in-markets lesson's core objection), and rewards are so noisy that credit assignment drowns. Most published successes here are backtest artifacts wearing sin #3.

The recurring failure mode deserves its own sentence: RL agents exploit simulator bugs — an agent that discovers your fill model lets it earn the spread risk-free has learned the simulator, not the market. Sim-to-real gap is the field's central practical problem.

The interview version

"Would you use RL for execution?" — Yes, framed properly: state = (inventory remaining, time left, book features), actions = child-order choices, reward = negative implementation shortfall; warm-start from Almgren–Chriss, validate against it as the baseline, and distrust anything the simulator can't defend. "What's the relationship between RL and dynamic programming?" — Same Bellman mathematics; DP when the model is known, RL when it must be learned — and in finance, preferring the DP end of that spectrum whenever a credible model exists is not conservatism, it's sample-efficiency. That calibrated enthusiasm is precisely the tone these questions are fishing for.