ML//RL

RL (reinforcement learning) is a branch of machine learning that trains an agent by trial and error to maximize its cumulative reward, with no labeled data: only actions, states and outcomes. It solves the problem of a Markov decision process and of optimal control (choose actions whose consequences pile up over time) without being handed the model: the transition probabilities and the reward are discovered by trying.

RL (reinforcement learning) is a branch of machine learning that trains an agent by trial and error to maximize its cumulative reward, with no labeled data: only actions, states and outcomes. It solves the problem of a Markov decision process and of optimal control (choose actions whose consequences pile up over time) without being handed the model: the transition probabilities and the reward are discovered by trying.

The agent acts, receives a reward and adjusts a policy, the function from state to action that in control is the controller itself. Q-learning learns the value of each action in each state from experienced transitions, pushed by the temporal-difference error, while policy gradient methods optimize the action probabilities directly. When the states are too many for a table, a neural network approximates the values or the policy: deep reinforcement learning. Learning by trying costs reward, and on a machine it can cost the machine (exploration-exploitation trade-off). RL reaches language models through RLHF, where the reward comes from a reward model, a learned proxy for human preferences.

RL wins with a fast, faithful simulator and a clear reward.

With a model of the system, dynamic programming, MPC or an operations-research solver usually win sooner and with fewer surprises, which is why RL in industrial production is still niche.

What works has a cheap simulator behind it. Atari games (2015) and Go (2016, AlphaGo) had perfect ones; legged locomotion is trained in simulation with domain randomization to cross the sim-to-real gap and already ships on commercial quadrupeds. In 2023 a policy trained in simulation (Swift, published in Nature) beat world-champion drone pilots, on a known track and in controlled conditions: real, and still a research result (technology maturity).

Learning on the real plant mostly fails. Deep RL needs millions of episodes, exploring means breaking things, and a chemical plant cannot be reset like a simulator. Once trained, though, the policy is cheap to run: MPC optimizes at every step, RL optimized once and only evaluates a network.

The reward is where projects go wrong (reward function). A delayed reward also leaves open which actions earned it, the credit assignment problem that value functions exist to solve. With reward misspecification the policy optimizes the reward exactly as written and finds every shortcut the writing leaves open (reward hacking, specification gaming). The real cost is weeks of engineering on the simulator, the randomization ranges and the reward, far more than the GPU hours.

A learned policy comes with no formal guarantee, so certifying one is open, and where it flies it usually sits behind a safety filter. Multi-agent RL against adversaries that also learn (game theory) is research.