RLHF Series: 3/17 [Training Overview — From Human Preferences to Optimizing Language Models]

← Part 2 — A Tiny History of RLHF | Part 4 — Instruction Fine-Tuning →

Modern language models are not made useful by pretraining alone. A pretrained model learns to predict the next token from large amounts of text, but that does not automatically make it a reliable assistant, a strong problem solver, or a model that consistently follows human preferences.

Post-training bridges this gap.

Reinforcement Learning from Human Feedback (RLHF) is one of the central techniques in this process. It takes a capable pretrained model, defines what desirable behavior looks like through a reward signal, and optimizes the model to produce responses that score well against that signal.

But there is a fundamental challenge: the reward model is only an approximation of human preferences. If we optimize it too aggressively, the model can learn to exploit its weaknesses rather than become genuinely better.

In this article, we will explore the training overview from three perspectives:

• Research: the mathematical formulation, assumptions, and trade-offs behind RLHF.
• System design: how the models, data pipelines, reward computation, and optimization loop fit…