RLHF

Reinforcement Learning from Human Feedback (RLHF) is a training methodology that aligns language models with human values and preferences. The process typically involves three stages: first, a base model is pre-trained on large text corpora; second, human annotators rank multiple model outputs for the same prompt by quality; third, a reward model is trained on these rankings and used to fine-tune the base model via reinforcement learning (usually Proximal Policy Optimization, or PPO).

RLHF was a key innovation behind ChatGPT and is used by most major AI labs to make models more helpful, harmless, and honest. Without RLHF, a pre-trained language model simply predicts the next token - it has no concept of being "helpful" or avoiding harmful content. RLHF provides the training signal that transforms a raw text predictor into a useful assistant.

The technique has known limitations. Human raters may have biases, disagree with each other, or inadvertently reward sycophantic behavior. The reward model is an imperfect proxy for true human preferences. Alternative approaches like Direct Preference Optimization (DPO) and Constitutional AI (CAI) have emerged to address some of these shortcomings, but RLHF remains foundational to understanding how modern AI assistants are built and why they behave the way they do.