Walk through RLHF end to end: reward model, policy optimization, and the KL penalty.

RLHF trains a preference or reward model from comparisons, then optimizes the policy while constraining drift from a reference model, often with a KL term. Reward misspecification, overoptimization, and preference bias require held-out human evaluation.

Use equations or tensor shapes where they clarify the claim, then name an experiment or ablation that would distinguish competing explanations.