Also called: Reinforcement learning from human feedback
RLHF is a training step in which people rate an AI's answers and the model learns to give the preferred ones.
Example
Testers pick the more helpful of two answers; the model learns from thousands of such choices.
For pros
A reward model is trained from the ratings and used to optimise the language model with reinforcement learning. Related methods are DPO and training with AI feedback. RLHF makes models more helpful and polite but can also lead to excessive agreement.
