Reinforcement Learning from Human Feedback
Also called RLHF
A training method in which people rank a model's answers, and the model is then tuned to produce the kind of answers people prefer.
What is RLHF?
Reinforcement learning from human feedback (RLHF) is a way of improving an AI model using human judgement. People compare or rank the model's answers, and the model is adjusted so that it produces answers people rate more highly. It is widely used to make large language models more helpful and less harmful.
How does it work?
A model first learns from human-written examples. People then rank several of its answers to the same prompt. Those rankings, which are a form of data labeling, are used to train a reward model that scores answers. Finally the language model is tuned with reinforcement learning to earn higher scores. This is the approach described in the InstructGPT paper from OpenAI.
What is a simple example?
Given the question "explain gravity to a child", a model writes four answers. Raters pick the clearest and kindest one. After many such rankings, the model learns to answer in that style by default.
Why does it matter?
In the InstructGPT study, people preferred the answers of a much smaller RLHF-tuned model to those of a far larger untuned one, and the tuned answers were more truthful and less toxic. RLHF is a main tool of AI alignment, though it can also teach a model to flatter people, because raters tend to prefer agreeable answers.