RLHF

Reinforcement Learning from Human Feedback (RLHF) is a training approach that aims to align model behavior with human preferences by combining human judgments with reinforcement learning. The model learns from both static data and feedback about which outputs are preferred.

The typical RLHF workflow has three parts: a reward model, a policy, and a feedback loop. Humans provide data by ranking outputs or giving demonstrations. This data trains a reward model that assigns a scalar score to outputs. A policy is then updated with reinforcement learning to maximize the predicted reward. The loop repeats as the policy and reward model are refined.

Variants include preference-based learning, where humans compare pairs of outputs, and direct reward learning. RLHF is commonly used to fine-tune large language models to follow instructions, be helpful, and avoid unsafe content.

Historical context: RLHF gained prominence with OpenAI's InstructGPT and subsequent chat models, illustrating a practical path to alignment when a fully specified objective is difficult. It has influenced safety, usability, and alignment research in several large AI systems.

Limitations and challenges include dependence on the quality and representativeness of human feedback, the possibility that the reward model misaligns with the true objective, data collection costs, and risks of reward hacking or distributional shift.