RLHF (Reinforcement Learning from Human Feedback)

Overview

RLHF addresses a fundamental challenge: language models trained only on internet text don't inherently know what humans want. A model might generate toxic content, refuse reasonable requests, or provide incorrect information confidently. RLHF solves this by directly incorporating human preferences into training. The result is models that follow instructions, admit uncertainty, and refuse harmful requests.

Three-Stage Training Process

• **Stage 1: Pre-training**: Train base language model on massive text corpus (trillions of tokens)
• **Stage 2: Supervised Fine-Tuning (SFT)**: Fine-tune on high-quality human demonstrations of desired behavior
• **Stage 3: Reward Modeling + RL**: Collect human preference rankings, train reward model, optimize policy with PPO
• **Iteration**: Continuously collect feedback and retrain reward model for ongoing improvement

How Human Feedback Works

Human labelers receive a prompt and 2-4 model outputs. They rank outputs from best to worst based on helpfulness, harmlessness, and honesty. These rankings train a reward model (typically another transformer) that predicts human preferences. The reward model then guides RL training, acting as a proxy for human judgment on billions of examples—far more than humans could label directly.

Business Integration

RLHF enables businesses to customize AI behavior to their specific values and use cases. A customer service chatbot can be trained to prioritize empathy and resolution speed. A legal document assistant learns to prioritize accuracy and conservative answers over creative speculation. A coding assistant learns to write clean, well-documented code rather than just functional code. The key advantage: you don't need to manually write rules—you simply provide examples of good and bad behavior.

Real-World Example: Custom Support Bot

An e-commerce company has specific policies: always offer to escalate frustrated customers, never promise refunds without manager approval, maintain brand voice. Traditional fine-tuning struggles with edge cases. With RLHF, labelers rank 5,000 support conversations by policy adherence. After training, policy violation rate drops from 12% to 2%, while customer satisfaction increases 18%. The model learns nuanced judgment rather than rigid rules.

Implementation Example

Technical Specifications

• **Reward Model Training**: Typically requires 10,000-100,000 human preference comparisons
• **RL Training**: 10,000-1,000,000 gradient steps depending on model size
• **KL Penalty**: Controls how far policy can drift from original model (typical β=0.01-0.1)
• **Algorithms**: PPO (most common), DPO (Direct Preference Optimization, no reward model), RLAIF (AI feedback instead of human)
• **Compute Cost**: 10-20% of pre-training cost for full RLHF pipeline
• **Human Labeling**: $0.10-$2.00 per comparison depending on task complexity

Best Practices

• Start with high-quality supervised fine-tuning before RLHF—garbage in, garbage out
• Use diverse prompts covering edge cases in your domain during preference collection
• Monitor KL divergence—if too high (>10), model forgets pre-training knowledge
• Test for reward hacking: model exploiting reward model flaws (e.g., excessive politeness)
• Consider DPO as simpler alternative to PPO—no reward model needed, more stable
• Iterate: collect feedback on RLHF model outputs, retrain reward model quarterly

Common Pitfalls

• **Reward hacking**: Model learns to game reward model (output "I'm helpful!" repeatedly)
• **Mode collapse**: Model generates safe but boring outputs to maximize reward
• **Inconsistent labeling**: Different labelers have conflicting preferences
• **Distributional shift**: Reward model fails on novel prompts outside training distribution