ML//model//GPT//InstructGPT

InstructGPT is OpenAI's GPT-3 aligned to follow instructions through RLHF (2022). Its recipe has three steps: SFT on demonstrations, then a RM trained on human comparisons, then optimization with PPO, the recipe that became the industry standard.

InstructGPT is OpenAI's GPT-3 aligned to follow instructions through RLHF (2022). Its recipe has three steps: SFT on demonstrations, then a RM trained on human comparisons, then optimization with PPO, the recipe that became the industry standard.

It was the bridge from GPT-3 to ChatGPT, and it proved that alignment makes models both safer and more useful. Human raters preferred the 1.3B InstructGPT over the 175B GPT-3: for user satisfaction, alignment beat scale.