We believe language is a key ingredient in making reinforcement learning practical and safe for real-world tasks. Previous⁠ work⁠(opens in a new window) on learning models of human preferences has focused on simple simulated environments (Atari games or robotics tasks) which do not capture the complexity of language. Language is also a necessary ingredient for algorithms such as amplification⁠ and debate⁠, which target the reasoning behind preferences.

This work applies human preference learning to several natural language tasks: continuing text with positive sentiment or physically descriptive language using the BookCorpus⁠(opens in a new window), and summarizing content from the TL;DR⁠(opens in a new window) and CNN/Daily Mail⁠(opens in a new window) datasets. Each of these tasks can be viewed as a text completion problem: starting with some text X, we ask what text Y should follow.A

We start with a pretrained language model (the 774M parameter version of GPT‑2⁠) and fine-tune the model by asking human labelers⁠(opens in a new window) which of four samples is best. Fine-tuning for the stylistic continuation tasks is sample efficient: 5,000 human samples suffice for strong performance according to humans. For summarization, models trained with 60,000 comparisons learn to copy whole sentences from the input while skipping irrelevant preamble; this copying is an easy way to ensure accurate summaries, but may exploit the fact that labelers rely on simple heuristics.

For the stylistic continuation tasks, samples comparing the raw 774M GPT‑2 model and our fine-tuned versions are shown below.B

According to the same human labelers used to train them, our fine-tuned models are preferred to the base GPT‑2 model (zero-shot) 88% and 86% of the time for sentiment and descriptiveness, respectively.