Traditionally, fine-tuning language models using reinforcement learning from human feedback (RLHF)⁠ has been the go-to method for ensuring they follow instructions⁠ accurately. OpenAI has been at the forefront of developing these alignment methods to create smarter and safer AI models.

To ensure AI systems behave safely and align with human values, we define desired behaviors and collect human feedback to train a "reward model." This model guides the AI by signaling desirable actions. However, collecting this human feedback for routine and repetitive tasks is often inefficient. Additionally, if our safety policies change, the feedback we've already collected might become outdated, requiring new data.

Thus, we introduce Rule-Based Rewards (RBRs) as a key component of OpenAI’s safety stack to align model behavior with desired safe behavior. Unlike human feedback, RBRs uses clear, simple, and step-by-step rules to evaluate if the model's outputs meet safety standards. When plugged into the standard RLHF pipeline, it helps maintain a good balance between being helpful while preventing harm, to ensure the model behaves safely and effectively without the inefficiencies of recurrent human inputs. We have used RBRs as part of our safety stack since our GPT‑4⁠ launch, including GPT‑4o mini⁠, and we plan to implement it in our models moving forward.

The process of implementing RBRs involves defining a set of propositions—simple statements about the desired or undesired aspects of the model’s responses, such as “being judgmental”, “containing disallowed content”, “referring to safety policies”, “disclaimer” and more. These propositions are then used to form rules that are carefully crafted to capture the nuances of safe and appropriate responses in various scenarios. For instance, a refusal (e.g. “Sorry I can’t help you with that.”) is a desired model response when facing unsafe requests – the associated rules would state that the refusal “should contain a brief apology" and that it "should state an inability to comply".