Debate Training Reduces Reward Hacking in RLAIF

Summary

Paper: Debate Training Reduces Reward Hacking in RLAIF Linkpost for GDM Alignment blogpost Work done by the GDM Amplified Oversight team (we're hiring). TL;DR: When you RL against an LLM judge, the judge gets hacked i.e. fooled into incorrectly giving high reward; adding a debate opponent reduces this. Many of the most impressive capabilities of current AI systems are produced by training on crisp tasks, like math and coding, where task success can be automatically verified. However, much of AI behavior that we actually care about is in some sense fuzzy, even for the most classical crisp tasks. For example, a coding agent should produce maintainable code, not just code that passes tests. More crucially, a coding agent should not learn to pass tests at all costs, especially by subverting the original intent of the user. However, using an LLM judge to provide reward for fuzzy tasks introduces its own issues. Convincing an LLM judge to give high rewards is often easier than solving the task correctly. So reward hacking becomes an even bigger problem. We show that training with debate, where two AIs argue against each to convince a judge, can mitigate reward hacking, potentially providing a hopeful direction for scaling up accurate training supervision for fuzzy tasks. Results Overview We trained LLM policies via debate with training rewards provided by an LLM judge. As a baseline, we directly trained a single LLM policy using LLM judge rewards. All policies were trained on mathematics tasks where answers were available, so that we could accurately measure the effectiveness of our debate protocols. Overall, our results show that directly training a single policy with LLM judge rewards leads to reward hacking: judge reward consistently increases while ground-truth accuracy initially increases but quickly peaks and then decreases. On the other hand, training with debate can mitigate reward hacking: judge rewards increase, and ground truth accuracy increases and then plateaus at a higher peak value than the direct LLM judge case. Debate recovers about 45% of the gap between the peak accuracy of training a single policy with an LLM judge and the peak accuracy of training with the ground truth answers. In the remainder of this post we will explain the motivations behind our setting, including our model of future AI development, the importance of fuzzy tasks, and how debate can help to avoid emergent misalignment arising from RL training. We will further discuss our view of the current limitations of debate training, along with future work that could make more progress in this direction. Debate training with an LLM judge We focus on the case of debate training with an LLM judge. As shown in Figures 2 and 3, the first debater, Alice, proposes a solution to a math problem, and the second debater, Bob, critiques this solution. An LLM judge is shown the full transcript of the debate, and decides whether or not Alice was correct. This decision is directly used as the reinforcement learning training reward. This is compared to the Alice-only baseline, where Alice produces a solution, and the LLM judge directly evaluates it. At deployment time, we only keep Alice’s first response and throw out everything else. That is, Bob, along with any later Alice turns, are used to ensure an accurate training signal only, while we are in the end solely interested in producing the best possible aligned policy for solving the task. This means that we mostly do not care about what precisely Bob is doing in the debate, so long as it results in correct, aligned behavior from Alice’s first turn. The reason for this choice is that we want to be as confident in the correctness and alignment of Alice’s solution as possible, and thus must subject it to the strongest possible critiques that we can find. One must imagine Bob as a highly motivated defense attorney who makes as strong an argument against Alice’s solution as possible, regardless of its correctness. Of course, if the solution contains flaws, this argument will likely be more effective, but Bob’s role is to hunt for such flaws as aggressively as possible. This means that Bob may lie, and this would still be considered the correct operation of the debate protocol: we only care about alignment and correctness of Alice’s first turn. The reason to focus on training, rather than just an inference-time debate scaffold as in some prior work, is that our main objective is in fact to produce the best possible aligned policy for a given task. One could also attempt to use inference-time debate for AI control, but our main focus in this work is alignment training.[1] Why use an LLM judge? One basic reason to use an LLM judge (or a reward model trained on human feedback) is that there is no other practical way to get a reward signal for tasks that are either partially or fully fuzzy by programmatic means. In general, nearly all tasks have at least some fuzzy elements, and many important tasks a