Paper: The Mirage of Optimizing Training Policies: Monotonic Inference Policies as the Real Objective fo...

Papers
Page content

Listen to this article.

Problem

Reinforcement learning (RL) is increasingly used to fine-tune large language models (LLMs), but this process can be unstable and prone to failure. The authors identify a core issue: the discrepancy between how an LLM trains versus how it infers. Essentially, the training and inference processes use different “engines,” leading to inconsistencies in probabilities assigned to the same sequences – even though they share parameters. This creates a unique form of off-policy learning that undermines RL training stability. Existing approaches have focused on mitigating this “off-policyness,” but this paper argues they’ve missed a bigger picture: optimizing the training policy doesn’t guarantee an improvement in the inference policy, which is what actually matters for deployment.

Method

The paper introduces a new approach called Monotonic Inference Policy Improvement (MIPI). It proposes a framework called Monotonic Inference Policy Update (MIPU), which uses a two-step process:

  1. Sampler-referenced candidate updates: This step generates potential policy improvements based on how the model samples sequences.
  2. Selective acceptance with an inference-side gap proxy: Only those candidate updates that are demonstrably beneficial to the inference policy (as gauged by a “gap proxy”) are accepted and synchronized.

Results & Limitation

The authors claim that MIPU improves both reasoning performance and training stability when applied to LLMs operating under conditions of high mismatch between training and inference. They demonstrate this across two model scales. However, the abstract doesn’t specify how the “inference-side gap proxy” is defined or calculated. We also don’t know the specifics of their experimental setup beyond “two model scales” and “high mismatch,” making it difficult to judge the generality of their findings from this summary alone.

Why It Matters

This paper’s core argument—that training policy optimization isn’t sufficient for effective LLM RL—is a valuable perspective. If this “monotonic inference policy improvement” objective is genuinely more reliable than existing techniques, MIPU could lead to more stable and predictable LLM fine-tuning processes. This is particularly relevant as the data science community continues to explore RLHF (Reinforcement Learning from Human Feedback) and other RL-based methods for aligning LLMs with desired behaviors. The ability to create more robust training regimes will be critical in scaling these techniques effectively.

References