The main issue arises from the fact that rewards are uncalibrated. When we get a 0/1 reward for a response to a math/coding prompt, it is unclear how to update the policy. Is this an easy question that the model solved? Is this a hard question the model has no way of solving?

The issue is even more exacerbated with BT reward models for more nuanced tasks like safety/helpfulness where we get a scalar reward in [0, 1]. What does a reward of 0.8 mean for a (prompt, response)? Is this a good/bad response compared to what the model typically generates?

In fact, BT reward distribution could vary significantly for different prompts. Here, you are looking at the helpfulness scores of PaLM-2 for 10 different prompts. Each curve represents one of 10 random prompts with scores calibrated over 100 different responses per prompt. We

When we perform RL, the lack of calibration hurts the performance by uplifting/downlifting responses randomly. The multiple rollouts in GRPO help calibrate the rewards and turning them into a Gaussian distribution where the raw value of the reward now means something similar

In concurrent work, we had also observed a similar phenomenon where instead of calibrating the reward to be N(0, 1), we calibrated the reward to be U[0, 1] with respect to the responses of the reference model, motivated by theoretical findings. In words, a reward of 0.8

This led to a very simple algorithm where the multiple rollouts happened offline prior to training, and the raw values of the reward were recorded so that the reward of any response could be calibrated via an empirical CDF inverse at training time.

The resulting algorithm is a two line change to your favorite standard RL algorithm with the simple inverse CDF transformation of the reward. The rollouts are done offline once and could be used across all epochs, hyperparameter sweeps, etc where we only roll out the model once

This leads to sizable improvement over standard RL and sets a new SOTA compared to various more sophisticated algorithms.

paper link: https://arxiv.org/abs/2412.197... This work was presented at ICML 2025 P.S. You can further transform the reward deliberately to improve the performance suitable for a given test-time scaling algorithm. https://x.com/SZiteng/status/1...

Also see Youssef Mroueh's paper that rigorously studies GRPO with similar conclusions and more: https://arxiv.org/abs/2503.066...

@abeirami Really cool! Does it make sense to also do the same more granularly, taking (eg good) rollouts and choosing points to randomly resample the rollout and then compare them?

@guy_dar1 Generally, engineering the data makes sense; but I am not sure if I understood the proposal. Do you mind elaborating?

@abeirami Thanks for the very clear explanation :)

@abeirami Interesting. Is this consistently reproducible, or are there outlier cases? What are the implications for real-world applications?

@abeirami Thanks for sharing this and relevant work great insights, was thinking along the same lines about reward calibration few days ago.

@abeirami Great sharing! What about GSPO? Group sequence policy optimisation? Thanks

@abeirami If I understand correctly you produce a global calibration and then during training map arbitrary responses to [0,1] nicely, but GRPO performs a "calibration" by comparsing a response to other completions of the same input.

@abeirami Shouldn't we calibrate the inverse CDF every n episodes during the training? Isn't this equivalent to having a frozen critic?

@abeirami Interesting approach. Offline calibration with inverse CDF sounds efficient, especially when dealing with diverse reward distributions across prompts.