Scaling is one of the most impactful concepts in the history of AI research. For large language models (LLMs), scaling has mostly been studied in the context of pretraining, where rigorous scaling laws have allowed us to clearly define the relationship between compute and performance. Inspired by these predictable trends, the LLM research community has empirically validated pretraining scaling laws across several orders of magnitude. Through this process, we have discovered that meaningful improvements in model capabilities can be consistently achieved by investing more data and compute into pretraining.

“The way ML used to work is that people would just tinker with stuff and try to get interesting results. That’s what’s been going on in the past. Then the scaling insight arrived. Scaling laws, GPT-3, and suddenly everyone realized we should scale. This is an example of how language affects thought. Scaling is just one word, but it’s such a powerful word because it informs people what to do.” - Ilya Sutskever

The success of scaling laws in the context of pretraining has inspired the same concept of scaling to be applied in other areas of the LLM training process. Most notably, scaling now plays a key role in reinforcement learning (RL), where researchers have demonstrated smooth and predictable model capability improvements with larger-scale training. In this overview, we will study scaling laws in the context of RL. Rather than studying this topic in isolation, however, we will first build a deep understanding of scaling laws for pretraining and aim to outline how scaling laws have evolved in their application to RL. As we will see, the exact definition of scaling laws is completely different between these two domains, but the fundamental concept of scale remains powerful in both.

Scaling Law Fundamentals

Many early advancements in LLMs were driven by scaling up the pretraining process. Put simply, investing more compute into pretraining—by training a larger model on more data—yields better performance. We can rigorously define the relationship between compute and performance via a scaling law [13], or an equation that models the decrease in an LLM’s test loss as compute increases. As we will see, the pretraining process for an LLM follows smooth trends that can be accurately predicted via a scaling law, allowing the performance of larger models to be estimated before they are even trained. This ability to granularly forecast the expected result of a certain training configuration has many benefits:

• Significant compute investments are less daunting, as we know what the result of this invested compute will be.

• Iteration speed for experiments can be increased by running smaller scale experiments and extrapolating their results.

We will now build an understanding of scaling laws for LLM pretraining from the ground up. This knowledge of the mechanics and practical application of scaling laws for pretraining is needed to form a contrast with the scaling laws used for RL training. Pretraining scaling laws are highly standardized and follow a well-defined approach to estimate very particular training metrics. On the other hand, RL scaling laws—while still informative—tend to be much messier and bespoke, both in terms of their structure and the quantities that we measure.

Further learning. Although we cover the key details of pretraining scaling laws in this section, this is a popular and complex topic with a long history of study. For more details and links to further reading, please see the overview above.

What is a power law?

We can model the LLM pretraining process with a power law. At the simplest level, a power law describes a relationship between two quantities. A basic power law can be expressed as y = a × x^p. The two quantities being studied are x and y, while a and p are constants that describe their relationship—a controls the vertical position of the curve, while p controls the steepness or direction of the curve. Plotting this simple power law function gives us the figure shown below.

We provide the power law plot in both normal and log scale because most papers that study LLM scaling laws tend to plot their results in log scale. However, the plots provided for LLM scaling do not look like the plot shown above—they are usually flipped upside down; see below for an example. This is just an inverse power law, which can be formulated as y = a × (1 / x)^p. This is nearly identical to a standard power law, but we just use a negative exponent for p. As we can see below, using a negative exponent for p flips the power law plot upside down.

LLM power laws. This inverse power law, when plotted with a log scale, yields the signature linear relationship that characterizes LLM scaling laws. The two quantities we model via this inverse power law in LLM pretraining are:

1. The LLM’s test loss L—the next token prediction or cross entropy loss in particular (or another entropy-based metric like bits-per-byte or perplexity)—measured over an in-distribution, held-out validation set.

2. The compute C spent during pretraining that is estimated via the number of training FLOPs C = 6 × N × D, where N is the number of model parameters and D is the number of tokens observed during pretraining.

The factor of six used when estimating training compute comes from the fact that the LLM performs a single forward and backward pass during each training step. A single forward pass costs about 2N FLOPs per token, and the backward pass is 2× the cost of the forward pass. Therefore, a training step costs about 6N FLOPs per token, and we multiply this quantity by the total number of tokens observed during training to yield the C = 6 × N × D approximation. This approximation of pretraining compute was used in one of the first papers to study pretraining scaling laws [13], leading to its adoption in other work on the topic.

Neural Scaling Laws [13] and Chinchilla [14]

To develop a more concrete understanding of scaling laws for pretraining, we will overview two seminal papers [13, 14] that established the foundational principles of scaling. In [13], authors study the impact of several settings on the pretraining process, discovering that performance improves smoothly as we increase:

1. Model parameters.

2. Data volume.

3. Training compute.

More specifically, a power law relationship is observed between each of these factors and the LLM’s test loss when performance is not bottlenecked by either of the other two factors. To observe these power laws, LLMs with sizes up to 1.5B parameters are trained on several subsets of the WebText2 corpus. As shown below, the performance of these models steadily improves as we increase model size, data volume, or compute. These trends span eight orders of magnitude in compute, six orders of magnitude in model size, and two orders of magnitude in dataset size. The exact power law relationships and equations are provided below.

Each of these equations is very similar to the inverse power law equation that we saw before, but we set a = 1 and have an additional multiplicative constant (i.e., C_c, D_c, or N_c) inside of the parenthesis. To fit these power laws, we train a collection of models with different sizes while varying the amount of compute and data used for training. We can then measure the test loss for each of these models, forming a dataset of training configurations with a corresponding test loss. We can then fit the parameters of our power law to this data. Although there are many ways to fit a power law, one common approach for simple power law relationships is to fit a linear model on the observed data in log-log space.

What do power laws tell us? Although the power law plots provided above look promising, we should notice that these plots are generated using a log scale. If we generate normal plots (i.e., without log scale), we get the figures below, where we see that the shape of the power law resembles an exponential decay. In this way, increasing the quality of an LLM becomes exponentially more difficult with scale.

Compute-optimal allocation. The key scaling trends for LLM pretraining were established in [13], where we see that the LLM’s test loss follows smooth power law trends with compute, model parameters, and data volume. One important takeaway from this analysis is that, given a fixed compute budget, we get the best results by training a larger model over less data—usually ending the training process before the model fully converges. Chinchilla [14] builds upon this analysis with an extensive study of optimal compute allocations for pretraining. In particular, the analysis in [14] studies how to optimally allocate a fixed compute budget between model parameters and the number of training tokens to minimize the test loss.

By training over 400 LLMs of varying sizes over different amounts of data, we learn that the scaling recommendations provided in [13] lead most LLMs to be undertrained—training these models on more data would yield better results. More specifically, Chinchilla finds that model and data scale should be increased proportionally for pretraining to be compute optimal; see above. This study is conducted using the same scaling law formulations, but authors explicitly sweep various model and data size combinations under a fixed compute budget to find the optimal balance of data and parameters for minimizing test loss.

Scaling Laws beyond Pretraining

Until recently, most of the compute used for training an LLM was invested into pretraining. We mostly focused on scaling up the pretraining process, while post-training was a less expensive endeavor used to optimize a model’s style and behavior. The advent of reasoning models drastically changed these standards.

Compared to a standard LLM, reasoning models output a long reasoning trace or chain of thought—typically encapsulated by <think> … </think> tokens—before providing a final answer. This idea was popularized by OpenAI’s o-series models, which demonstrated drastic improvements in reasoning capabilities by training models to generate reasoning tokens prior to their final answer. The initial release of o1 highlighted two important new axes of scaling:

1. RL training compute.

2. Inference-time compute.

As shown in the figure below, we observe a smooth increase in performance—resembling a scaling law—by increasing RL training and inference-time compute.

This breakthrough in reasoning was followed by the release of the open-weight DeepSeek-R1 [15] reasoning model, which performed on par with o1 and provided a technical report describing how the model was trained. Today, such reasoning capabilities have become the new standard for both open and closed models.

RL for reasoning. Reasoning models are trained via RL with verifiable rewards. As shown above, model performance improves as we scale up the RL training process. As a result, recent LLM research has heavily focused on scaling RL training for verifiable tasks (e.g., math or coding). Large-scale RL training has unlocked huge improvements in general reasoning capabilities and the quality of coding agents. However, a non-negligible fraction of total training compute is now being spent on RL, and optimally allocating compute for RL is difficult.

For pretraining, we use known scaling laws to reason about how to properly invest available compute—these laws provide a standardized understanding of how performance changes with model size, data, and compute. Given that compute is the primary bottleneck to AI progress, we need analogous scaling laws that enable us to better understand and predict the results of RL training at scale.

Background on Reinforcement Learning

We will soon build upon our understanding of pretraining scaling laws to study RL scaling laws. However, we cannot properly interpret the scaling properties of RL without first understanding the basics of the RL training process. In this section, we will briefly outline the key concepts needed for this discussion, focusing on the GRPO algorithm and its many variants. We focus on GRPO in particular because it is the most common algorithm to use for large-scale RL training with reasoning models—at least in publicly disclosed research. For example, the popular DeepSeek-R1 reasoning model [15] uses GRPO for RL training.

Group Relative Policy Optimization (GRPO) [4]

“We introduce the Group Relative Policy Optimization (GRPO), a variant of Proximal Policy Optimization (PPO). GRPO foregoes the critic model, instead estimating the baseline from group scores, significantly reducing training resources.” - from [4]

Proposed in [4], GRPO is an RL optimization algorithm that builds upon prior algorithms like Proximal Policy Optimization (PPO). Whereas PPO was the most popular RL optimizer for LLMs in the RLHF era, GRPO is now almost universally used for large-scale RL with reasoning models1. GRPO is a simpler and lighter weight optimizer compared to PPO, which has aided its adoption by the LLM research community (especially for open research). The main change made by GRPO relative to PPO is in the advantage estimation technique.

Instead of using a value model and GAE to estimate advantage as in PPO, GRPO estimates the advantage by sampling multiple completions or rollouts (i.e., a “group” of completions) for each prompt in a batch and using their rewards to form a baseline. This group-derived baseline replaces the value function and allows GRPO to not train a value model, thus drastically reducing the memory and compute overhead of RL training. Concretely, the advantage for completion i is computed by normalizing the reward for this completion r_i with the mean and standard deviation of rewards in the group; see below. The same advantage value is assigned to every token t in the completion.

Intuitively, GRPO looks at the relative difference in rewards between multiple completions to the same prompt. The advantage is defined as the delta of one completion’s reward relative to the average reward observed in a group. This approach teaches the model to emphasize completions with higher-than-average reward.

Loss function. Once the advantage has been computed, the loss function used for GRPO is quite similar to that of PPO. The center point of the loss function for both PPO and GRPO is the token-level policy (or importance) ratio. Specifically, this is the ratio of the probability assigned to a token by the current policy and the policy used to generate the rollout (i.e., the “old” policy); see below.

Using this policy ratio and the advantage, we can compute the loss function for GRPO as shown below. This loss function uses the same clipping mechanism proposed by PPO; see here for more details. Similarly to PPO, GRPO takes the minimum of a clipped and unclipped objective in its loss formulation, where the objective is just the product of the policy ratio and advantage for token t.

The inner term of the loss function for GRPO is computed on the token level. By default, we aggregate this loss over our batch by:

1. Averaging the token-level losses within each completion.

2. Averaging completion-level losses over the group.

The exact manner in which we aggregate the loss in GRPO can change, and we will soon see that the manner in which we aggregate loss over a batch can impact performance. Given that GRPO computes advantage based on group-level reward statistics, we must sample a large number of completions per prompt to obtain a reliable advantage estimate. As a result, GRPO usually needs relatively large batch sizes in order for training to be stable2; see here for more details.

GRPO & reward models. GRPO is mostly used in verifiable reward settings without a neural reward model. A common misconception about GRPO is that it eliminates the need for a reward model, but GRPO can be used with or without a reward model. In fact, the original GRPO paper [4] used a reward model instead of verifiable rewards! Removing the reward model is a benefit of verifiable rewards, not an intrinsic benefit of GRPO itself—the primary advantage of GRPO is the elimination of the value model. For more details on GRPO, including example code and a discussion of prior work that led to GRPO, please see the overview below.

Recent GRPO Variants

The GRPO algorithm exploded in popularity after the release of DeepSeek-R1 [15] as many researchers began to replicate or extend results from the paper. Despite details of the model being openly published, fully replicating the training pipeline for DeepSeek-R1 proved non-trivial, leading many subsequent works to propose tweaks to the GRPO algorithm. In this section, we will overview the most successful modifications that are now commonly adopted for better RL training.

Group Sequence Policy Optimization (GSPO) [5] modifies the GRPO objective by computing the policy ratio on a sequence level rather than at the token level. The GRPO loss (shown above) introduces a misalignment between how the model is optimized and how rewards (or advantages) are assigned:

• Advantage is computed at the sequence level (in an outcome reward setting).

• Policy ratios—and the loss in general—are computed at the token level.

As shown in [5], per-token policy ratios tend to have high variance during RL training, which increases the variance of policy gradients and, in turn, leads to training instability. Specifically, the high variance of policy ratios can lead a single token to dominate the loss expression or even cause numerical instability during the RL training process. This problem is particularly acute when training LLMs on long sequences or using large, sparse Mixture-of-Experts models.

To protect against this variance, token-level importance ratios are clipped in the range [1 - ε, 1 + ε]. This clipping operation is formulated such that tokens have zero contribution to the gradient update if they are clipped within the objective. The importance ratio captures the change in a token’s probability after multiple policy updates over the same data—we clip tokens for which we observe a sufficiently large change in their probability. However, simply removing the contribution of these tokens to the policy gradient can be problematic. These can be rare (low probability) tokens that are identified as important by the policy update. Such tokens may capture key reasoning steps that the model needs to learn, but we suppress the learning process via the clipping operation.

The key idea of GSPO is to compute importance ratios for the sequence rather than each token. Once we have derived the sequence-level importance ratio, the GSPO training objective is almost identical to that of GRPO; see above. We apply clipping to the sequence-level importance ratio, use the same advantage, and take a minimum of clipped and unclipped objectives at the sequence level.

The sequence-level importance ratio can be derived by factorizing the probability of a sequence into a product of individual token probabilities. However, authors in [5] choose to define the sequence-level importance ratio using the logarithmic form of a geometric mean, which is defined as shown below. This geometric mean is taken over token-level probabilities, which normalizes the sequence-level policy ratio by the length of the sequence. By using this approach, we ensure that importance ratios for sequences of different lengths are comparable, as well as improve numerical stability—especially for long sequences—by formulating the ratio as a sum over logprobs instead of a product over raw probabilities.

We see in [5] that GSPO improves training stability, sample efficiency, and overall performance. The stability of GSPO is found to be especially useful when training large MoE models, such as Qwen3-235B-A22B. For these reasons, GSPO was adopted in the training process for the popular Qwen 3 model series.

Dynamic Sampling Policy Optimization (DAPO) [6] is not a single algorithm, but rather a modified recipe that proposes several useful tweaks to the vanilla GRPO optimizer. We see in [6] that the vanilla GRPO optimizer suffers from notable issues like:

• Entropy collapse: the entropy of the model’s next token distribution collapses during the training process. Probability mass is primarily assigned to a single token and outputs are more deterministic.

• Reward noise: the training reward is very noisy and does not steadily increase during the RL training process.

• Training instability: the training process is unstable and may diverge.

To solve these issues, authors in [6] propose a suite of tricks that can be used in tandem. First, the entropy collapse problem in GRPO is shown to be caused by the fact that clipping emphasizes high probability tokens and punishes low probability (exploratory) tokens. The “clip higher” approach is proposed in [6] to solve this issue by decoupling lower and upper clipping bounds. Specifically, we clip in the range [1-ε_low, 1+ε_high], where ε_low=0.2 (default setting in GRPO) and ε_high=0.28 in [6]. Increasing ε_high prevents entropy collapse and improves overall GRPO performance; see below.

As RL training progresses, the number of samples for which all completions in a group are accurate increases. Such groups have zero advantage and, in turn, no impact on the policy gradient. As a result, these groups effectively reduce the batch size in GRPO, leading to noisier gradient estimates and degraded sample efficiency. Dynamic sampling is proposed in [6] to solve this problem by:

1. Filtering all prompts with perfect accuracy from a batch.

2. Continuing to sample prompts until we have a full batch.

This approach can increase the cost of constructing a batch, as we dynamically continue sampling prompts until the batch is full. However, we see in [6] that this cost is offset by the improved sample efficiency of RL training; see below.

Finally, DAPO also proposes a modified loss aggregation strategy and a new approach for handling completions that exceed the maximum sequence length. Vanilla GRPO aggregates token-level losses by i) computing the average loss in each sequence and ii) averaging sequence-level losses in the batch. However, this approach introduces a subtle bias—tokens within longer sequences have relatively less contribution to the overall batch gradient. To solve this, DAPO computes a token-level loss that is simply averaged over all tokens in the batch; see below.

Additionally, a length-based penalty term is introduced to the reward to apply a “soft” punishment to completions that are too long. Instead of assigning a hard negative reward to any completion that exceeds the maximum sequence length, authors in [6] argue that we should slowly increase the overlong penalty to its maximum value as we approach the maximum sequence length. This approach provides a smooth length penalty from which the model can effectively learn.

GRPO Done Right (Dr. GRPO) [7] outlined two key sources of bias that exist in the vanilla GRPO algorithm (depicted above):

1. Response-level length bias: GRPO normalizes the summed loss of tokens in each sequence by the total number of tokens in that sequence, leading to biased gradient updates based on the length of each response.

2. Question-level difficulty biases: the standard deviation term in the denominator of the advantage formulation in GRPO causes the advantage to become very large for questions that are either too easy (i.e., most responses have a reward of one) or too hard (i.e., most responses have a reward of zero).

To solve the first bias, Dr. GRPO aggregates the loss by summing token-level losses in a sequence and dividing this sum by a fixed constant MAX_TOKENS, thus removing response length from the aggregation process. The difference between this loss aggregation strategy and that of DAPO is nuanced. In DAPO, each token in the batch has an equal contribution to the gradient. As a result, DAPO still places more emphasis upon longer sequences in a batch, as these sequences have a larger ratio of total tokens (even if all tokens are weighted equally across the batch). On the other hand, replacing the sequence-level average with division by a fixed constant in Dr. GRPO effectively decouples aggregation from response lengths and, in turn, protects against length-based optimization bias.

The question-level difficulty bias is handled by removing the standard deviation term from the advantage estimator; see below. By making these two changes, Dr. GRPO improves training stability and efficiency, while making the resulting model more token efficient (i.e., responses are not artificially long).

Truncated Importance Sampling (TIS) [9] attempts to address mismatches in token probabilities introduced by efficient RL training frameworks. As we know, there are two main operations that occur during RL training: i) sampling rollouts and ii) computing policy updates. In modern RL frameworks, these operations are usually handled via separate engines:

• Optimized inference engines like vLLM or SGLang—often with lower precision inference (e.g., int8 or fp8) for extra efficiency—are used to generate rollouts.

• Distributed training frameworks like FSDP or DeepSpeed are used to compute policy updates.

Given that generating rollouts consumes the majority of compute during RL3, this approach is usually necessary—we want the inference process to be as efficient as possible. However, the use of separate engines can also introduce non-negligible differences in the token probabilities produced by each engine; see below.

Additionally, this difference in token probabilities is not easy to fix by simply standardizing implementations across engines. Authors in [9] investigate several code interventions to decrease the gap in token probabilities with little success, and this process would have to be repeated for every combination of engines used for RL training. Instead, a more flexible approach is proposed in [9] that uses an importance sampling term to automatically correct for this engine mismatch during RL within the policy gradient expression. The exact expression is shown below and is formulated as a REINFORCE-style policy update.

The above expression explicitly uses a different engine for sampling rollouts (sampler) and computing policy updates (learner). The importance ratio between these two engines is simply the quotient of probabilities from learner and sampler engines. In [9], authors truncate this importance ratio by capping it at a maximum size of ρ. Compared to the clipping operation used in PPO or GRPO, this truncation operation has a few differences:

• We are directly truncating the importance ratio. Clipping is also applied to the importance ratio, but it is followed by a minimum of clipped and unclipped objectives, making the operation two-sided.

• We truncate the importance ratio with a maximum value of ρ, a one-sided truncation that simply prevents extreme up-weighting.

However, the practical application of TIS is quite simple—we just compute the truncated importance ratio and multiply our policy gradient expression by this ratio. As shown below, including this importance ratio in the policy gradient has a huge impact on RL training stability and model performance, leading to quick adoption of TIS in popular training frameworks (e.g., verl and OpenInstruct).

We formulated TIS above using a sequence-level importance ratio with REINFORCE. However, we can also create a token-level formulation with PPO or GRPO; see below. As we can see, the truncated importance ratio is computed in addition to the other components of the PPO-style policy gradient expression. We then multiply the existing expression by this correction term, and this can be done either at the sequence level—similarly to the gradient expression used by GSPO—or at a token level—as in the normal expression for PPO or GRPO.

Token-level TIS is commonly used in practice due to aligning well with PPO and GRPO objectives and is, therefore, relatively simple to integrate into existing training frameworks in addition to offering stability benefits. In recent work [10], however, authors have argued from an analytical perspective that sequence-level TIS is less biased compared to token-level TIS. Currently, there is no clear consensus on which of these approaches is superior. The best implementation in practice may differ depending upon the setup or domain being considered.

Clipped Importance Sampling-Weight Policy Optimization (CISPO) [8] is another recent RL variant that, similarly to TIS, builds upon a REINFORCE-style objective with an added importance ratio. When using a PPO-style clipping approach, we know any token that is clipped from the objective has no contribution to the policy gradient. In [10], authors observe empirically that the important “fork” tokens in the model’s reasoning trace (e.g., “aha” or “wait”) are rare and are initially assigned low probabilities in the base model. Due to the importance of these tokens, their probability usually increases drastically after the first policy update, leading these tokens to have a very large importance ratio—that is then clipped by the PPO objective—for subsequent policy updates.

As a result, important fork tokens are usually masked from the PPO-style loss after the first policy update for a batch of data. Although this masking may not always be an issue (i.e., most standard RL setups perform only ~2-4 updates on each batch of sampled data), MiniMax-M1 performs 16 policy updates for each batch of data. Therefore, important tokens being masked out of the loss after only one or a few updates can significantly damage training efficiency. To solve this issue, authors adopt the modified REINFORCE-style loss shown below. As we can see, CISPO adopts some of the recommendations proposed by DAPO [6] as well, including the token-level loss formulation to correct for length biases.

This loss formulation applies a stop gradient to the clipped importance ratio, ensuring that each token contributes to the loss even when it is clipped. Put differently, the importance ratio is used as a weight that controls the contribution of a token to the policy gradient. Clipping in CISPO puts a cap on this weight, ensuring no single token is over-amplified due to a large importance ratio.

When we look at the GRPO objective, the clipping mechanics are quite different; see above. In particular, token probabilities are only present in the importance ratio, and the gradient flows through the token probability terms inside the importance ratio. When the importance ratio is clipped, the gradient for that token is zero and there is no contribution to the policy gradient. The modified clipping approach used by CISPO ensures that all tokens contribute to the policy gradient, improving the stability and efficiency of RL; see below.

The loss formulation of CISPO and TIS look quite similar, but these algorithms—despite both using an importance ratio—aim to solve different issues. CISPO uses the same definition of the importance ratio adopted in PPO and GRPO. This importance ratio is clipped to ensure that token probabilities do not change too much over a single batch of data, thus enforcing a trust region. CISPO simply modifies the manner in which the importance ratio is clipped to ensure that all tokens continue contributing to the policy gradient (with a capped weight) even if they are clipped. On the other hand, TIS uses an importance ratio to capture the difference in token probabilities between training and inference engines, thus correcting for the mismatch between engines during RL training.

Further reading. For more details on each of these algorithms, please see the overview linked below, which covers many GRPO variants and modifications.

Regularization for RL

There are two primary regularization terms commonly added to RL training:

1. Entropy bonus: rewards the LLM for remaining uncertain and helps to avoid overly-confident token distributions.

2. KL divergence: anchors the policy to a reference policy throughout training to prevent the LLM from changing too much.

Regularization terms are less commonly used in recent RL training pipelines, but we will see examples of both strategies being applied later in the overview. To avoid future confusion, we will briefly explain each regularization strategy now.

KL divergence. During RL training, we can compute the Kullback-Leibler (KL) Divergence between the current policy and a reference policy—usually the policy from before RL training begins (i.e., the base model). There are several techniques that can be used to approximate the KL divergence between two models; see here. The easiest—and most common—approximation of KL divergence [7] is the difference in token-level log probabilities between the current policy and the reference policy. This approximation and another common variant used in the original GRPO paper [12]4 are outlined below. Both estimators are usually supported in open RL implementations; e.g., see their implementation in TRL.

After the KL divergence has been computed, there are two common ways that it can be incorporated into the RL training process:

1. By directly subtracting the KL divergence from the reward.

2. By adding the KL divergence to the loss function as a penalty term.

Both of these approaches can be found in practice depending on the RL optimizer—or exact implementation—being used. PPO incorporates KL divergence into the reward, while GRPO adds it as a penalty to the objective function; see below.

Due to the popularity of GRPO, recent RL implementations more often include the KL divergence in the loss, but completely omitting the KL divergence—and not using any regularization—is becoming increasingly common. During training, the KL divergence term penalizes our policy from becoming too different from the reference policy, but drifting away from the reference policy may not be negative if we are performing large-scale, reasoning-oriented RL training.

Entropy bonus. From an information theory perspective, entropy captures the level of uncertainty associated with the possible states for a variable:

• High entropy: probability mass is spread across many outcomes.

• Low entropy: probability mass is concentrated on a few outcomes.

In the LLM domain, we can measure the entropy of a model’s token distribution—low entropy means that the LLM places most of its probability into a small set of tokens and vice versa. Specifically, we can compute entropy using the equation below.

Usually, entropy is computed for each token (i.e., at each decoding step) and then averaged across the generated trajectory. After computing the entropy, we can turn it into an entropy bonus and use it as a regularization term by simply scaling it with a coefficient β and incorporating it into either the reward—this is done in the original PPO paper—or the objective function. The purpose of the entropy bonus is to prevent the LLM from becoming overly confident in its token distribution and, in turn, avoid entropy collapse that prevents the policy from exploring during training. Similarly to the KL divergence, entropy bonuses are now more commonly incorporated into the loss function. In fact, we will soon study a paper that adds an entropy bonus to the GRPO loss [3].

Scaling the RL Training Process

“While RL compute for LLMs has scaled massively, our understanding of how to scale RL has not kept pace; the methodology remains more art than science.” - from [1]

Scaling laws provide researchers with the ability to extrapolate the performance of expensive training runs from those that require less compute. Despite the expanding role of RL in training frontier models, however, our understanding its fundamental scaling properties remain somewhat rudimentary, especially relative to pretraining. In this section, we will take a look at several notable papers that are trying to solve this issue. As we will see, RL scaling laws are very different from those used for pretraining, and many of these differences arise from the massive design space of RL training. Put simply, RL is complicated, and we are far from a single standardized approach for handling RL “correctly”. However, there are still useful scaling insights that can be gleaned from this work that will help us to allocate available compute for RL experiments more effectively.

The Art of Scaling Reinforcement Learning Compute for LLMs [1]

Unlike pretraining, RL has no established predictive scaling laws for reliably estimating performance trends. Best practices for RL are found in new algorithm proposals, but these findings may not generalize at scale. Model reports also frequently provide practical recommendations for RL training, but these methods are often anecdotal and dependent upon training settings. As a result, we must test RL design choices the hard way—by running large-scale experiments and seeing what works. Given the computation cost of modern RL, this approach is a major bottleneck that limits iteration speed and hinders technical progress. We need a standardized approach to identify strong RL candidates at smaller scales.

RL scaling. In [1], authors model the RL training process with sigmoidal compute-performance curves. We fit such a curve separately for each RL training run to model the relationship between expected reward—calculated over a validation set at regular intervals during training—and compute (in units of GPU hours); see below.

This curve models the relationship between two quantities:

1. Reward gain: the difference between the reward after RL training with compute C and the initial reward before RL training.

2. Asymptotic reward ceiling: the maximum possible gain in reward we can achieve by spending unlimited compute on RL training.

The relationship between these quantities is controlled by the term 1 / [1 + (C_mid​/C)^B​]. This term includes i) the compute level at which we reach the midpoint of the curve C_mid, ii) an efficiency exponent B for the steepness of the curve, and iii) the current compute level C. Intuitively, this term captures how much of the total possible performance gain has been unlocked by running RL with compute C. The shape of this curve is visualized in the figure below.

According to this structure, RL training is flat in terms of reward during the early phase of training, then undergoes a phase of fast improvement before reaching a plateau. Authors find in [1] that these saturating curves between compute and reward robustly model the RL training process in practice. As we will see, this structure has also been validated and adopted by other work on RL scaling.

We fit this curve to the results of each RL training run, allowing us to compare results obtained from multiple runs with different training setups. There are two ways that these scaling curves may differ:

1. Their value of A may be different, indicating that one training setting achieves better asymptotic performance.

2. Their value of B (or C_mid) may be different, meaning that one training setting is more compute efficient than the other.

However, not all training settings yield a benefit in both A and B. In such cases, authors prioritize asymptotic improvements over efficiency improvements, arguing that improving the model’s asymptotic performance is more valuable because a degradation in efficiency can be offset by just training for longer.

Applying RL scaling laws. The RL scaling laws proposed in [1] allow us to extrapolate the performance of a training run without incurring the full cost. We can use the early phase of training to predict what would be the final performance after training with more compute. The scaling law is fit using validation performance measured at regular intervals during training5, allowing us to efficiently assess the scalability of different changes to the RL training process.

Authors in [1] use this approach to derive an optimal training recipe, called ScaleRL. Beginning with a baseline setup, authors test interventions to the RL training process in multiple phases of increasing scale—4K, 8K, 16K, and 100K GPU hours. In each phase, scaling laws are fit to extrapolate the performance of each setting, allowing authors to both i) verify the accuracy of their scaling law formulation and ii) efficiently discover scalable design choices for RL.

“We present the first large-scale systematic study, amounting to more than 400,000 GPU-hours, that defines a principled framework for analyzing and predicting RL scaling in LLMs.” - from [1]

Baseline RL setup. RL experiments in [1] primarily use the Polaris-53K math-focused reasoning dataset. Analysis begins with a baseline RL recipe that uses the GRPO loss with no KL divergence and the clip higher approach from DAPO [6]. All models produce a reasoning trace before their final output. A context length of 16K tokens is used—12K reasoning tokens, 2K input tokens, and 2K output tokens—as well as a batch size of 768—a total of 48 prompts with 16 rollouts each.

To enforce the 12K-token reasoning budget, interruptions are used during training. When a reasoning trace reaches 12K tokens, we append a static end-of-reasoning phrase “Okay, time is up. Let me stop thinking and formulate a final answer. </think>" to the model’s output so that the model can stop reasoning and begin generating a final answer.

From an engineering perspective, authors adopt a split generator-trainer approach, where a subset of GPUs run optimized inference engines (e.g., vLLM) for generating rollouts and remaining GPUs run a training backend (e.g., FSDP) to update policy parameters. To improve efficiency, an asynchronous RL training approach is adopted. Two algorithms are considered: asynchronous PPO and PipelineRL [11]. As shown above, both approaches achieve similar asymptotic performance A, but PipelineRL has significantly improved compute efficiency B due to its design principles that minimize GPU idle time (e.g., in-flight weight updates). Authors also find in [1] that it is important to bound the degree of asynchrony by ensuring policy updates are not more than K steps ahead of the generated rollouts. K = 8 is found to be optimal for this setting; see above.

Ablating RL modifications. To build upon the baseline RL recipe, authors run small-scale RL experiments (i.e., ~4-8K GPU hours) to test the impact of various RL design choices. The baseline loss is first compared with both GSPO and CISPO loss formulations; see below. We see that both GSPO and CISPO noticeably outperform the baseline formulation in terms of asymptotic performance. CISPO also has marginally improved compute efficiency relative to GSPO, leading authors to use CISPO for remaining experiments in [1].

As we learned from the explanation of TIS, using different engines for generating rollouts and computing policy updates can lead to a non-negligible mismatch in log probabilities between training and inference. One change that we can make to minimize this mismatch is using full (float32) precision in the LLM’s language modeling head—the final linear layer that predicts token probabilities. As shown above, using a full precision head in training and inference engines significantly improves both asymptotic performance and compute efficiency. We should note, however, that authors do not adopt any approach for correcting the trainer-generator mismatch (e.g., TIS), which could also help to solve this issue.

Authors also test different loss aggregation strategies, including vanilla GRPO aggregation versus DAPO-style aggregation, finding that the loss aggregation proposed by DAPO [6] tends to perform the best; see below. In a similar vein, several advantage normalization techniques are tested. Specifically, authors test dividing mean-centered rewards by the standard deviation of rewards in a group—as in vanilla GRPO—or the standard deviation of rewards in the entire batch, as well as not dividing the mean-centered reward by anything—as in Dr. GRPO [7]. All techniques perform comparably, indicating that advantage normalization does not significantly impact asymptotic performance; see below. Remaining experiments normalize the advantage using the standard deviation of rewards across the batch due to the slight boost observed in asymptotic performance.

Authors also discover data curation and filtering strategies that benefit the asymptotic performance of RL. Prompts with zero variance in rewards across a group have zero advantage and, therefore, no contribution to the policy gradient. Filtering these zero variance prompts from the batch benefits asymptotic performance; see below. Notably, this approach is different from the dynamic sampling method proposed in DAPO [6], as we do not continue sampling prompts until the batch is full. Rather, we just filter zero-variance prompts from the batch, forming a smaller effective batch. By doing this, we avoid dampening the policy gradient signal, as we are averaging the policy gradient over a smaller effective batch instead of the full batch that includes prompts with no gradient.

Many data curriculum strategies have been proposed for RL training, but we learn in [1] that simple approaches can be quite effective. During training, the number of prompts that are solved easily by the current policy increases, and these prompts usually remain easy for the model throughout the rest of training. As shown above, dynamically removing these prompts from the training process improves asymptotic performance. To do this, authors maintain a history of pass rates for each prompt and permanently remove prompts that exceed a pass rate of 90%. This approach, called no positive resampling in [1], avoids wasting compute on prompts that the model already knows how to correctly solve.

The ScaleRL recipe, which combines all best practices identified in the smaller-scale experiments outlined above, uses the loss formulation shown above. As mentioned before, a PipelineRL setup, forced interruptions for reasoning, and a full precision language modeling head are also used for ScaleRL.

To validate this recipe, larger-scale experiments are performed with up to 16K GPU hours. Authors perform leave-one-out ablations by removing individual components of the ScaleRL recipe to determine if they still have an impact when used in tandem with other components. When fitting sigmoidal scaling curves up to 8K GPU hours, we see that extrapolated results accurately predict performance up to the end of the 16K GPU hour run. In these experiments, the full ScaleRL recipe is found to yield the best performance; see above. Not all components significantly benefit performance in the leave-one-out analysis, but authors argue that these design choices still tend to benefit training stability.

“Even when individual design choices appear redundant within the combined recipe, they often enhance training stability, robustness, or efficiency in ways that generalize across models and setups. ScaleRL retains such components not just for marginal gains in a specific configuration, but because they address recurring sources of instability and variance that arise across RL regimes.” - from [1]

Scaling up. Based on the analysis in [1], authors perform a final training run of ScaleRL up to 100K GPU hours, finding that the extrapolated performance continues to match actual performance in extended RL training runs; see below. Prior to this large-scale experiment, different methods for scaling up the RL training process (e.g., longer context, larger batch size, larger models, etc.) are considered in [1]. By analyzing these options in this extended run of ScaleRL, we learn the following:

• The scaling laws proposed in [1] are also found to accurately extrapolate the performance of Mixture-of-Experts models, indicating generalizability to larger models with different architectures.

• Using a longer context window during RL slows down training progress initially but yields higher asymptotic performance in the long run.

• Increasing the batch size improves the asymptotic performance of RL and prevents stagnation on downstream benchmarks.

• How we allocate the batch in terms of number of prompts and number of rollouts per prompt is less impactful—the total batch size matters most.

Key takeaways. The empirical analysis in [1] is extensive and contains a wide variety of practical details that are incredibly useful for those working on RL. For this reason, those who are interested in gaining a practical grasp of RL training should definitely read the full paper. However, the comprehensive empirical analysis presented in [1] can be largely summarized as follows:

• Asynchronous RL (i.e., PipelineRL) with a split generator-trainer setup is highly efficient and yields models that perform well, so long as we bound the level of asynchronicity during training.

• The proposed ScaleRL training recipe combines all of the practical GRPO modifications that were found to be useful across experiments in [1].

• The performance ceiling of RL A can be impacted by changes to the RL setup (e.g., loss type or batch size). However, many common RL interventions (e.g., loss aggregation, data curriculum, or advantage normalization) impact compute efficiency B rather than asymptotic performance.

• The methods that appear superior in smaller-scale RL runs do not always generalize to the high-compute regime. However, we can still identify the recipes that are most scalable by fitting a sigmoidal scaling curve and estimating scaling parameters A and B from early training dynamics. This approach is used constantly throughout the analysis in [1] to judge the scalability of RL recipes without performing full training runs (i.e., 16K-100K GPU hours).

Scaling Behaviors of LLM RL Post-Training [2]

In [2], authors investigate scaling behaviors of RL post-training using the full Qwen-2.5 model suite—both base and instruct models—ranging from 0.5B to 72B parameters. As in [1], this paper studies the impact of factors like model size, data volume, and compute on the performance of models trained with RL. However, this analysis focuses specifically on the mathematical reasoning domain, uses only the vanilla GRPO algorithm, and adopts a different scaling formulation. From the analysis in [2], we learn that RL follows a predictive power-law relationship between test loss and compute or data; see above.

Scaling formulation. The scaling law formulation in [2] fits a relationship between test loss—defined as the error rate (i.e., error rate = 1 - accuracy) on an in-domain validation set—and compute or data. As shown below, RL scaling behavior is modeled using a log-linear power law between the test loss L, model size N, and a resource budget X. Here, the resource budget can either be the amount of compute C or the amount of data D used during RL training.

In the figure below, we plot this power law—using both log-log scale and linear scale to make interpreting the plots easier—for different values of the learning efficiency. We use a fixed value of E(N) = 1.0 in this plot for simplicity. As we can see, performance improves log-linearly as the resource budget X increases, and higher learning efficiency K(N) leads to a steeper decrease in test loss.

Performance extrapolation. As we might have inferred, the scaling formulation used in [1] is quite different from the pretraining scaling laws that we learned about before. More specifically, the scaling trends in [1] can only extrapolate the results of a specific training run to a higher compute regime—we are predicting what will happen if we continue the RL training process for longer. In contrast, the power law in [2] enables multiple extrapolation regimes:

1. Inter-model: fit the scaling law using data from training runs with smaller models (i.e., 0.5B to 32B Qwen-2.5 models) and predict the performance of a larger model (i.e., Qwen-2.5-72B).

2. Intra-model: fit the scaling law using the early training trajectory of a model and predict its performance for the remainder of training.

Both kinds of extrapolation are validated in [2] across base and instruct model variants of several sizes, demonstrating that RL training follows predictable scaling trends across model size N, compute C, and data volume D. Scaling plots shown in [2] always provide both inter and intra-model extrapolation results.

More on learning efficiency. In our above scaling law expression, we should notice that the learning efficiency term has a dependence upon N—learning efficiency follows a saturating trend with model size. Put simply, this means that i) larger models have higher learning efficiency, but ii) marginal efficiency gains begin to diminish with increasing model size. As shown in the plot below, this formulation matches empirical observations. In practice, learning efficiency follows a saturating S-curve—similar in structure to the scaling law formulation proposed in [1]6—that plateaus at a maximum learning efficiency of K_max.

Experimental setup. All experiments in [2] use vanilla GRPO—with a KL divergence term—and the verl training framework. Scaling laws are empirically fit and validated on results from over 60 models, including base and instruct variants from the Qwen-2.5 series with sizes ranging from 0.5B to 72B parameters. The model family is fixed to ensure that only parameter count N and data volume D are changing. RL training is conducted over 50K samples taken from the mathematics subset of guru-RL-92K, which performs extensive deduplication and difficulty filtering. Additionally, authors in [2] sort problems by increasing difficulty—as assessed by the pass rate of Qwen2.5-7B-Instruct—to form a data curriculum of increasing problem difficulty as RL training progresses. Following standard practice for fitting scaling laws, we compute test loss on an in-domain dataset of 500 held-out problems sampled from the training distribution.

Compute-constrained regime. The power law formulation in [2] can be used to characterize scaling behavior under a fixed compute budget. Given a compute budget C, we are interested in the optimal model size N that minimizes the test loss. In [2], the compute budget is estimated via cumulative training FLOPs C = 6 × N × T, where T is the number of tokens processed during training. T is related to the data volume D, but whereas D counts data samples, T measures the total token volume. T is inferred from fixed values of C and N. We can study the relationship between compute C and model size N by running RL training with various model sizes and compute budgets, then fitting scaling laws on the results. We use compute as our resource budget (i.e., X = C) for these scaling laws; see below.

In these plots, we can observe the results of inter-model (top plot) and intra-model (bottom plot) extrapolation using the scaling law formulation proposed in [2]. When studying scaling trends for smaller models (i.e., 0.5B to 32B parameters), we see that the best performance under a fixed compute budget is usually achieved by using the largest model. Larger models (i.e., 32B and 72B parameters) violate this trend: the 32B model performs best at lower compute budgets, but a crossover occurs at higher compute budgets after which the 72B model performs better.

This crossover arises from the fact that learning efficiency k(N) saturates for larger models. Given a fixed compute budget C, a smaller model can train for a larger number of steps relative to a larger model. Therefore, the larger model must have significantly improved learning efficiency in order to outperform the smaller model. As we see in the scaling analysis above, this is true until we reach 72B scale, at which point efficiency gains saturate and the 32B model is able to exceed the performance of the larger model under tight compute constraints.

Data-optimal scaling. Given that LLM training is usually bottlenecked by the availability of high-quality data, we also want to understand the optimal model size for RL training given a fixed data budget D7. To do this, we can train models with various model sizes N and data budgets D, then fit the scaling laws—where data is our resource budget (i.e., X = D)—to these results; see below. The conclusion from this analysis is simple: for a fixed amount of data, larger models demonstrate superior sample efficiency and consistently achieve lower test loss. We also see that scaling laws accurately extrapolate performance in all regimes considered.

If we remove data and compute constraints (i.e., train models to convergence on sufficiently large datasets), test loss monotonically decreases with model size—bigger models are better given enough data and compute. However, this trend does not follow a power law; see below. Smaller models show weaker gains, indicating diminishing returns for training smaller models to convergence.

Data reuse. In addition to the scaling analysis, authors in [2] test if repeating data during training is problematic. These experiments fix the total data budget D_total but vary the number of unique data samples such that D_total = τ × D_unique, where τ is a data reuse factor. As shown below, we learn in [2] that performance is primarily determined by D_total rather than D_unique. In fact, test loss is relatively insensitive to τ, and we see that there is no significant degradation in performance until larger reuse factors (i.e., τ = 25).

However, unique data is not sampled randomly in these experiments. To ensure that data subsets are sufficiently diverse, authors partition the training set into difficulty subsets and preserve the data difficulty distribution across subsets of different sizes. Additionally, the same data curriculum is maintained by ordering data based upon difficulty. The robustness of RL training to data reuse is likely dependent upon the diversity, quality, and difficulty of the unique samples.

Optimally Scaling Sampling Compute for LLM RL [3]

Scaling laws can be applied to RL training in many ways. The work we have seen so far shows that performance during RL follows a sigmoidal trajectory [1] and scales in a predictable manner with model size under fixed compute budgets [2]. However, these results, while informative, do not directly recommend how we can practically allocate a fixed compute budget for RL training in a similar manner to pretraining scaling laws. Inspired by this, authors in [3] perform a prescriptive analysis of optimal compute allocations for RL. Specifically, the analysis in [3] focuses on understanding how to optimally allocate sampling compute—or the amount of compute spent generating completions for on-policy RL.

“We study the compute-optimal allocation of sampling compute for on-policy RL methods in LLMs, framing scaling as a compute-constrained optimization over three resources: parallel rollouts per problem, number of problems per batch, and number of update steps.” - from [3]

Sampling compute. The relationship between compute and performance is less straightforward in RL relative to pretraining. For both pretraining and RL, the training process involves a sequence of training—or model update—steps. At each pretraining step, a single forward and backward pass is performed. On the other hand, an RL training step includes multiple components:

• Data collection: sampling completions from the current policy.

• Optimization: updating the policy over collected data.

With this in mind, we can model the total compute cost of an RL training run as C = B_p​ × n × M, where B_p is the number of unique prompts per batch, n is the number of rollouts generated per prompt, and M is the number of steps taken during RL training. The analysis in [3] primarily focuses on compute spent on sampling completions (B_p and n) rather than sequential training steps (M).

Scaling laws for sampling. Given the compute footprint for RL outlined above, our goal is to better understand how varying the allocation of a fixed compute budget C_0 across the three factors B_p, n, and M impacts model performance. The scaling analysis is conducted in [3] by sweeping over settings of B_p ∈ {2^5, 2^6, …, 2^10} and n ∈ {2^3, 2^4, …, 2^11}, where both of these parameters are uniformly sampled via grid search on a log scale. Due to hardware constraints, a maximum effective batch size (B_p​ × n ≤ B_max) is also enforced.

From a given (B_p, n) setting, we can perform a single RL training run to capture increasing settings of M throughout training. Following scaling law best practices, model performance is evaluated during training by measuring reward on an in-domain validation set. The evaluation results in a training run are sub-sampled to only include record-breaking points—defined as points in the reward curve that exceed all prior rewards—along the learning trajectory; see below. By considering only record-breaking points when modeling an RL training run, we fit our scaling law to the frontier of the reward trajectory for this run.

To robustly identify record-breaking points in the reward trajectory, rewards are separated into discrete bins, and the first point at which the reward enters a new bin is selected as the record-breaking point. Once the cleaned reward trajectory is available, we model this single training run with a sigmoidal scaling law—similar to the scaling law formulation used in [1]. This gives us a collection of scaling laws for RL training curves with different settings of B_p and n, allowing the optimal configuration to be identified at each compute level from run-specific curves.

Building on these run-specific scaling laws, we can also fit a scaling law on the optimal settings identified for each compute level. Namely, we can use a similar sigmoidal scaling law to model how the optimal value of n varies according to our compute budget C, allowing us to extrapolate optimal training settings at higher compute budgets. In theory, the same approach can be used for B_p, but no clear pattern is observed in practice for the optimal value of B_p.

Experimental settings. The scaling analysis described above is conducted using Qwen2.5-7B-Instruct, Qwen3-4B-Instruct, and Llama3.1-8B-Instruct as base models. All RL training runs use binary outcome rewards and the vanilla GRPO optimizer. Guru-Math is used as the primary dataset and is split into easy and hard subsets by assessing the difficulty of each prompt—judged by the accuracy of the base model over 16 rollouts (i.e., Avg@16). The difficulty distribution is shown below with the easy and hard subsets shaded in blue and orange, respectively. Empirical scaling analysis is performed separately on both easy and hard data subsets in [3] to observe how difficulty distributions impact trends in scaling.

In order to make the RL training process stable, the correct regularization strategy is needed. Interestingly, we see in [3] that optimal regularization is difficulty-dependent. Authors consider adding both a KL divergence and entropy bonus to the RL training objective. On easy problems, the entropy bonus helps to prevent premature entropy collapse in the policy. However, using an entropy bonus on difficult problems can actually cause an entropy explosion by pushing the policy towards rare but successful reasoning trajectories, making it better to remove regularization entirely. As shown below, the following regularization strategy is found to yield the most stable results8:

• Apply both the entropy bonus and KL divergence—which helps to delay entropy explosion—in tandem when training on the easy dataset.

• Use no regularization when training on the hard dataset.

In addition to difficulty-dependent regularization, the learning rate must be increased with the batch size to ensure stable training. In particular, a square root scaling rule is used for the learning rate in [3], which increases the learning rate proportionally to the square root of the batch size; see below.

How should we allocate compute? The primary takeaway from the scaling analysis in [3] focuses upon the number of rollouts to sample (n) for each prompt in a batch. As the compute budget increases, the optimal setting of n increases as well, eventually saturating at higher compute budgets; see below. In other words, allocating increased compute towards sampling more rollouts per prompt yields better results compared to just training the model for longer. Interestingly, the exact scaling law also depends on the problem difficulty—smaller optimal values of n are observed when training on a harder dataset.

This trend holds for all base models across both easy and hard training datasets. Intuitively, scaling n has a different impact depending on the problem difficulty:

• Sampling more rollouts on easy problems can sharpen performance (i.e., improve Avg@K) for problems that are already solvable and improve policy robustness by lowering the probability of an incorrect rollout.

• Increasing the number of rollouts increases exploration and, in turn, aids in discovering rare solutions to difficult problems in order to improve the ratio of problems that can be solved (i.e., Pass@K) by the policy.

Interestingly, B_p has only a moderate performance impact when kept within a reasonable range and is found to primarily influence training stability.

Although the optimal setting of n scales with increasing compute, the exact shape of this scaling law—and the point at which it saturates—changes depending on the exact training setup. Therefore, while the scaling trends hold across different settings, the exact scaling parameters must be fit to the particular RL training setup being used. Practically, authors in [3] recommend the following approach for determining an optimal compute allocation in RL:

1. Execute RL training runs at lower compute budgets by varying the value of B_p and n but restricting the value of M.

2. Fit a scaling law, using the approach described above, from these results.

3. Infer the optimal value of n from the scaling law.

4. Choose the minimum value of B_p that yields stable training.

5. Invest the remaining compute budget into additional training steps M.

This approach provides us with a predictable process for extrapolating the optimal compute allocation for RL from lower-budget experiments.

Comparing RL and Pretraining Scaling Laws

We now have a detailed understanding of scaling laws for both pretraining and RL. However, one of the primary takeaways from this overview is the fact that a “scaling law” is completely different between these two domains. To close, we will briefly discuss the key ways that scaling laws differ between pretraining and RL, explain why these differences exist, and outline key takeaways from research on RL scaling that remain useful despite the overall messiness of RL.

Measuring performance. Pretraining scaling laws predict a particular metric: the cross entropy loss (or another related entropy metric) measured over an in-domain, held-out validation set. This performance metric is stable and is typically computed over a large, diverse dataset (i.e., a random sample of the pretraining corpus). Such a stable, diverse, and specific metric provides the perfect y-axis for fitting a scaling law and allows us to clearly define the impact of specific design decisions on the resulting model. RL scaling laws make an attempt to retain this robustness; e.g., performance is computed over an in-domain validation set. However, RL scaling laws typically use the reward (or accuracy) of the policy9 as the underlying metric to which scaling laws are fit. This is a downstream performance metric that can fluctuate substantially depending on the domain being studied, the benchmark being used, and the composition of data in that benchmark. As a result, scaling laws for RL tend to be more noisy and domain specific relative to those used for pretraining, which capture a more general trend in model performance.

Defining compute. Pretraining has a very clean compute footprint that is usually estimated with the number of training FLOPs C = 6 × N × D. This clean definition of compute provides an obvious x-axis for our scaling law. In contrast, RL compute is difficult to define due to the presence of both sampling and policy updates. The exact definition of compute used in RL scaling laws may change depending on the paper we are reading. For example, some papers derive a FLOP-like metric similarly to pretraining [3], while others rely on the number of GPU hours used [1]. Either way, the wall-clock time of RL training varies substantially depending on the framework being used, which means that the relationship between GPU hours and raw compute is not consistent. These factors must be considered when fitting a scaling law for RL because they cause the details of scaling laws to change depending on the exact setup being used.

Intra and inter-model extrapolation. Pretraining scaling laws are used to fit trends in performance across many model training runs with different settings to understand how model size, data volume, and compute impact the results of training. Such an approach allows us to cleanly extrapolate the results of costly training runs and use these predictions to reason about how compute should be optimally allocated. In RL, we actually fit two kinds of scaling laws that are used to extrapolate performance in different ways (i.e., inter-model and intra-model extrapolation). Inter-model extrapolation is the primary focus of pretraining scaling laws, whereas intra-model extrapolation is not usually addressed. The main reason intra-model extrapolation is necessary for RL is the sensitivity of the training process. In addition to understanding inter-model trends, we need to be able to predict whether a particular training configuration is viable or not.

Lack of standardization. The design space for RL algorithms is quite large: there are simply more “knobs” to tweak relative to pretraining. Additionally, we lack a comprehensive understanding of which design decisions meaningfully impact the scaling properties of RL. Although we have seen several papers that study the impact of design decisions on RL scaling, the findings from these papers—despite being informative—do not address the fact that scaling trends for RL are coupled to the exact training setup being used. Slight changes in the configuration for RL can completely change the scaling trends we observe. For this reason, most RL scaling laws are bespoke—the recommendations offered by one specific analysis may not hold in a different environment. As a result, findings can be difficult to replicate or extend, thus slowing scientific progress on the topic.

Practical takeaways. Despite the fact that RL scaling laws tend to be messy and bespoke, there are still several useful trends that we can learn from the papers in this overview:

• The scaling behavior of RL is predictable within a given setup. Intra-model extrapolation works well and can be used to judge the viability of your setup during the early training phases. Inter-model extrapolation is also effective and can yield useful insights, though these insights may not always transfer across different training configurations.

• The impact of design decisions on RL is not singular. Some decisions impact learning efficiency, while others impact asymptotic model performance. This distinction is important because degradations in efficiency can be solved by simply training for longer, while a degradation in asymptotic performance may not be trivially recoverable. Interestingly, many recent GRPO variants seem to primarily benefit learning efficiency and stability [1].

• Using larger models yields consistently positive results in the RL scaling laws we have seen, though the presence of compute constraints can create interesting tradeoffs. When training with less data or compute, we may actually benefit from using a smaller model due to the fact that learning efficiency saturates with model size.

• To invest more compute into RL, we can i) run training for more steps or ii) use more inference compute at each step. Interestingly, even though the compute cost of RL is dominated by inference, most scaling laws suggest that allocating more compute to sampling completions is helpful. RL training is surprisingly robust to data reuse, benefits from large batch sizes, and scales predictably as we sample more completions per prompt in a batch.

New to the newsletter?

Hi! I’m Cameron R. Wolfe, Deep Learning Ph.D. and Staff Research Scientist at Netflix. This is the Deep (Learning) Focus newsletter, where I help readers better understand important topics in AI research. The newsletter will always be free and open to read. If you like the newsletter, please subscribe, consider a paid subscription, share it, or follow me on X and LinkedIn!

Subscribe now

Bibliography

[1] Khatri, Devvrit, et al. “The art of scaling reinforcement learning compute for llms.” arXiv preprint arXiv:2510.13786 (2025).

[2] Tan, Zelin, et al. “Scaling behaviors of llm reinforcement learning post-training: An empirical study in mathematical reasoning.” arXiv preprint arXiv:2509.25300 (2025).

[3] Cheng, Zhoujun, et al. “IsoCompute Playbook: Optimally Scaling Sampling Compute for LLM RL.” arXiv preprint arXiv:2603.12151 (2026).

[4] Shao, Zhihong, et al. “Deepseekmath: Pushing the limits of mathematical reasoning in open language models.” arXiv preprint arXiv:2402.03300 (2024).

[5] Zheng, Chujie, et al. “Group sequence policy optimization.” arXiv preprint arXiv:2507.18071 (2025).

[6] Yu, Qiying, et al. “Dapo: An open-source llm reinforcement learning system at scale, 2025.” URL https://arxiv. org/abs/2503.14476 1 (2025): 2.

[7] Liu, Zichen, et al. “Understanding r1-zero-like training: A critical perspective.” arXiv preprint arXiv:2503.20783 (2025).

[8] Chen, Aili, et al. “Minimax-m1: Scaling test-time compute efficiently with lightning attention.” arXiv preprint arXiv:2506.13585 (2025).

[9] F. Yao, L. Liu, D. Zhang, C. Dong, J. Shang, and J. Gao. Your efficient rl framework secretly brings you off-policy rl training, Aug. 2025. URL https://fengyao.notion.site/off-policy-rl.

[10] Chen, Aili, et al. “Minimax-m1: Scaling test-time compute efficiently with lightning attention.” arXiv preprint arXiv:2506.13585 (2025).

[11] Piché, Alexandre, et al. “Pipelinerl: Faster on-policy reinforcement learning for long sequence generation.” arXiv preprint arXiv:2509.19128 (2025).

[12] Shao, Zhihong, et al. “Deepseekmath: Pushing the limits of mathematical reasoning in open language models.” arXiv preprint arXiv:2402.03300 (2024).

[13] Kaplan, Jared, et al. “Scaling laws for neural language models.” arXiv preprint arXiv:2001.08361 (2020).

[14] Hoffmann, Jordan, et al. “Training compute-optimal large language models.” arXiv preprint arXiv:2203.15556 10 (2022).

[15] Guo, Daya, et al. “Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.” arXiv preprint arXiv:2501.12948 (2025).

Throughout the history of AI research, progress has been measured—and accelerated—by high-quality benchmarks. AI is an empirical field that is driven by discovering interventions that improve performance on key benchmarks. For large language models (LLMs) in particular, creating useful benchmarks is hard due to rapidly advancing model capabilities. Tough evaluations are regularly saturated as new models are released, creating the need for continual evolution toward harder problems and new dimensions of performance. Despite the pivotal role of benchmarking in driving progress, evaluation has traditionally received less attention compared to core modeling research. Additionally, creating high-quality benchmarks requires unique skills that are emphasized less heavily in the literature. This overview aims to solve these problems by providing an extensive survey of useful LLM benchmarks and the techniques—including both practical tricks and more recent directions of research—used to create them.

Disclaimer. Agent and coding benchmarks are notably absent from this overview. These domains are rapidly advancing and require unique evaluation techniques that have led to the creation of completely new areas of research in LLM evaluation. Due to their depth, these topics will require an overview of their own, and several useful resources on these topics are already available.

Dissecting Popular LLM Benchmarks

The best way to understand how LLM benchmarks are created—and how we can create a useful benchmark for our own task of interest—is to simply study details of the most popular and effective LLM benchmarks. In this section, we will select a wide variety of LLM benchmarks, including both recent benchmarks and those that have been around for a while, and outline the following characteristics:

• How the data is sourced

• How data quality is ensured

• How model performance is measured

• How each benchmark has evolved as models have improved

Admittedly, this section is far from comprehensive—a vast number of LLM benchmarks exist, and surveying them all would be impossible. Instead, this section optimizes for diversity and aims to provide a wide view of the different kinds of benchmarks that exist and the various strategies that are commonly used to create useful evaluation datasets across these many different domains.

Massive Multitask Language Understanding (MMLU) [1]

“To succeed at our test, future models should be well-rounded, possess extensive world knowledge, and develop expert-level problem solving ability. These properties make the test likely to be an enduring and informative goalpost.” - from [1]

MMLU is one of the most widely used general knowledge benchmarks for LLMs. The data curation strategy for MMLU is simple: questions are sourced from freely available online sources and manually curated by graduate and undergraduate students. The benchmark contains ~16K questions divided into 57 subjects1 that span various topics like STEM, humanities, social sciences, and more. The full MMLU benchmark contains a development set of five examples per subject (i.e., used for few-shot prompting), a validation set of 1.5K questions, and the main test set. For each task, we have a minimum of 100 questions in the test set.

Data format. The questions within the MMLU benchmark use a multiple choice format, and models are evaluated using a zero or few-shot prompting strategy. Authors of the benchmark specifically avoid open-ended generation due to the increased evaluation complexity. Multiple choice correctness can be validated with string matching, allowing MMLU to be evaluated using accuracy. Several example questions from MMLU are provided below for reference.

Difficulty. Some subjects are separated into sub-tasks based on their difficulty level. More specifically, MMLU defines subjects at elementary, high school, college, and professional levels, where difficulty is inferred from the source of the questions. For example, the professional subset of the Psychology domain pulls from the exam for professional practice in Psychology, whereas the high school subset pulls from advanced placement exams (i.e., tests for high school students). Notably, not all subjects have a task for each difficulty level.

As we might expect, human-level accuracy on MMLU varies significantly based on the human, domain, and level of difficulty being considered. Given that MMLU is still popular even today, several extensions have been proposed (e.g., MMLU-Pro [2] and MMLU-Redux [3]) to diagnose quality issues and to keep the benchmark from becoming saturated by newly-released LLMs over time.

“[Benchmark performance] has begun to plateau, making it increasingly difficult to discern differences in model capabilities. This paper introduces MMLU-Pro, an enhanced dataset designed to extend the mostly knowledge-driven MMLU benchmark by integrating more challenging, reasoning-focused questions and expanding the choice set from four to ten options. Additionally, MMLU-Pro eliminates the trivial and noisy questions in MMLU.” - from [2]

MMLU-Pro. We learn in [2] that MMLU has a non-negligible ratio of easy (i.e., knowledge-only or low reasoning) questions, as well as some questions that are flawed or incorrect. To avoid saturation and reduce noise, MMLU-Pro [2] reconstructs the benchmark in order to make it more accurate, difficult, and discriminative. The 57 subjects from MMLU are consolidated into a set of 14 broader domains, and the majority of easy questions are removed from MMLU-Pro using model-based difficulty filtering. A pool of eight models is tested on each question, and any question that the majority of models answer correctly—5,886 questions in total—is removed. From here, the remaining MMLU questions are supplemented with harder questions from sources like TheoremQA and SciBench, yielding a final benchmark of ~12K questions; see below.

For new data sources, questions are converted into a multiple choice format by asking GPT-4-Turbo to extract a correct answer and generate distractor answers. The result of this process is manually verified by asking human annotators to compare extracted answers to the original solution for each question. To reduce the impact of random guessing, the number of choices for each question is also expanded from four to ten—this is referred to as “option augmentation” in [2].

After data filtering and curation, MMLU-Pro undergoes an extensive quality control phase with multiple stages of verification by humans and LLMs. The quality control process aims to identify bad questions, incorrect answers, and false positive distractors. Human validation is performed first, then Gemini-1.5-Pro flags any remaining issues for a second stage of human review.

The full curation pipeline for MMLU-Pro is depicted above. MMLU-Pro still uses accuracy as the main performance metric, though we can also separately examine accuracy within each specific domain. Most LLMs perform worse on MMLU-Pro relative to MMLU—the benchmark is more difficult and has headroom before saturation—and model capability gaps tend to be more noticeable. We also see in [2] that MMLU-Pro offers improved prompt stability and benefits from advanced reasoning techniques (e.g., chain of thought prompting).

MMLU-Redux. An in-depth quality audit of the MMLU benchmark is performed in [3] over a subset of 100 questions randomly sampled from each MMLU task (i.e., 5,700 questions in total). Quality issues are categorized using a hierarchical error taxonomy; see above. This taxonomy contains five error categories that are used to granularly categorize questions with poor quality or incorrect ground truth. When necessary, questions are re-annotated and verified according to the original source material or, when the original source is absent, a trusted source (e.g., government websites). We see in [3] that an estimated 6.49% of MMLU questions contain errors, but the ratio of errors varies between subjects; e.g., 57% of Virology questions were flagged due to quality issues; see below.

The result of this sampling and re-annotation procedure is MMLU-Redux, a subset of 5,700 manually inspected MMLU questions. For several high-error subjects, authors monitor agreement across three separate annotators using Cohen’s Kappa. Re-annotation agreement is found to be strong even on difficult subjects, providing confidence in the quality of the human-audited data. The aim of this effort is not to produce a harder version of MMLU but rather to audit (and fix or discard) existing questions for quality and accuracy—MMLU-Redux is an updated subset of MMLU that can be adopted for more reliable evaluation.

We see in [3] that removing incorrect evaluation data meaningfully impacts performance and model rankings; see below. For example, Llama-3.1-405B improves from 16th to first in rank for Virology and Qwen-2-72B-Instruct drops from first to eighth place for College Chemistry when only evaluating on correct instances from MMLU-Redux—these results suggest improved reliability.

GPQA: A Graduate-Level Google-Proof Q&A Benchmark [4]

GPQA is another popular LLM benchmark that takes a different approach from MMLU. Namely, GPQA is a much smaller dataset: the extended version contains 596 questions, while the main and diamond subsets contain 448 and 198 questions, respectively. Rather than providing broad coverage, GPQA focuses on curating a small number of expert-verified questions that are difficult to solve even with internet access (i.e., a “Google-proof” benchmark). Three primary domains are covered—Biology, Chemistry, and Physics—each of which is divided into several sub-domains2. Similarly to MMLU, however, GPQA does adopt a multiple choice question format with four answers per question.

Expert curation. The data from GPQA is manually curated by a group of 61 human experts that each have—or are pursuing—a PhD in a relevant field. The data curation pipeline for GPQA is depicted below. To begin, experts in each domain write a set of candidate questions. These questions are written from scratch, rather than being collected from existing exams or datasets. As a guiding principle, experts are specifically asked to write questions that are:

• Difficult.

• Answerable by experts in the same domain.

• Not possible for non-experts to answer, even with internet access.

Questions are always written such that they can be answered with or without choices being presented, thus enabling GPQA to be easily extended to an open-ended generation format in the future. In addition to writing each question, a written explanation is provided for both the answer and all distractors.

After a question is written, two separate domain experts interact with it. The first expert solves and validates the question, then suggests possible revisions. After the writer revises the question based on suggestions, a second domain expert answers the revised question. Finally, three different non-expert validators—selected from the group of experts for other, non-overlapping domains—try to answer the question with unrestricted internet access, spending a minimum of 15 minutes and nearly 40 minutes on average answering each question.

“The process consists of four main stages: question writing, expert validation, question revision, and non-expert validation.” - from [4]

Verification principles. The GPQA curation process validates both correctness and difficulty. Correctness is handled via expert validation and revision, while difficulty is assessed based on the ability of non-experts to solve questions. The results of these two stages are used to define the different subsets of GPQA:

• GPQA Extended: full dataset (546 questions).

• GPQA Main: questions where at least one expert agrees with the answer and at most two non-experts answer the question correctly (448 questions).

• GPQA Diamond: questions where both experts agree with the answer and at most one non-expert answers the question correctly (198 questions).

As shown below, the resulting subsets are quite difficult, with experts achieving around 70-80% accuracy and non-experts a much lower accuracy of 30-40%.

Beyond the Imitation Game Benchmark (BIG-Bench) [5]

“BIG-bench… includes a set of 204 or more language tasks. As reflected in the BIG-bench review criteria, benchmark tasks are novel, cover a diverse range of topics and languages, and are not fully solvable by current models.” - from [5]

BIG-Bench explores a community-based strategy for curating difficult LLM evaluation tasks. The benchmark was openly constructed on Github, where researchers were asked to contribute tasks by creating a pull request. Each task was then manually reviewed in a corresponding PR discussion according to detailed submission criteria; e.g., correctness, difficulty, decontamination, and justification (i.e., why is this an important task for LLMs to solve?). The version of BIG-Bench outlined in [5] contains 204 tasks that were curated by 405 authors. The set of included tasks is incredibly broad, covering topics like math, coding, reasoning, science, and more; see here for a summary of task domains.

Task interface. Unlike the benchmarks we have seen so far, BIG-Bench does not have any unified data format—tasks have varying formats ranging from multiple choice to open-ended generation and multi-turn (interactive) chat. In order to handle the diversity of tasks present in BIG-Bench, authors introduce a standard API structure that is used by all tasks. This API specifies two task types:

1. JSON: defined by a JSON file containing a list of input-output examples.

2. Programmatic: defined by a Python function that can interact directly with the model over multiple chat turns and compute custom metrics.

By using these standardized structures for all tasks, we can easily evaluate any public model or onboard new tasks with minimal implementation changes. The distribution of BIG-Bench tasks follows an 80-20 split between JSON and programmatic task types. In programmatic tasks, we interact with the model via two standard functions:

1. generate_text: generate a text continuation from the model.

2. cond_log_prob: compute log probabilities of a target given input.

The model can be queried multiple times within a programmatic task, enabling support for multi-turn chat or iterative tasks within BIG-Bench. Each task must have a minimum of 32 evaluation samples, though authors are encouraged to create much larger tasks; see below for a distribution of task sizes.

Performance metrics. Given that BIG-Bench tasks follow a variety of formats, we cannot evaluate all tasks with a unified performance metric like accuracy. Instead, a suite of standard metrics is provided for all tasks, and programmatic tasks are even allowed to define their own custom metrics. In [5], authors list the following performance metrics as being used in BIG-Bench:

• Exact String Match.

• Multiple Choice Accuracy.

• Text Similarity Metrics (e.g., BLEU, BLEURT, or ROUGE).

• Multi-Category Brier Score: evaluates the calibration—a measure of how well confidence3 aligns with observed correctness—of a model’s outputted probabilities on options for a multiple choice question.

• Expected Calibration Error: another calibration metric that measures how well the model’s accuracy matches the probability assigned to a response in the multiple choice setting.

Interestingly, BIG-Bench even allows multiple evaluation metrics to be defined per task, but one metric must be defined as the primary metric. Additionally, each task must specify a high and low reference score on the primary metric. Using this information, we can normalize each task’s preferred metric using the high and low reference scores. Then, we can compute aggregate performance over the entire benchmark by averaging normalized metrics across tasks—this approach summarizes benchmark performance with a single score in the range [0, 100].

As shown above, all models at the time of BIG-Bench’s proposal performed well below human baseline performance; see above. Although performance improves with model scale, all models perform poorly in an absolute sense, indicating that the benchmark was quite difficult to solve for models at that time. Human performance metrics in the above plot—reported as both a max and mean score across multiple annotators—were collected using a team of expert annotators that were given full internet access. However, properly measuring human performance is difficult given the breadth of tasks present in BIG-Bench.

BIG-Bench Lite. The size and breadth of BIG-Bench makes it computationally expensive to run. To solve this, authors in [5] provide a smaller task subset, called BIG-Bench Lite, to use for faster evaluation. This subset is made up of 24 JSON-style tasks that are chosen via a manual selection process that considers task diversity and inclusion of specific task types (e.g. coding or non-English tasks).

BIG-Bench Hard (BBH). Less than a year after the release of BIG-Bench, LLMs had already begun to surpass average human performance on the majority of tasks. BIG-Bench Hard [6], a difficult subset of the BIG-Bench dataset, was created in response to these quick improvements in capabilities. The steps used to select the tasks within BIG-Bench Hard are outlined in the table below.

All tasks in BIG-Bench Hard are derived from BIG-Bench. Initially, tasks are filtered according to several heuristics; e.g., not containing too many subtasks, having too few evaluation examples, or using evaluation metrics beyond multiple choice or exact match accuracy. Any task without a human performance baseline is also removed, and the remaining task subset is further refined by only retaining tasks where models underperform humans. From here, tasks are then manually inspected to remove any tasks that are overly difficult or out of scope4, leaving us with the final set of 23 tasks in BIG-Bench Hard; see below.

Despite focusing on a much smaller set of difficult tasks—about 10% of the original benchmark—that have a standard format, BIG-Bench Hard is mostly able to maintain the breadth of BIG-Bench. The tasks present in BIG-Bench Hard can be roughly categorized into natural language (e.g., detecting translation errors or recommending movies) and algorithmic (e.g., evaluating boolean expressions or performing multi-step arithmetic) tasks. When examining model performance on BIG-Bench Hard, we see that the models considered in [6] usually surpass average human performance but fall short of the best performance of a human. However, the best LLMs today achieve almost perfect accuracy on BIG-Bench Hard.

Given that BIG-Bench is constructed as a community effort, benchmark tasks have a high level of variance—cleanliness and quality fluctuate, and each task may have different metadata. Tasks are selected based on both quality and difficulty by using a combination of heuristics and manual inspection. Additionally, BIG-Bench Hard restricts the benchmark to tasks that use an exact match or multiple choice format. This choice is made to simplify the analysis of chain of thought prompting by enabling the use of a unified prompt format across different tasks. In this way, BIG-Bench Hard does not solely maximize difficulty—it identifies a subset of hard tasks that also work well with chain of thought prompting.

As shown above, several top models at the time of release for BIG-Bench Hard noticeably underperform the average human baseline. This gap can be closed in many cases via chain of thought prompting, but benchmark performance still falls short of maximum human performance for even the largest models.

BIG-Bench Extra Hard (BBEH). The BIG-Bench family is one of the few reasoning-focused evaluation suites that prioritizes general reasoning rather than math and coding. However, both BIG-Bench and BIG-Bench Hard were saturated by early 2025, with top reasoning models achieving nearly perfect scores on both benchmarks. As a solution, BIG-Bench Extra Hard was created by replacing each of the BIG-Bench Hard tasks with a corresponding task that tests a similar category of reasoning capabilities but is significantly more difficult.

“BIG-Bench Extra Hard replaces each task in BIG-Bench Hard with a novel task that probes a similar reasoning capability [with] increased difficulty.” - from [5]

Examples of new reasoning skills tested by BIG-Bench Extra Hard include many-hop reasoning, long context reasoning, properly handling distractors, finding errors in reasoning traces, reasoning under constraints, and more. To perform well on BIG-Bench Extra Hard, models must command a breadth of different reasoning capabilities. An itemized list of the reasoning tasks present in BIG-Bench Extra Hard is provided in the figure below. Each task matches the general reasoning domain of some corresponding task in BIG-Bench Hard, ensuring that the diversity of BIG-Bench Hard is preserved while increasing task difficulty.

As seen in the middle column of the table, tasks in BIG-Bench Extra Hard are sourced from a variety of existing reasoning benchmarks and manually chosen according to their topic and difficulty. When curating the benchmark, authors aim to solve the following known issues with BIG-Bench Hard:

• Many tasks have high random chance performance due to the presence of multiple choice questions with a small number of options (e.g., ~35% of tasks have binary output and ~20% of tasks use multiple choice with <5 options).

• Some tasks permit shortcuts that allow the task to be “solved” without actually reasoning through a proper solution.

• Task inputs tend to be very short—around 700 characters on average—across BIG-Bench Hard tasks, which is unrealistic compared to how LLMs are typically used in practice.

• True multi-hop reasoning is rarely tested in BIG-Bench Hard due to limitations in LLM capabilities when the benchmark was created.

Ideally, we would like to solve all of these issues while expanding the set of reasoning capabilities being tested by the benchmark. BIG-Bench Extra Hard tasks contain 200 questions—except for DisambiguationQA, which has only 120. Although the task selection process was mostly manual, data was curated using a combination of manual human inspection with model assistance. Two models are used—a general purpose model and a reasoning model (both Gemini-based)—to iteratively evaluate data that is selected for each task. Tasks that were easily solved by the reference models were either i) discarded and replaced with more difficult tasks or ii) enhanced with harder reasoning examples. This process continued until both models achieved an accuracy below 70% on each task.

“In most cases, we tried to use the reference models only as a black box that provided feedback on the difficulty of our tasks. In some cases, however, making tasks more difficult required looking into the approach adopted by the model.” - from [7]

The combination of human and model oversight in BIG-Bench Extra Hard is interesting and provides motivation for unique ways in which humans can interact with LLMs to curate better evaluation data. For example, authors in [7] even mention manually inspecting reasoning traces from the models to help them think of more difficult examples that would actually challenge the model. Tasks in BIG-Bench Extra Hard have significantly expanded context compared to the prior benchmark, have negligible random chance performance, and provide a lot of headroom in performance even for top models (e.g., o3-mini); see below.

IFEval [8] and IFBench [9]

The IFEval [8] benchmark tests LLM instruction following capabilities, with an emphasis on instructions that are objectively verifiable (i.e., as opposed to instructions that are more subjective). For example, if we instruct an LLM to generate an output containing 100 to 200 words, we can easily verify whether this instruction was followed by using a basic script. However, verifying whether an LLM obeys a certain tone specification in its output is less straightforward.

“The task of precise instruction following evaluates a language model’s ability to perform a task t, such as summarization or creative writing, while adhering to one or more output constraints c, which can be automatically verified.” - from [9]

To start, 25 instructions—structured as verifiable constraint templates for the model’s output—are manually curated based on practicality and verifiability; see below.

From these instructions, evaluation samples are curated as follows:

1. Create a set of base prompts5.

2. Combine these base prompts with one to three randomly selected verifiable instructions by concatenating instructions to the end of the prompt.

3. Use few-shot prompting and manual inspection to identify instruction combinations that are illogical or contain conflicts.

4. Use few-shot prompting to rephrase each prompt and, in turn, improve the diversity of instructions in the benchmark.

5. Manually review all rephrased prompts.

Exact details of the data curation process are not fully outlined in [8]. However, we know from the information provided that a model-in-the-loop approach is used with manual human review to ensure quality. To measure performance, a binary verification check is created for each instruction that can be used to determine if a model followed an instruction or not. Instruction-level binary verification signals can be used to compute the following strict metrics:

• Instruction-level strict accuracy: the percentage of all individual instructions that the model follows.

• Prompt-level strict accuracy: the percentage of prompts for which the model follows all instructions.

Additionally, several loose metrics are considered in [8] that perform verification under a variety of transformations to the model output (e.g., removing markdown and removing the first or last lines). After applying a transformation, we can compute instruction and prompt-level accuracy similarly to before, resulting in a loose version of each metric. An instruction is considered solved if it passes verification after any of the possible transformations that are tested.

“The new constraints we introduce were created manually – sourced by collecting feedback from LM users beyond the authors on the types of constraints they have tried with models, or manually written to cover core instruction following skills. Then, we filtered constraints for the benchmark to those that can be easily paired with a verification function written in Python, making for reproducible evaluation and training tools.” - from [9]

The IFEval benchmark only tests 25 instructions and, therefore, risks overfitting to a small set of constraints. As a solution, IFBench [9] proposes an expanded set of 58 verifiable, manually-curated constraints. When deriving new constraints, authors i) inspect feedback from LLM users on instruction following issues, ii) focus on core areas of instruction following, iii) emphasize difficult constraints, and iv) only use constraints that can be verified with a Python function. Going further, an additional set of 29 constraints (IFTrain) are provided for training purposes. These training constraints can be used for RLVR training, enabling investigation into the generalization properties of instruction following.

The 58 constraints in IFBench are grouped into seven categories—count, ratio, words, sentence, format, custom, and copy—that cover a broad range of instruction following skills. To create prompts for these instructions, authors take unseen prompts from WildChat and combine them with either one or two constraints from the expanded set. Every test prompt is manually inspected by a human annotator to ensure constraint compatibility, and the final benchmark consists of 300 total prompts. As shown below, performance on IFBench is noticeably lower than on IFEval, indicating some level of overfitting to specific constraints.

Authors in [9] provide a potential reason for the overfitting to IFEval constraints. Many LLMs have curated training data that specifically targets instruction following capabilities. Most of this training data is synthetically generated because precise instruction following can be deterministically verified. Given the popularity of IFEval, model developers often adopt the same constraint taxonomy when generating synthetic instruction following data; see Nemotron-4 340B as an example. As a result, some models may be explicitly trained to follow the same constraints being tested by IFEval, leading to inflated performance metrics6.

AlpacaEval [13]

AlpacaEval is a pairwise instruction following benchmark that measures model performance by using an LLM judge to compare candidate model completions to those of a baseline model; see above. The most recent version of AlpacaEval uses GPT-4-Turbo as both the baseline and judge model. The data used in AlpacaEval is sourced from the earlier AlpacaFarm dataset, which contains a total of 805 prompts derived by combining the evaluation sets from:

• Self-Instruct

• Open Assistant

• Anthropic Helpfulness

• Vicuna

• Koala

Despite the variety of data sources, most of this data is curated using a similar approach. For example, Self-Instruct proposes a synthetic data generation strategy for instruction tuning, but prompts from the evaluation dataset for Self-Instruct are manually written by human experts. Similarly, Anthropic helpfulness is a human preference dataset, while the Vicuna and Koala test sets are manually curated by researchers working on the projects. The only outlier of these evaluation sets is Open Assistant, which is derived from crowdsourced human conversations with an LLM, rather than being curated by experts.

After the initial release of AlpacaEval, several follow-up versions of the benchmark were published, but the underlying evaluation data did not change much. Instead, subsequent improvements to AlpacaEval focused on changing the reference and judge models to improve the benchmark’s correlation with human preferences. Full code and updates to AlpacaEval can be found here.

Math Evaluation

Many evaluation datasets exist in the math domain, and most of them are either i) expert-curated or ii) drawn from test banks for math competitions. For example, GSM-8K contains 8.5K human-written grade school math problems, while MATH contains 12.5K questions compiled from high school math tests. Additionally, the American Invitational Mathematics Examination (AIME), which is commonly used to evaluate LLMs, is released every year with a set of 15 new questions. Questions from the American Mathematics Competitions (AMC) are also commonly used for LLM evaluation. Solutions to questions in these benchmarks are usually graded with an automatic verifier or exact string matching.

The benchmarks outlined above have been saturated by modern LLMs, but many frontier-level math benchmarks have been recently proposed:

• FrontierMath contains hundreds of expert-crafted problems at the cutting edge of mathematical research that require hours or days to be solved by an expert-level researcher.

• RealMath is a continuously-evolving benchmark that automatically updates with new problems derived from research papers and discussion forums.

• MathArena is an evolving benchmark that evaluates LLMs on math competition problems soon after their release to avoid contamination risk.

• OmniMath contains 4.5K competition-level math problems that have been annotated by human experts, covering a diverse range of topics (i.e., over 30 sub-domains) and difficulty levels.

Solutions to questions in these benchmarks are still commonly evaluated with automatic verifiers, but this is not always the case. For example, proof-based questions in MathArena are manually checked by human experts. Despite the impressive math capabilities of modern LLMs, most of these frontier-level math benchmarks have not yet been fully saturated. However, LLMs are advancing rapidly in their capabilities, so several of these datasets are designed in a way that enables continual evolution in order to avoid contamination and saturation.

Iteratively Improving a Benchmark

When studying the benchmarks outlined above, we see several examples of iterative benchmark refinement. Benchmarks become saturated and less informative over time, which is usually addressed by releasing an improved benchmark. To create such an improved benchmark, there are several common techniques and directions that are usually followed, such as:

• Difficulty-based refinement: curating more difficult tasks or data to use for evaluation within a benchmark.

• Quality-based refinement: identifying and fixing issues in the benchmark (e.g., mislabeled data, vague or unrealistic questions, poor format, etc.).

• Diversity-based refinement: expanding the scope of questions and topics covered by a particular benchmark.

Usually, these directions of improvement are handled via manual human review, a model-in-the-loop approach, or some combination of both. In some cases, we can even design a benchmark in a way that continually evolves over time without too much manual effort (e.g., RealMath and MathArena). However, the range of techniques that can be used for iterative benchmark improvement is vast—there is a lot to learn in this area. To provide pointers for future learning, a set of useful resources for benchmark improvement is listed below:

• Do Large Language Model Benchmarks Test Reliability?: corrects labeling errors in common LLM benchmarks to better measure LLM reliability.

• Improving Model Evaluation using SMART Filtering of Benchmark Datasets: a framework for systematically identifying and filtering evaluation data that is too easy, similar to other questions, or possibly contaminated.

• From Crowdsourced Data to High-Quality Benchmarks: an LLM-based approach for post-processing crowdsourced data into high-quality evaluation samples.

• Reliable and Efficient Amortized Model-based Evaluation: a model-based approach for difficulty filtering and difficult question generation.

• Evidence-Centered Benchmark Design for NLP: an evidence-backed framework for properly designing evaluation benchmarks.

• Evaluation Guidebook (from Hugging Face): a practical field guide for evaluating LLMs, assessing benchmark quality, and curating evaluation data.

There are also many papers that have been proposed for optimally selecting subsets of benchmark data to improve efficiency [14, 15, 16, 17].

Advanced Benchmarking for LLMs

Now that we understand practical details for constructing LLM benchmarks, we will take a deeper look at some advanced techniques for LLM evaluation that have been proposed in recent research. Specifically, we will focus on a set of papers that use Item Response Theory (IRT) to select the most informative data for evaluation. Coming from the field of psychometrics, IRT uses statistical modeling to dynamically measure how an individual’s latent abilities interact with the properties of an item (or question) to determine the probability of a correct response. Although IRT is commonly applied in standardized testing environments, the same concepts have been adopted by LLM researchers. We can directly apply techniques from IRT to LLM evaluations by considering the LLM as our individual and the evaluation dataset as our standardized test!

In the context of LLM evaluations, IRT considers a model l, dataset items i, and the probability p_il that model l gets item i correct. We can use a variety of different models—usually just different variants of logistic regression—to predict this probability. IRT models include parameters for both the model and the item being evaluated. Whereas model parameters capture the capabilities of a given model, item parameters capture the following properties:

• Difficulty: whether the item is easy or difficult to answer correctly.

• Discrimination: whether answer correctness has a strong relationship with the capability level of a model.

By capturing these properties within our IRT model, we gain a rich description of our evaluation data that can be directly applied to benchmark improvement. For example, items with low discrimination are often problematic (e.g., due to mislabeling), and we can consider filtering out items that are too easy from the evaluation process. Within this section, we will see several IRT formulations that demonstrate a broad set of potential applications to the evaluation process.

tinyBenchmarks: Evaluating LLMs with Fewer Examples [11]

“Evaluating the performance of a single LLM on HELM costs over 4K GPU hours (or over $10K for APIs). Benchmarks like AlpacaEval also require a commercial LLM as a judge to perform evaluation, further increasing the costs… evaluation of a single model is often performed many times to… explore different prompting strategies or a wider range of hyperparameters.” - from [11]

To mitigate excessive inference costs during evaluation, an IRT-based approach called tinyBenchmarks is proposed in [11] that intelligently samples evaluation data in a way that maintains the accuracy of a model’s performance metrics. We assume access to a dataset of historical evaluation results that can be used for selection and performance estimation. More specifically, this dataset contains items i and models l, where each item and model combination has a binary score Y_il ∈ {0, 1}. We can also handle continuous evaluation results in the range [0, 1]—nearly any evaluation setting can be converted into this format by normalizing scores—by simply binarizing scores according to a fixed threshold.

Baselines. There are a few simple and effective approaches that can be adopted to sample a subset of data from an evaluation dataset:

1. Stratified random sampling: ensure proportional representation across benchmark sub-domains by randomly sampling a subset of evaluation samples separately within each subdomain.

2. Correctness-based clustering: sample evaluation data based on patterns in correctness by representing each item i as a vector of correctness scores for each model l, performing K-means clustering on these vectors, and selecting the evaluation samples closest to each cluster centroid.

Despite their simplicity, these techniques have notable drawbacks. Stratified sampling leads to high variance and uncertainty when the number of samples is small, while correctness-based clustering tends to suffer from the curse of dimensionality if we have evaluation results from a large model pool.

IRT model. In [11], IRT is used to derive a much smaller representation of our evaluation data that can be more effectively used to both select samples and estimate performance. We define item i using two parameters:

• α_i: captures the skills required to solve item i.

• β_i: captures the overall difficulty of item i.

Similarly, we describe model l with the parameter θ_l, which captures model capabilities. From here, we define a multidimensional IRT model, which predicts the probability p_il that item i will be answered correctly by model l; see below. We can fit the IRT model—or learn the correct values for all of the model and item parameters—by using our historical evaluation dataset as training data.

As we can see, the center point of this equation is the inner product of the item and model parameter, which captures how well the capabilities of a model match those needed for an item. Intuitively, a model is more likely to answer an item correctly if it has strong capabilities in the same directions required to solve an item and vice versa. Additionally, we add an extra bias term to this inner product to account for overall item difficulty before passing the full expression through a sigmoid (or logistic) function to yield a probability in the range [0, 1].

“The IRT model creates a meaningful representation for each example i based on their difficulty and the abilities required to respond to those examples correctly. This approach immediately solves the dimensionality problem, since E_i is low-dimensional… IRT should represent which examples have similar difficulty and require similar abilities.” - from [11]

Once fitted, parameters of the IRT model naturally provide a d + 1 dimensional vector E_i = (α_i, β_i) that can be used to represent items in our evaluation dataset. This representation is low dimension (d < 16 in [11]) compared to vectors used for correctness-based clustering, thus solving issues related to the curse of dimensionality. The IRT model is used in two ways in [11]:

1. To perform cluster-based sampling, similarly to correctness-based clustering (but with embeddings from the IRT model E_i).

2. To predict model performance over items—this is more efficient than actually running the evaluation itself.

p-IRT estimator. In [11], the two approaches described above are used in tandem to efficiently estimate model performance on an evaluation set. Assume we want to evaluate a new model l’ on an existing evaluation set for which we already have an IRT model fitted. We can use clustering to identify “anchor points”—or high-signal evaluation samples—in the data and evaluate our model only on these samples. The number of anchor points is a hyperparameter that can change with our evaluation budget. We can then keep our existing item parameters fixed in the IRT model and only train the parameter for our new model θ_l’, using real evaluation results on our anchor points as training data. After obtaining θ_l’, we can predict performance on the remaining items by using our IRT model.

A formal description of this approach, called the p-IRT estimator in [11], is outlined above. Put simply, we are interested in measuring the model’s actual performance on the full evaluation set, but running an entire benchmark is expensive. Instead, we use IRT model parameters to obtain K anchor points via clustering—where K is much smaller than the full dataset size—and only evaluate our model on these anchor points. Then, we can estimate performance on the rest of the evaluation dataset using the IRT model and derive an overall performance estimate by averaging real and predicted evaluation results; see above.

Beyond the p-IRT estimator, we can estimate performance with a sample average of the model’s performance on the anchor points only. This sample average has low bias because we are using correctness values obtained from our model on the actual evaluation data. However, the variance of the sample average is high when the number of anchor points K is small. On the other hand, the p-IRT estimator is biased—due to the fact that our IRT model is not perfectly accurate—but has low variance. Therefore, we can create an estimator that combines the strengths of both approaches by taking a convex combination of each estimate; see below.

This revised estimator is referred to as IRT++ in [11]. The per-item weight in this expression is optional but can be used to assign non-uniform weights to anchor points. For example, this weight can correspond to the ratio of evaluation samples present in the cluster used to derive a given anchor point. In [11], λ lies in the range [0, 1], and the optimal value of λ depends upon several factors (e.g., the number of anchor points and the variance of our performance estimate). The value of λ is derived in [11] by using a heuristic proposed in prior work.

Efficient evaluation. To test the efficacy of IRT-based performance estimation, four benchmarks are considered—Open LLM Leaderboard, MMLU [1], HELM, AlpacaEval 2.0 [13]—and we compare the estimated and actual performance on each benchmark. Training data for the IRT model is collected from a large number of LLMs—395 models for Open LLM leaderboard and MMLU, 30 models for HELM, and 100 models for AlpacaEval 2.0—to ensure the quality of the IRT model. The LLMs are split into training and test sets using two approaches:

• Random: randomly sample a subset of LLMs to use for testing.

• Date-based: use the most recent LLMs for testing.

As shown below, the proposed IRT-based estimators perform well across all scenarios considered. With as few as 100 anchor points per sub-domain of the evaluation set—a reduction of 140× for MMLU and 160× for the Open LLM Leaderboard—we can estimate performance with less than 2% error.

Fluid Language Model Benchmarking [12]

Most LLMs are evaluated in a static fashion (i.e., by computing accuracy on a fixed dataset). Whereas raw accuracy treats every evaluation sample equally, IRT estimates a model’s underlying capabilities, taking into account factors like the difficulty and discrimination of each question. Leveraging this insight, authors in [12] propose an approach called Fluid Benchmarking that uses an IRT model to dynamically select evaluation data for a particular model. The key idea behind this approach is that the value of an evaluation sample depends upon a model’s capabilities. Instead of assuming there is a single best subset of examples on which to evaluate an LLM, Fluid Benchmarking dynamically selects the most informative evaluation examples for a particular model and, in turn, provides a more accurate estimate of that model’s performance.

“Fluid benchmarking is based on the insight that the relative value of benchmark items depends on an LM’s capability level… a hard question might be too difficult for a weak LM, but informative for a strong LM.” - from [12]

Unidimensional IRT. Similarly to before, the approach in [12] fits an IRT model using a dataset of historical evaluation data derived from evaluating a large set of models on a benchmark of interest. However, a different IRT model structure is used in [12]. As shown below, this is again a two-parameter IRT model that is used to predict binary evaluation outcomes, but we use unidimensional—as opposed to the multidimensional approach used in [11]—model and item parameters. Authors mention testing a multidimensional IRT approach in [12] but found that this formulation performs poorly compared to a unidimensional IRT model.

Despite the different IRT model structure used in [12], the purpose of these parameters remains the same:

• θ_l: a scalar parameter that represents the capability of model l.

• α_i: a scalar parameter that captures the discrimination7 of item i.

• β_i: a scalar bias that represents the difficulty of item i.

The Fluid Benchmarking approach proposed in [12] is depicted above. There are two main phases for obtaining a benchmark result:

• An offline (or historical) phase, where we fit item and model parameters in the IRT model from leaderboard-style results on our benchmark.

• An online phase, where we learn the parameter of a new model given a subset of evaluation results for this model on our benchmark.

The IRT model is initially fit using an offline dataset of evaluation results. Given a new model l’, we first evaluate this model on a subset of our evaluation set to obtain some training data for the new model parameter θ_l’. Similarly to [11], we then leave item parameters fixed and fit only the new model parameter θ_l’ by using the actual evaluation data collected with our new model for training.

By examining the structure of our IRT model, we can intuitively understand how item parameter values can influence the value of θ_l’. Easy questions have a small (or negative) difficulty parameter β_i, so answering them correctly has minimal impact on θ_l’. On the other hand, correct answers to a difficult question will meaningfully impact the value of θ_l’. The same arguments hold in reverse for incorrectly-answered questions: answering a difficult question incorrectly is not a big deal, but easy questions will impact θ_l’ when answered incorrectly. The value of the discrimination parameter α_i impacts the magnitude of updates to θ_l’. Highly-discriminative items have large values of α_i, leading them to meaningfully impact the value of θ_l’ and vice versa.

Estimating performance. Instead of measuring performance with accuracy metrics, Fluid Benchmarking directly uses the value of θ_l’ as the performance metric for a model. While accuracy simply captures the ratio of items answered correctly in a benchmark, Fluid Benchmarking asks an inverse question: What capability level of our model is most likely to produce the pattern of incorrect and correct answers we observed? By answering this question, we can estimate performance in a way that meaningfully considers the difficulty and discrimination of each item in our evaluation set. Raw accuracy on a discrete evaluation dataset is a common proxy for measuring model capabilities. However, Fluid Benchmarking [12] forgoes this proxy, instead using IRT to directly estimate model capabilities.

Dynamic sampling. The final detail necessary to understand Fluid Benchmarking is the data selection process. As mentioned previously, we use a subset of real evaluation results to estimate the parameter for a new model θ_l’. These items that are evaluated could be taken from a static evaluation set—this is a common approach in practice. However, Fluid Benchmarking argues that the set of items used for evaluation should be dynamically selected based on the model. For a weaker model, easier items will be more informative and vice versa.

Evaluation items are selected in [12] by computing the Fisher information of each item in the dataset. This metric prioritizes items that are most informative for a particular model by considering i) item discrimination and ii) item difficulty with respect to the capability level of the model being evaluated. Notably, the Fisher information changes depending on the capability level of a model. The figure above illustrates changes in the Fisher information during the training process. As the model continues training, it becomes more capable, leading to changes in the Fisher information that prioritize the selection of more difficult examples.

To select evaluation data based on the Fisher information, authors in [12] propose the following set of steps:

1. Start with an empty evaluation set.

2. Compute the Fisher information of all items.

3. Select the item with the highest Fisher information.

4. Compute the true evaluation score of this item.

5. Re-fit the model parameter using this new data.

6. Repeat the above steps until your evaluation budget is reached.

While most LLM evaluations are static, Fluid Benchmarking is dynamic—the data used for evaluation is adapted based on each model being evaluated. Such an approach demonstrates the incredible potential of IRT for both selecting data and measuring performance, as well as its overall versatility as a tool. Notably, a very similar data selection approach is adopted by the more recent ATLAS paper.

Does this work? In [12], authors focus on evaluating model checkpoints during the pretraining process. Six different open LLMs in the 7B parameter range are selected, and checkpoints are taken from these models evenly throughout their training process to arrive at a set of 102 LLMs for fitting the IRT model. All evaluation experiments are performed on the Open LLM leaderboard, which is a composite leaderboard comprised of six different benchmarks. A separate IRT model is fit for each benchmark in the leaderboard. As shown below, Fluid Benchmarking provides a stable and accurate estimate of model capabilities and is found to be effective for a wide range of different evaluation budgets.

DatBench: Discriminative, Faithful, and Efficient VLM Evaluations [10]

“We identify critical failure modes that violate faithfulness and discriminability, misrepresenting model capabilities: (i) multiple-choice formats reward guessing, do not represent downstream use-cases, and saturate early as models improve; (ii) blindly-solvable questions which can be answered without images, constitute up to 70% of some evaluations; and (iii) mislabeled or ambiguous samples compromise up to 42% of examples in certain datasets.” - from [10]

Most popular Vision-Language Model (VLM) benchmarks have limitations that make research and progress difficult. Problems with these benchmarks include:

• Data quality issues (e.g., incorrect labels or low-resolution images) that make solving certain questions overly difficult or impossible.

• Blindly-solvable questions that can be solved purely based upon text priors without using the actual image.

• Multiple choice questions that are easily reward hacked via guessing and do not much the generative style in which most VLMs are deployed.

Beyond these issues, the evaluation process alone is beginning to consume non-negligible compute for most models. LLM research is empirical, and as much as 20% (or even more) of total model development costs can be spent running evaluations. Based on this trend, we want to avoid wasted compute and ensure that the data in these benchmarks is actually useful for discerning model capabilities. Authors in [10] aim to solve these issues by developing and applying a targeted data curation approach over a wide set of VLM benchmarks to create DatBench, a composite benchmark that prioritizes high signal evaluation examples for VLMs.

Source data. The curation process in [10] begins with a large set of 33 evaluation datasets for VLMs that span the capability groups depicted above. A set of 27 state-of-the-art models ranging from 1-10B parameters is evaluated over these datasets, yielding a dataset of model evaluation results to use for data curation. From here, DatBench is constructed via a multi-step filtering process:

1. Converting multi-choice questions into a generative format.

2. Removing blind-solvable questions.

3. Filtering examples with incorrect or ambiguous ground truth.

4. (Optional) Identifying examples that yield maximum discrimination.

The last step of the pipeline is optional but can be used to sample a smaller amount of data that retains the ability to detect differences in model capabilities. Two different evaluation suites are created in [10]—DatBench and DatBench-Full—that cover distinct evaluation modes:

• High-efficiency evaluation over a subset of data for rapid iteration.

• High-quality evaluation over all data for cases with relaxed computational constraints and a need for better coverage.

For example, DatBench is most useful for ablation experiments, as we can lower inference costs and run faster experiments while still providing a useful capability signal. On the other hand, DatBench-Full can be used for final model reporting, which is run less often but requires comprehensively capturing the performance of a model. We will now outline each of the above curation steps in more detail.

Multiple choice to generative conversion. Practically, most VLMs are used in a generative fashion, where users ask questions to a model and the model generates a response for the user. However, many benchmarks used to evaluate VLMs ask question in a multiple choice format. Such a format can artificially inflate VLM performance due to the random guessing and the fact that selecting an answer is generally easier than generating that same answer from scratch.

DatBench reformulates multiple choice questions into a generative format where the VLM generates an answer that is verified against a ground truth answer using an LLM judge. In cases where multiple choice is structurally necessary, authors in [10] rely upon a circular evaluation approach. As shown in the figure above, converting multiple choice questions into a generative format leads to a noticeable drop in model performance, indicating that generative evaluation is harder for current VLMs and better reflects the current state of model capabilities.

Removing blind-solvable questions. One key insight from [10] is the fact that a surprising number of VLM evaluation samples can be solved without using any visual data; see below. Models can rely upon language priors to solve questions (or provide a high-probability guess), thus inflating the performance of VLMs with strong language backbones. To identify these cases, we can re-run evaluation while removing image inputs to identify those that are blind-solvable. In [10], the entire suite of 27 models is run in a blind fashion, and any questions that can be solved by at least one model are removed. Though this filtering approach is aggressive, the likelihood of a correct blind answer in a generative setup is relatively low, and the curation process begins with a large source dataset.

Quality filtering. A two-stage pipeline is used in [10] to identify incorrect, low quality, and ambiguous evaluation data; see below. In the first stage, we flag any evaluation examples that are not solved by any model in the suite. These samples are usually either i) a data quality issue or ii) a valid frontier evaluation case.

To differentiate between these cases, we perform a second stage of filtering based upon a frontier-level VLM judge. In this stage, every flagged example is passed through the judge to determine whether it is correct an unambiguous. Such an approach is reliant upon the asymmetry of verification (i.e., verifying a provided solution to a problem should be easier than generating a valid solution).

In an effort to prioritize quality over quantity, any data identified as ambiguous, incorrectly labeled, or unsolvable due to insufficient image resolution is removed. As shown above, this stringent filtering policy results in relatively high ratios of discarded data in certain domains. For example, over 42% of the spatial reasoning data is removed from DatBench due ambiguity or issues with data quality

Discriminative selection. Given increasing costs of evaluation, we would like to sample an evaluation subset to reduce costs without degrading discriminability—or the ability to identify differences in performance. One common approach is to sub-select evaluation samples while optimizing for rank correlation to find a smaller evaluation dataset that ranks models in the same way. However, this approach is prone to overfitting on a particular evaluation suite. An evaluation subset can preserve model rankings while still having noisy data that does not genuinely capture difference in model capabilities—we prioritize model rankings without deeply capturing the kind of data that is actually being selected.

“The core optimization problem is not merely to maintain ranking stability, but to maximize total discrimination. By ensuring every sampled example possesses high discriminative power, we can implicitly guarantee robust ranking while maximizing the information content per inference token.” - from [10]

Authors in [10] propose a solution to these problems that is based upon IRT. Directly applying IRT to VLM evaluation would work poorly, as we do not have enough data. Specifically, each data point would need to be evaluated with hundreds of different models in order to fit a stable IRT model. We do not have anywhere near this amount of data—only 27 models are used in [10] and getting access to hundreds of state-of-the-art VLMs would be very difficult (if not impossible).

Instead of directly using IRT, data in [10] is selected based on information density, as captured by the point-biserial correlation (r_pb); see above. Computed per evaluation example, r_pb captures the relationship between scores on a single data point and global performance. As explained in [10]: “An item with high r_pb is one that strong models consistently answer correctly and weak models consistently miss; conversely, a low or negative r_pb indicates a noisy item.” The left term in the above equation captures the relative difference in global performance of models that get a given data point correct or incorrect, while the right term captures the ratio of models that get the data point correct or incorrect.

We select evaluation data in [10] by prioritizing examples with high r_pb per domain. To measure the total discriminative power of an evaluation subset, we can divide the total sum of selected r_pb score by the sum of r_pb scores across all data. As shown above, selecting data based upon r_pb allows us to preserve 90% of total discriminability with only 40% of the data, whereas rank correlation metrics saturate almost immediately. Interestingly, we also see that selecting all data is not optimal from the perspective of discriminative power. Noisy data (i.e., with low or negative r_pb) is left until the end of the selection process in [10].

The IRT-inspired approach is used to select 80% of evaluation data in [10], while the final 20% is manually reserved for frontier examples with low discriminative power. Namely, there exists a subset of evaluation data that has been validated by the LLM judge but is not answered correctly by any model. Any example in this subset will receive a low r_pb score because of the low ratio of correct model responses. However, such data captures legitimate frontier evaluation scenarios that should not be completely ignored within our evaluation dataset.

Key findings. Evaluation results on both DatBench and the original benchmarks are plotted above. Results on DatBench have a larger performance spread relative to those of the original benchmarks. For example, scores on general benchmarks range from 10-65% for DatBench versus 65-80% for original benchmarks, showing that DatBench mitigates benchmark saturation. In fact, just converting multiple choice questions to a generative format causes as much as a 35% performance drop. DatBench is found to yield a 13× speedup in the evaluation process while roughly matching the discriminative power of the original benchmarks.

We can also repurpose the evaluation artifacts created by the DatBench pipeline to diagnose common failure modes of VLMs. Specifically, authors in [10] make the following observations:

• A tradeoff between perception and reasoning exists in VLMs. Models that perform well on higher-level semantic processing tasks have degraded low-level perceptual fidelity. Finding a model that balances performance on both semantic and perceptual tasks is rare.

• An “overthinking” problem exists within current VLMs, meaning that significantly fewer tokens are used when answering questions correctly versus incorrectly; see above. This problem is especially pronounced in reasoning models, where we see that an average length of a correct and incorrect response is 425.2 and 1,196.9 tokens, respectively.

• The dependence of VLMs upon language priors, which can be measured via the performance difference between normal and blind evaluation, varies per capability; see below. For example, counting and grounding rely heavily upon vision information, but math and spatial reasoning are found to rely more upon language priors to guess a correct answer.

Although many VLM benchmarks are shown to be noisy and inflated in [10], we can learn a lot about current state-of-the-art by addressing these problems and selecting evaluation data that accurately captures model performance. Once we can identify shortcomings in performance (e.g., overthinking and perceptual gaps), improving model capabilities in these specific areas is much easier.

Keys to Creating a Useful Benchmark

We have studied a wide variety of LLM benchmarks and evaluation techniques in this overview. Given the many practical details peppered throughout the papers we have seen, we can gain a lot by considering the common concepts that continually arise across disparate benchmarks. By identifying these trends, we can (hopefully) identify key design principles for making a useful benchmark.

Domain Taxonomy. Most popular LLM benchmarks categorize their data into a fixed set of domains and sub-domains. By doing this, we make it easier to debug an LLM’s performance, as we can compute domain-level metrics within the benchmark. Additionally, organizing a benchmark into such a taxonomy naturally ensures that data is diverse and covers a decent breadth of topics. Leveraging a taxonomy can also make the evolution of a benchmark simpler over time by granularly measuring saturation at a domain level and enabling researchers to individually evolve each domain (e.g., as in BIG-Bench Extra Hard).

Human annotation. Despite the prevalence of synthetic data within LLM research, nearly all successful evaluation benchmarks rely on human experts to annotate data in some way. Some benchmarks begin with questions written by human experts (e.g., FrontierMath), while others leverage human opinions to measure question difficulty or accuracy (e.g., GPQA). Even when synthetic data is being used, human verification of data quality is usually helpful (e.g., IFEval and IFBench). In fact, review by human experts is even used in some cases to improve the quality of large-scale data obtained from noisy sources (e.g., crowdsourcing). Even today, manual inspection is one of the most effective tools for LLM evaluation.

Model-in-the-loop. Although humans play a massive role in the evaluation process, augmenting human efforts with an LLM can be beneficial. For example, LLMs are often used for difficulty filtering by simply identifying the questions that they get wrong. Additionally, trends in model performance allow us to fit IRT models and even identify less informative subsets of data (e.g., blind-answerable data in DatBench). Model-based approaches are helpful for identifying areas of a benchmark that may contain mistakes that can be routed to human review. We can also use LLMs to efficiently generate or reformat evaluation data that is later verified by a human annotator (e.g., MMLU-Pro adopts such a strategy).

Data quality. The best evaluation benchmarks tend to pull from high-quality data sources. For example, popular math benchmarks include questions that are taken directly from recognized math competitions, and reasoning benchmarks like BIG-Bench are sourced from vetted sources such as other proven datasets (as in BIG-Bench Extra Hard) or questions that have been extensively verified with human review (as in the original BIG-Bench). In fact, manually written questions from human experts are another commonly-used source of evaluation data, but we must implement measures to ensure data quality. The GPQA curation pipeline is a great example of an effective system for ensuring data quality and difficulty.

Realistic. Benchmarks are an imperfect proxy for measuring what we actually care about: the capabilities of an LLM. Depending on the questions that it tests, a benchmark may or may not accurately reflect the true performance of an LLM in the real world. Ideally, we want our benchmark to accurately capture an LLM’s true capabilities on a given task. To achieve this, we should make sure that evaluation data is as realistic as possible. One great example of how to achieve this goal is CursorBench, a coding benchmark that directly sources evaluation data from real coding agent sessions in Cursor and constantly releases new benchmark versions to better capture recent trends in agent usage.

Evolution. The capabilities of frontier-level LLMs are advancing rapidly, which can lead to benchmark saturation. In order to remain relevant, a good benchmark must evolve (and improve) over time. One of the best examples of this trend is BIG-Bench, which was already saturated less than a year after its initial release. Instead of simply allowing the benchmark to become irrelevant, improved versions were consistently released, such as BIG-Bench Hard and BIG-Bench Extra Hard. Many datasets can remain relevant and useful if we are willing to adjust the difficulty and scope of the benchmark as LLMs improve.

New to the newsletter?

Hi! I’m Cameron R. Wolfe, Deep Learning Ph.D. and Staff Research Scientist at Netflix. This is the Deep (Learning) Focus newsletter, where I help readers better understand important topics in AI research. The newsletter will always be free and open to read. If you like the newsletter, please subscribe, consider a paid subscription, share it, or follow me on X and LinkedIn!

Subscribe now

Bibliography

[1] Hendrycks, Dan, et al. “Measuring massive multitask language understanding.” arXiv preprint arXiv:2009.03300 (2020).

[2] Wang, Yubo, et al. “Mmlu-pro: A more robust and challenging multi-task language understanding benchmark.” Advances in Neural Information Processing Systems 37 (2024): 95266-95290.

[3] Gema, Aryo Pradipta, et al. “Are we done with mmlu?.” Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.

[4] Rein, David, et al. “Gpqa: A graduate-level google-proof q&a benchmark.” First conference on language modeling. 2024.

[5] Srivastava, Aarohi, et al. “Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.” Transactions on machine learning research (2023).

[6] Suzgun, Mirac, et al. “Challenging big-bench tasks and whether chain-of-thought can solve them.” Findings of the Association for Computational Linguistics: ACL 2023. 2023.

[7] Kazemi, Mehran, et al. “Big-bench extra hard.” Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

[8] Zhou, Jeffrey, et al. “Instruction-following evaluation for large language models.” arXiv preprint arXiv:2311.07911 (2023).

[9] Pyatkin, Valentina, et al. “Generalizing verifiable instruction following.” arXiv preprint arXiv:2507.02833 (2025).

[10] Joshi, Siddharth, et al. “DatBench: Discriminative, Faithful, and Efficient VLM Evaluations.” arXiv preprint arXiv:2601.02316 (2026).

[11] Polo, Felipe Maia, et al. “tinyBenchmarks: evaluating LLMs with fewer examples.” arXiv preprint arXiv:2402.14992 (2024).

[12] Hofmann, Valentin, et al. “Fluid language model benchmarking.” arXiv preprint arXiv:2509.11106 (2025).

[13] Dubois, Yann, et al. “Length-controlled alpacaeval: A simple way to debias automatic evaluators.” arXiv preprint arXiv:2404.04475 (2024).

[14] Vivek, Rajan, et al. “Anchor points: Benchmarking models with much fewer examples.” Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

[15] Xu, Cong, et al. “Data efficient evaluation of large language models and text-to-image models via adaptive sampling.” arXiv preprint arXiv:2406.15527 (2024).

[16] Perlitz, Yotam, et al. “Efficient benchmarking (of language models).” Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024.

[17] Kipnis, Alex, et al. “metabench--A Sparse Benchmark of Reasoning and Knowledge in Large Language Models.” arXiv preprint arXiv:2407.12844 (2024).

Research on large language models (LLMs) is empirically driven. For this reason, model evaluations play a pivotal role in the field’s progress. We improve models by making changes, evaluating them, and iterating. Despite their foundational role, however, evaluations are usually handled in a naive manner. In most cases, we just test a model’s performance over a finite evaluation dataset and directly compare performance metrics to those of other models with no consideration for whether these results are statistically significant or not. Such an approach leads to incorrect or misleading interpretations of evaluation results. As researchers, we want to avoid mistaking noise for progress and instead equip ourselves with the statistical tools needed to run informative model evaluations.

“Language models are measured in the literature by evaluations, or evals. Evals are commonly run and reported with a highest number is best mentality; industry practice is to highlight a state-of-the-art result in bold, but not necessarily to test that result for any kind of statistical significance.” - from [1]

In this overview, we will build a statistical foundation for LLM evaluations from the ground up. To begin, we will review basic statistical ideas with a practical focus on the topics that are most useful for model evaluations. We will then take a deeper look at how these ideas can be directly used to interpret LLM evaluation results in an uncertainty-aware manner. Specifically, we will cover a set of statistical best practices for model evaluation and implement each of them to show how they can be concretely applied. Although it may seem daunting, taking a statistically grounded approach to model evaluation is not especially difficult and can help us make faster progress by avoiding spurious results.

Basic Statistics for LLM Evaluations

In order to develop a statistical framework for LLM evaluations, we need to first learn about the fundamental tools from statistics that can be used to create such a framework. This section will cover a selection of topics related to the properties of random variables, such as computing the mean or variance and constructing a confidence interval. After covering the fundamentals, we learn how these ideas can be applied to properly analyze LLM evaluation results in the next section.

Random Variables and Estimators

A random variable X is defined as a quantity that has a value dependent upon chance. We can take several independent samples from the distribution {x_1, x_2, …, x_n}, and the values of these observations will be sampled from the distribution of X (i.e., x_i ~ X). We define the mean (or average) of this random variable via the expectation, which can be computed in a continuous or discrete fashion as shown in the figure below. Additionally, we can compute a sample mean by averaging the values of n observations sampled from the distribution.

Formally, the lower case letters x_i represent concrete values sampled from a distribution, while upper case letter X_i denotes the i-th random variable in our sample—this is a notational detail, but it’s worth covering to avoid confusion. For example, if we evaluate our LLM on n questions, X_i is a random variable that represents the distribution of possible scores for question i1, while x_i is an actual evaluation score observed for a single evaluation run. We can also define the sample mean in terms of random variables as shown in the equation below. We use an uppercase X̄ in this case because we are defining a random variable.

The distribution of our random variable X also has variance Var(X), which describes how “spread out” the distribution is around the mean. In this overview, we will assume that this variance is finite (i.e., less than infinity). If we have a distribution with high variance, then samples taken from this distribution will be more spread out around the mean and vice versa; see below for an illustration.

The expression for Var(X) is provided below. Similarly to the sample mean, we can also estimate variance using a fixed set of samples from our distribution X—this is how the variance is usually computed in practical settings. We can also compute the standard deviation σ by taking the square root of the variance. The variance and standard deviation describe the variability of individual samples from X.

While variance measures the variability of a single random variable X, covariance measures how two random variables X and Y vary together. Intuitively, if these variables vary in the same direction (e.g., they are both above or below their means at the same time), then their covariance will be positive and vice versa. A covariance near zero indicates there is no clear relationship between X and Y. We can also compute a sample covariance similarly to the sample variance shown above. Expressions for covariance and sample covariance are provided below.

The law of total variance is a useful identity that decomposes the variance of a random variable X with respect to another random variable Y; see below. For the purposes of this overview, this law is useful because it lets us separate multiple sources of randomness in an evaluation result. Later, we will use it to decompose the variance of an evaluation score into two key components:

1. Variability due to the question sampled for evaluation.

2. Within-question variability arising from stochastic generation by the LLM or an LLM judge.

Standard Error and Sample Means

If we repeatedly draw samples from X and compute the sample mean, we will get a different result every time. The resulting sample means form a sampling distribution (i.e., basically a list of sample means we have drawn). The standard deviation of this sampling distribution is called the standard error of the sample mean. Put simply, the standard error is just the standard deviation over sample means. While the standard deviation captures variability in individual data points x_i sampled from X, the standard error captures variability in the sample mean estimator (i.e., the spread of sample means after computing it multiple times with different samples). A formal definition of the standard error is provided below, as well as an estimator for the standard error that uses sample standard deviation2 because the true value of σ is rarely known in practice and must be estimated.

This standard error equation makes the assumption that samples drawn from X are independent and identically distributed (IID). Independence implies that Cov⁡(X_i,X_j) = 0 for i≠j, and identical distribution implies that each X_i​ has the same variance Var(X). From this assumption and a few other properties of the variance, we can derive the above expression for the standard error as shown below. The assumption of IID samples is not always satisfied—we should only use this expression when the samples being drawn are truly independent.

Within this derivation, we use the variance of a sum identity, which can be generally expressed as shown in the equation below. This identity allows us to capture the (non-zero) covariance terms within our variance expression.

Bernoulli variables. Let’s assume X is a Bernoulli random variable, meaning that our scores are binary x_i ∈ {0, 1}. In this case, our standard error expression can be simplified even further. To begin, we know that E[X] = 1×P(X=1) + 0×P(X=0) = P(X=1). Given that the values of X are either zero or one, it is also true that E[X^2] = E[X] because x^2 = x when x = 0 or x = 1.

We can easily plug these two identities into our prior expression for the variance Var(X) = E[X^2] - (E[X])^2 = Pr(X=1) - (Pr(X=1))^2 = μ(1 - μ), where μ is the mean of X. Practically, we can estimate μ by taking a sample mean X̄. Then, we can plug this simplified Var(X) into our previous formula for the standard error, yielding the simplified expression shown below. Therefore, we can use this simpler standard error expression when the values of X are binary.

Law of Large Numbers and the Central Limit Theorem (CLT)

The law of large numbers is a fundamental concept in statistics that builds upon our prior definition of the sample mean. Given a random variable X, we are often interested in its true mean μ. This mean can be estimated with the sample mean over n samples, but this is a random estimate that can differ from μ. The law of large numbers tells us that as the value of n increases, the sample mean will approach (i.e., converge in probability) the true mean μ; see below.

The law of large numbers only tells us that the sample mean will eventually settle around μ with sufficiently large n. It does not tell us how much the sample mean differs from the true mean at finite n or how quickly we converge to μ as n increases. We can express the intuition for the law of large numbers as follows: with enough data, our estimator (i.e., the sample mean) approaches the true mean.

Standardization and z-score. Given a random variable X (or a realized value x), we can standardize by subtracting the mean μ and dividing by the standard deviation σ; see below. This process produces a standardized random variable Z (or a realized value z). The z-score z3 indicates how many standard deviations—in units of σ—the value x lies above (z > 0) or below (z < 0) the mean μ.

Any variable or value can be standardized in this way. For example, we will next standardize the sample mean while formulating the Central Limit Theorem.

The Central Limit Theorem (CLT) goes beyond the law of large numbers by describing how our sample mean estimates will be distributed around the true mean μ. Our random variable X has a mean of μ, and we estimate this mean with a sample mean. We know from our prior derivation that this sample mean has a variance of σ^2 / n (assuming IID random variables and finite variance σ^24).

Using this mean and variance, we can standardize the sample mean to obtain Z_n by subtracting the mean and dividing by the standard error; see below. The denominator of Z_n is our previous equation for standard error—this is just the standard deviation of our sample mean! We rarely know the actual value of σ, so we can estimate the true value with the sample standard deviation s.

The CLT tells us that the distribution of Z_n will converge to a standard normal distribution5—meaning a normal distribution with a mean of zero and variance of one—as the value of n increases. Stated differently, this means that the distribution of our sample mean becomes approximately normal with sufficiently large n, as shown in the orange distribution above. From this information, we know that the standard deviation of the sample mean distribution will decrease proportionally to 1 / sqrt(n) and the error of our sample mean estimate is on the order of σ / sqrt(n)—the standard deviation of the above distribution.

Confidence Intervals

Consider a random variable X with a true mean μ that we estimate with the sample mean X̄_n computed from n samples. To quantify the uncertainty of this estimate, we can next compute a 95% confidence interval that has the following form: x̄_n ± y. This confidence interval indicates that if we repeated the sampling procedure many times and recomputed this confidence interval each time, 95% of the resulting confidence intervals would contain the true mean μ. Our goal is to find the value of y that statistically yields such a 95% confidence interval. To find a formula that allows us to compute this confidence interval, we actually need to combine all of the ideas we have learned so far.

First, let’s consider our sample mean estimator X̄_n. Assuming IID samples with finite variance, we know from the CLT that this estimator has an approximately normal distribution N(μ, σ^2 / n) assuming that the value of n is sufficiently large, as well as a standard error given by SE(X̄_n) = σ / sqrt(n). When computing a 95% confidence interval, we consider a normal distribution N(0, 1) and try to find a bound that includes 95% of the probability mass for this distribution; see below for an illustration.

Given a standard normal distribution, we have P(|Z| < 1.96) = 0.95. This is a two-sided confidence interval, meaning 2.5% of the total 5% of probability mass outside our confidence interval is allocated to each side of the distribution. In most cases, however, we will want to compute a confidence interval for a non-standard normal distribution. To do this, we can just standardize the distribution as discussed previously. For example, given our distribution N(μ, σ^2 / n) from the CLT, we can derive a standardized variable Z that follows a standard normal distribution. From here, we can just transform the confidence interval with the same standardization process; see below.

This approach yields a formula—based upon our sample size and the standard error of our sample mean—that can be used to compute a 95% confidence interval.

A Statistical Approach to LLM Evaluations [1]

Now that we have built a solid statistical foundation, we can use these ideas to create a framework for LLM evaluations that better quantifies uncertainty. In doing this, we can be more confident in our model evaluations and understand whether certain evaluation results are legitimate or just caused by noise. Our discussions will be based on a seminal paper from Anthropic [1] that provides several key recommendations for performing LLM evaluations in a way that is grounded in statistics, rather than just comparing raw performance metrics.

“Fundamentally, evaluations are experiments; but the literature on evaluations has largely ignored the literature from other sciences on experiment analysis and planning. This article shows researchers with some training in statistics how to think about and analyze data from language model evaluations.” - from [1]

Statistical framing for LLM evaluations. In theory, when evaluating an LLM, there exists a super-population of questions (illustrated below) that exhaustively covers all the ways in which the LLM can be evaluated. Practically speaking, any evaluation dataset represents only a finite subset of questions from this super-population, as represented by the red shaded region in the figure below.

This framing can be used to rethink our perspective on model evaluations. Instead of trying to maximize the performance of our model on a finite benchmark, we should be trying to improve an underlying skill of the model. Any evaluation dataset captures a corresponding skill imperfectly, as it is only a finite sample from the super-population that is associated with that skill.

Key recommendations. There are a set of concrete recommendations proposed in [1] that outline how one can approach LLM evaluations in a rigorous manner. We first outline these recommendations here, then spend the rest of this section explaining each of them in more depth:

1. When questions are IID, LLM evaluation results should be accompanied by standard errors that are computed using the CLT.

2. If questions are not IID (e.g., drawn from related clusters or groups), then our CLT standard error formula is no longer valid and we should instead compute a clustered standard error.

3. To reduce the variance of evaluation results, we can re-sample outputs from the LLM multiple times—or even analyze next token probabilities—to better account for the variance of each individual evaluation result.

4. When comparing two models, we can perform analysis of their paired difference (i.e., rather than just providing separate, aggregated evaluation scores over the dataset) to yield a more confident result.

Preliminaries. The evaluation dataset in [1] is assumed to contain n questions, and each question receives an evaluation score s_i; e.g., a binary correctness signal or an LLM-as-a-Judge score. A score can be decomposed as s_i = x_i + ϵ_i, where x_i is the expected score (i.e., E[s_i] = x_i) and ϵ_i adds randomness to the score. We assume zero-mean randomness (i.e., E[ϵ_i|i] = 0) that does not change the expected score. Put simply, this setup models a non-deterministic evaluation setting. Notably, LLM evaluation is fundamentally non-deterministic, as it involves sampling from the next token distribution of one or more LLMs (i.e., the model being evaluated and possibly an LLM judge).

Standard Errors and the CLT

The simplest case when analyzing evaluation results is when each question i is independent. Our goal in analyzing an evaluation result is to understand the true performance of our model, represented by the mean score μ = E[s] = E[x]6 from our super-population. We only have access to a finite set of scores from our evaluation dataset. However, we know from the law of large numbers that we can estimate the true mean by taking a sample mean s̅ over a finite set of evaluation scores. This estimator approaches μ as the value of n becomes larger.

In other words, taking an average score over a large number of independently-sampled questions generally provides a good estimate of a model’s true performance. However, “good” is difficult to quantify, and how do we know if n is sufficiently large? To quantify uncertainty, we can use the CLT to compute the standard error for our sample mean; see above. As we can see, this expression is identical—other than switching x with s—to our previously-derived standard error expression. We can also derive a confidence interval from the standard error similarly to before.

If we assume a Bernoulli distribution—meaning that for all i we have s_i ϵ {0, 1}—this expression can be simplified even further; see above. However, the Bernoulli formula requires that scores are truly binary (i.e., not fractional7).

“We suggest reporting the standard error of the mean alongside (beneath) the mean when reporting eval scores.” - from [1]

Now that we know how to compute these quantities for an LLM evaluation, the recommendation in [1] is simple: just report this standard error and the number of samples n alongside the actual evaluation result. Computing this standard error is not difficult—it requires forming a sample estimate of the standard deviation of s. A toy example of the proposed reporting structure for two models evaluated over three evaluation datasets is provided in the table below for reference.

From the standard error, we can compute a confidence interval for each model’s evaluation metric. These intervals summarize uncertainty in the estimated mean performance. When comparing models, non-overlapping confidence intervals suggest a real performance difference, but overlapping intervals do not by themselves rule one out. A precise comparison requires directly analyzing the difference between the models, which we will handle in a future section.

As an example, confidence intervals for the table above have been computed below for all model and dataset combinations. We see here that all models have overlapping confidence intervals. In future sections, we will learn methods that can be used to compare models with a greater level of precision.

Bootstrapping is another common approach to use for evaluating machine learning models (including LLMs) that proceeds as follows:

1. Sample n question scores with replacement.

2. Compute the sample mean s̅.

3. Repeat steps 1-2 multiple times.

4. Measure the standard deviation of these sample means.

5. Use this standard deviation as an estimate of the standard error.

While this approach is valid and commonly used in LLM evaluations, authors in [1] argue that bootstrapping is unnecessary when the CLT is valid. Therefore, we can just use the CLT when questions are sampled independently, n is sufficiently large, and the variance of our scores is finite. However, the CLT does fall short when n is small—the handling of this evaluation regime is discussed extensively in [2].

Clustered Errors

“We show how to use clustered standard errors, a technique developed in the social sciences, to account for the dependence and correlation structure present in question clusters.” - from [1]

If questions are not sampled independently, the standard error expression from the CLT is no longer valid. In this case, the CLT underestimates uncertainty—our confidence intervals are too narrow. We are evaluating on n questions, but some of the questions are actually related to each other. As a result, the “effective” number of evaluation questions is smaller than n, thus increasing the standard error. Some practical examples of non-independent questions include:

• The same prompt in different languages.

• Prompts that reference the same document or source.

• Questions that are generally related in format or topic.

To avoid underestimating uncertainty, authors in [1] recommend using a clustered standard error. We use s_{i, c} to denote the score for question i in cluster c. The cluster-adjusted standard error assumes that clusters are independent: questions in a cluster can be correlated, but questions across clusters cannot.

To evaluate an LLM on these clusters, we still compute the sample mean across all question scores S̅, but we modify our standard error expression. Before, we assumed that scores S_i were IID, which implies that Cov(S_i, S_j) = 0 when i ≠ j. When questions are clustered, we no longer have zero covariance, so we need to adjust our derivation of the standard error; see below.

The above clustered standard error equation interpolates between two cases:

1. Scores within a cluster are perfectly correlated and each cluster is treated as if it were a single question i.

2. Scores within a cluster have no correlation, so our expression reduces to the original standard error expression from the CLT.

When questions are not sampled independently, authors in [1] recommend reporting cluster-adjusted standard errors, as well as the number of questions n and the number of clusters C; see below. Similarly to before, the cluster-adjusted standard error can be used to compute a confidence interval. In practice, the clustered standard error may be drastically larger than the CLT standard error. For example, authors provide a concrete example in [1] where the standard error increases by 3× when accounting for clusters. Failing to consider whether questions are actually independent can drastically impact the interpretation of evaluation results.

We assume that questions are sampled independently in future sections unless stated otherwise. However, we can use similar steps as outlined above to derive most results in a cluster-adjusted fashion. Many of the derivations extend to the clustered setting once the covariance structure is accounted for appropriately.

Reducing Variance

We now understand how to compute standard errors and confidence intervals for our evaluation results. The next reasonable question to ask is: What can we do to reduce the standard error? First, recall that our evaluation score is defined as s_i = x_i + ϵ_i, where we have E[s_i] = x_i and Var(ϵ_i) = σ_i^2. To answer this question, we begin with our expression for the standard error and perform a decomposition with the law of total variance; see below.

To apply the law of total variance, we use the following two random variables:

• A random variable over evaluation scores S.

• A random variable over the question that gets sampled I.

We apply the law of total variance by conditioning S on I, where X_I = E[S|I] is the expected score for the sampled question I. We can then further simplify the equation using known properties of the mean and variance of a score.

This derivation yields the variance expression shown above, which provides some actionable insights. First, we see that the simplest method for reducing variance is simply increasing n—evaluating over a larger set of questions naturally improves reliability. Additionally, Var(x) captures the variability in the mean score across our evaluation dataset—this is a fundamental property of our super-population that cannot be easily changed. In simple terms, this quantity captures the spread in question difficulty across all possible evaluation questions. However, there are several approaches we can explore for decreasing the value of E[σ_i^2].

Resampling can be used to reduce score variance when evaluating any model. Instead of generating and scoring a single output per question, we generate and score K outputs for the same question i (i.e., by sampling multiple completions from the LLM). In [1], authors assume that resampled scores for a fixed question i are IID. After sampling K scores, we can take an average of the scores S̅_i, which decreases the score variance by a factor of K; see below for a full derivation.

Therefore, resampling—or producing K scores for question i to yield a mean score S̅_i—provides a linear reduction of the within-question variance σ_i^2 compared to using a single score. The variance for our sample mean has two key terms—Var(x) and E[σ_i^2]—that are summed together in the numerator. As mentioned before, Var(x) is not mutable, so to reduce variance we can—in addition to increasing n—increase the value of K until E[σ_i^2 / K] ≪ Var(X). By doing this, the within-question variance term shrinks toward zero and the variance of our sample mean approaches Var(x) / n; see below.

Token probabilities. If an evaluation metric can be computed from the model’s next token probabilities, we can replace a sampled score with its conditional expectation—basically just the probability of the correct response—and remove the within-question variance (i.e., meaning that σ_i^2 = 0). Using output token probabilities, we can easily compute the probability of a response from our LLM. For example, if our response is just a single token (e.g., a multiple choice answer), then we know that the probability for this score is the probability of that token within the LLM’s next token distribution. If our response is more complex (i.e., multiple tokens), then we can also compute the probability of the entire response via the product of probabilities for each individual token; see below.

For a question i, we will refer to the probability of a response to this question as p_i. If we have access to this probability, then we can use s_i = x_i = p_i, and the variance term for our score goes away (i.e., σ_i^2 = 0). As a result, directly using token probabilities is an effective variance reduction technique. In [1], authors also recommend against changing the sampling temperature—both for resampling and with token probabilities—because this alters the underlying response distribution and, in turn, the evaluation target. For this reason, these results study a different model configuration that is not fully comparable to our original LLM.

Going further, we should note that this approach cannot be used in all cases. First of all, many closed LLMs do not provide direct access to token probabilities. Even if these probabilities are available, using them to compute p_i can be complex depending on the evaluation setup. For example, long-form responses with many tokens—though their probability can be computed—will usually be evaluated with an LLM judge, which uses a sampling procedure of its own and, therefore, adds variability into the resulting score. Additionally, recent reasoning models output a reasoning trajectory alongside their final response, which makes computing the output probability more complicated. In these cases, correctly computing p_i is not straightforward, and we cannot assume zero variance by setting x_i = p_i. In these cases, the resampling strategy described above is a better approach.

Model Comparisons

Now that we deeply understand how to analyze an evaluation score for a single model, we need to focus more on properly comparing the evaluation results of multiple different models. Usually, the goal of evaluation is to understand the performance of a model with respect to other models; e.g., determining if a new model version is better than the current or creating a leaderboard of the best models for a certain evaluation task. Although the techniques we have learned about so far can be applied to comparing evaluation results, we can usually make comparisons more statistically efficient by performing a pairwise analysis.

Difference of means. As we saw when learning about standard errors and confidence intervals, a common comparison heuristic is to compute separate confidence intervals for multiple models and check whether they overlap. If two 95% confidence intervals do not overlap, then there is a statistically significant difference between the evaluation results. As we will see, however, this test is actually overly conservative for detecting performance differences—intervals can overlap even when there is a statistically significant difference in mean scores. Instead, we can analyze the difference in mean between two models; see below. We will refer to the two models being compared as model A and model B for simplicity.

We can compute the standard error of the estimated difference in mean scores; see below. The standard error of the estimated difference in means is the square root of the sum of the variances of the mean estimators for models A and B.

In this derivation, we use the variance of a difference identity, as expressed below. This identity is a special case of the variance of a sum identity we saw previously. In [1], authors consider an unpaired comparison where S̅_A and S̅_B​ are treated as estimates from independent evaluation runs (e.g., computed on independent question samples) such that Cov(S_A, S_B) = 0. This unpaired assumption could be violated (e.g., if models are evaluated over the same set of questions)—we should use the paired analysis from the next section in this case.

We can easily compute a 95% confidence interval using this standard error. To determine if one model is better than the other, we can check whether this confidence interval overlaps with a value of zero. If the 95% confidence interval does not include zero, then—assuming the true difference is zero—there is less than a 5% chance that we would observe a difference this extreme. Our expression for computing a 95% confidence interval has been copied below for convenience.

For model A to outperform model B according to this confidence interval, the difference in the mean score of models A and B must be greater than 1.96 × sqrt(SE_A^2 + SE_B^2). If we compute separate confidence intervals for each model, then this same difference must be greater than 1.96 × (SE_A + SE_B), which is stricter. In this way, checking overlap of separate confidence intervals is conservative, while constructing a confidence interval using the difference—and checking whether it excludes zero—is a better test.

Paired difference. If models A and B evaluate on the same set of questions, we can further reduce variance by analyzing the question-level differences in scores. To begin, we can define question-level paired score differences as shown below.

We can then estimate the standard error of question-level score differences by drawing upon our same standard error expression used previously; see below. We can then use this standard error to compute confidence intervals like before.

We can compute this standard error as shown above, but we are mostly interested in understanding whether this expression provides a meaningful reduction in variance. Ideally, we want the above paired standard error to be smaller than that of the difference of means so that we can better detect statistically significant model differences. To determine if this is the case, we can expand the above variance expression using the variance of a difference identity; see below. Unlike the prior unpaired analysis, we no longer assume that this covariance is zero.

The variance reduction for the above expression depends upon whether the mean scores of models A and B are correlated or not. If so, then this covariance term will be positive and we will see a corresponding reduction in variance. Intuitively stated, a positive correlation indicates that models A and B agree on whether certain prompts are easy or hard (i.e., per-item scores are directionally similar).

“Because eval question scores are likely to be positively correlated, even across unrelated models, paired differences represent a “free” reduction in estimator variance when comparing two models. We therefore recommend using the paired version of the standard error estimate wherever practicable.” - from [1]

In practice, most LLMs tend to agree on per-prompt difficulty, so analyzing paired differences is a useful approach that can offer meaningful reductions in variance. In [1], authors recommend reporting pairwise differences, standard errors, confidence intervals, and score correlations between models; see below.

Practical Implementation

Although we have learned a lot of statistics throughout this discussion, actually implementing these ideas—once we understand them—does not add much extra complexity to the evaluation process. Computing standard errors and confidence intervals is straightforward and, once an implementation is available, can be readily adopted as a standard practice for model evaluations. However, we must be wary of the key assumptions being made when computing the standard error to avoid overconfidence; e.g., questions that are not independent require a cluster-adjusted standard error. A reference implementation of the techniques we have learned so far is provided below. This implementation outlines how all of the recommendations proposed in [1] can be applied when evaluating an LLM.

import math
import numpy as np


#################################
# Evaluation settings and scores
#################################

# model names
model_a_name = "Galleon"
model_b_name = "Dreadnought"

# example scores for the models (n = 10)
model_a = np.array([1, 1, 0, 1, 0, 1, 1, 0, 1, 1], dtype=float)
model_b = np.array([1, 0, 0, 1, 0, 1, 0, 0, 1, 1], dtype=float)

# form toy clusters
# two clusters, assignment based on even / odd index
clusters = np.array([i % 2 for i in range(len(model_a))])

# z score for 95% confidence interval
Z_95 = 1.96


####################
# Utility functions
####################

def mean_score(scores):
    return float(np.mean(scores))

def sample_sd(scores):
    return float(np.std(scores, ddof=1))

def ci_95(mean, se):
    return (mean - Z_95 * se, mean + Z_95 * se)

def fmt_pct(x):
    return f"{100 * x:.1f}%"

def fmt_pct_paren(x):
    return f"({100 * x:.2f}%)"

def fmt_ci(ci):
    return f"({100 * ci[0]:.2f}%, {100 * ci[1]:.2f}%)"


########################################################
# CLT SE
#
# Standard CLT SE for the sample mean: SE = s / sqrt(n)
# where s is the sample standard deviation.
########################################################

def clt_standard_error(scores):
    n = len(scores)
    return sample_sd(scores) / math.sqrt(n)


#####################################################
# Clustered SE
#
# Cluster-adjusted standard error:
# \sqrt{
#     SE_{CLT}^{2}
#     +
#     \frac{1}{n^{2}}
#     \sum_{c}\sum_{i}\sum_{j\neq i}
#     (s_{i,c} - \overline{s})(s_{j,c}-\overline{s})
# }
#####################################################

def clustered_standard_error(scores, clusters):
    n = len(scores)
    s_bar = np.mean(scores)

    # clt variance
    se_clt_sq = clt_standard_error(scores) ** 2

    # within-cluster cross terms
    cross_terms = 0.0
    for c in np.unique(clusters):
        idx = (clusters == c)
        residuals = scores[idx] - s_bar
        cross_terms += residuals.sum()**2 - np.sum(residuals**2)

    var_hat = se_clt_sq + (cross_terms / (n**2))
    return math.sqrt(var_hat)


######################################
# Summary statistics for single model
######################################

def summarize_model(scores, clusters):
    mean = mean_score(scores)

    se_clt = clt_standard_error(scores)
    ci_clt = ci_95(mean, se_clt)

    se_cluster = clustered_standard_error(scores, clusters)
    ci_cluster = ci_95(mean, se_cluster)

    return {
        "mean": mean,
        "se_clt": se_clt,
        "ci_clt": ci_clt,
        "se_cluster": se_cluster,
        "ci_cluster": ci_cluster,
        "n": len(scores),
        "num_clusters": len(np.unique(clusters)),
    }


###########################################################
# Comparing models w/ difference in means and separate SEs
###########################################################

def difference_in_means(scores_a, scores_b, clusters_a=None, clusters_b=None):
    mean_a = mean_score(scores_a)
    mean_b = mean_score(scores_b)
    diff = mean_a - mean_b

    # CLT version
    se_a_clt = clt_standard_error(scores_a)
    se_b_clt = clt_standard_error(scores_b)
    se_diff_clt = math.sqrt(se_a_clt ** 2 + se_b_clt ** 2)
    ci_diff_clt = ci_95(diff, se_diff_clt)

    # Clustered version
    if clusters_a is not None and clusters_b is not None:
        se_a_cluster = clustered_standard_error(scores_a, clusters_a)
        se_b_cluster = clustered_standard_error(scores_b, clusters_b)
        se_diff_cluster = math.sqrt(se_a_cluster ** 2 + se_b_cluster ** 2)
        ci_diff_cluster = ci_95(diff, se_diff_cluster)
    else:
        se_diff_cluster = None
        ci_diff_cluster = None

    return {
        "diff": diff,
        "se_diff_clt": se_diff_clt,
        "ci_diff_clt": ci_diff_clt,
        "se_diff_cluster": se_diff_cluster,
        "ci_diff_cluster": ci_diff_cluster,
    }


##########################################################
# Comparing models with paired instance-level differences
##########################################################

def paired_differences(scores_a, scores_b, clusters=None):
    diffs = scores_a - scores_b
    mean_diff = mean_score(diffs)

    se_paired_clt = clt_standard_error(diffs)
    ci_paired_clt = ci_95(mean_diff, se_paired_clt)

    if clusters is not None:
        se_paired_cluster = clustered_standard_error(diffs, clusters)
        ci_paired_cluster = ci_95(mean_diff, se_paired_cluster)
    else:
        se_paired_cluster = None
        ci_paired_cluster = None

    # Correlation between model scores across questions
    corr = float(np.corrcoef(scores_a, scores_b)[0, 1])

    return {
        "diffs": diffs,
        "mean_diff": mean_diff,
        "se_paired_clt": se_paired_clt,
        "ci_paired_clt": ci_paired_clt,
        "se_paired_cluster": se_paired_cluster,
        "ci_paired_cluster": ci_paired_cluster,
        "correlation": corr,
    }


############################################
# Functions to recreate tables from paper
#
# Numbers will not match exactly because we
# hard-code model scores at top of script
############################################

def print_table_2_style(model_name, summary):
    print(f"| {model_name:12s} | "
          f"mean = {fmt_pct(summary['mean']):>6s}  "
          f"SE = {fmt_pct_paren(summary['se_clt']):>8s}  "
          f"95% CI = {fmt_ci(summary['ci_clt'])}  "
          f"n = {summary['n']}")

def print_table_3_style(model_name, summary):
    print(f"| {model_name:12s} | "
          f"mean = {fmt_pct(summary['mean']):>6s}  "
          f"clustered SE = {fmt_pct_paren(summary['se_cluster']):>8s}  "
          f"95% CI = {fmt_ci(summary['ci_cluster'])}  "
          f"n = {summary['n']}, clusters = {summary['num_clusters']}")

def print_table_5_style(model_name, baseline_name, paired_results, clustered=False):
    if clustered:
        se = paired_results["se_paired_cluster"]
        ci = paired_results["ci_paired_cluster"]
        label = "paired clustered"
    else:
        se = paired_results["se_paired_clt"]
        ci = paired_results["ci_paired_clt"]
        label = "paired CLT"

    print(f"| {model_name:12s} | baseline = {baseline_name:12s} | "
          f"diff = {fmt_pct(paired_results['mean_diff']):>6s}  "
          f"SE = {fmt_pct_paren(se):>8s}  "
          f"95% CI = {fmt_ci(ci):>18s}  "
          f"corr = {paired_results['correlation']:.3f}  "
          f"[{label}]")


################################
# Run all evaluation statistics
################################

def main():
    summary_a = summarize_model(model_a, clusters)
    summary_b = summarize_model(model_b, clusters)

    print("=" * 90)
    print("Raw toy data")
    print("=" * 90)
    print(f"{model_a_name}: {model_a}")
    print(f"{model_b_name}: {model_b}")
    print(f"clusters:  {clusters}")
    print()

    print("=" * 90)
    print("Table 2 style: CLT standard errors / confidence intervals")
    print("=" * 90)
    print_table_2_style(model_a_name, summary_a)
    print_table_2_style(model_b_name, summary_b)
    print()

    print("=" * 90)
    print("Table 3 style: Clustered standard errors / confidence intervals")
    print("=" * 90)
    print_table_3_style(model_a_name, summary_a)
    print_table_3_style(model_b_name, summary_b)
    print()

    diff_means = difference_in_means(model_a, model_b, clusters, clusters)
    paired = paired_differences(model_a, model_b, clusters)

    print("=" * 90)
    print("Model comparison method 1: difference in means")
    print("=" * 90)
    print(f"Naive / CLT version:")
    print(f"  diff = {fmt_pct(diff_means['diff'])}")
    print(f"  SE(diff) = {fmt_pct_paren(diff_means['se_diff_clt'])}")
    print(f"  95% CI = {fmt_ci(diff_means['ci_diff_clt'])}")
    print()

    print(f"Clustered version:")
    print(f"  diff = {fmt_pct(diff_means['diff'])}")
    print(f"  SE(diff) = {fmt_pct_paren(diff_means['se_diff_cluster'])}")
    print(f"  95% CI = {fmt_ci(diff_means['ci_diff_cluster'])}")
    print()

    print("=" * 90)
    print("Model comparison method 2: paired instance-level differences")
    print("=" * 90)
    print(f"Question-level differences (A - B): {paired['diffs']}")
    print(f"Correlation(A, B) = {paired['correlation']:.3f}")
    print()

    print(f"Paired CLT version:")
    print(f"  mean diff = {fmt_pct(paired['mean_diff'])}")
    print(f"  SE(diff) = {fmt_pct_paren(paired['se_paired_clt'])}")
    print(f"  95% CI = {fmt_ci(paired['ci_paired_clt'])}")
    print()

    print(f"Paired clustered version:")
    print(f"  mean diff = {fmt_pct(paired['mean_diff'])}")
    print(f"  SE(diff) = {fmt_pct_paren(paired['se_paired_cluster'])}")
    print(f"  95% CI = {fmt_ci(paired['ci_paired_cluster'])}")
    print()

    print("=" * 90)
    print("Table 5 style: pairwise reporting")
    print("=" * 90)
    print_table_5_style(model_a_name, model_b_name, paired, clustered=False)
    print_table_5_style(model_a_name, model_b_name, paired, clustered=True)

main()

More Topics to Explore in Statistics

The above section covers most of the key information needed to start taking a statistically-oriented approach to evaluating LLMs. However, once we adopt the mindset of applying statistics to LLM evaluations, we open a new realm of possibilities! In this section, we provide a brief look into other areas of statistics—both from [1] and beyond—that can be applied to LLM evaluations, as well as highlight a few extra papers on the topic for future reading and motivation.

Power Analysis for LLM Evals [1]

For most of the overview so far, we have focused upon measuring uncertainty and reducing variance so that we can have more confidence when evaluating and comparing models. The techniques we have learned about are primarily focused on post-hoc analysis, and we have not spent much time considering the validity of the actual evaluation process itself. In [1], authors go beyond their discussion of standard errors, confidence intervals, and model comparisons by closing with a practical explanation of how power analysis can be applied to LLM evaluations.

What is power? The idea of power in statistics refers to the ability of some statistical experiment to make a valid measurement in the presence of noise. For example, we want to know whether one model actually improves over the performance of another in the LLM evaluation setting. Moving in this direction, power analysis allows us to answer the following question: Is the evaluation we are using capable of detecting the kind of improvement for which we are aiming?

Standard errors and confidence intervals allow us to quantify the uncertainty of an evaluation result. Power analysis focuses on the complementary concept of determining the number of questions n needed in order to reliably detect a difference in performance of a certain size. In [1], a sample size formula is derived that allows us to compute the necessary value of n under different settings. By using this formula, we can do things like:

• Check whether a certain evaluation is even worth running given the number of available samples.

• Determine a sufficient sample size when curating a new evaluation dataset.

Defining power. The discussion of power analysis in [1] uses the same exact setup used for paired model evaluations. We are comparing two models A and B, both models are evaluated on the same questions, and we analyze question-level score differences. Similarly to before, the true difference in means in this setting μ_{A-B} = μ_A - μ_B can be estimated with a sample mean difference s̅_{A-B}. This sample mean may or may not be near the true value due to the noise from sampling evaluation questions and the conditional randomness of each score.

Power refers to the ability to detect a real improvement when it actually exists. We define this based on a few different quantities:

• Significance level (α): the desired false positive rate (i.e., probability of detecting a difference in mean when it does not actually exist).

• Power (1 - β): the probability of detecting an effect (e.g., a true difference in mean) when it actually exists.

• Minimum detectable effect (δ): the smallest true difference in mean that we want to detect.

The significance level and power are used to capture Type I—or concluding there is a true difference when one does not actually exist—and Type II—or failing to detect a true difference when one does exist—errors, respectively. Intuitively, we can change these values to control the probability of false alarms or missed detections.

A sample size formula is provided in [1] for applying power analysis to LLM evaluations; see below. z_p represents the (1 − p)-th percentile of a standard normal distribution and is computed with the same approach we used to find the value of 1.96 in our prior confidence interval formulas. We will not go through the full details of this derivation, but the terms in this expression follow the same pattern as our prior discussion on variance reduction. We compute the question-level average variance for each model and use the variance of a difference identity to capture the variance of the question-level mean difference. We can derive a sample estimate of these variances from historical evaluation data.

This sample size formula provides useful intuition for statistical significance in LLM evaluations. Our evaluation requires a larger sample size when:

• The amount of variance is large.

• The size of the effect being detected is small.

• A stricter confidence or higher level of power is desired.

Additionally, we can decrease the necessary value of n by performing resampling, revealing that our previously-outlined techniques for variance reduction are still applicable. Notably, the sample size also grows quadratically with the inverse of the minimum detectable effect: detecting a gap in performance that is half the size requires 4× the number of samples. We can also rearrange this sample size equation to solve for the minimum detectable effect δ, allowing us to determine the smallest gap in performance that can be detected with some benchmark.

Sample size implementation. To compute the above sample size formula, we must solve for the correct z_p values given a specified significance level α and power 1 - β, as well as estimate the three variance terms from actual evaluation data. An example implementation is provided below for reference, which adopts the same patterns as our evaluation statistics code from the prior section.

import math
import numpy as np
from statistics import NormalDist


##########################
# Power analysis settings
##########################

alpha = 0.05
beta = 0.20
delta = 0.10


#################################
# Model scores (three resamples)
#################################

model_a_scores = np.array([
    [1, 1, 1],
    [0, 1, 0],
    [1, 1, 0],
    [1, 1, 1],
    [0, 0, 0],
    [1, 1, 1],
    [1, 0, 1],
    [0, 0, 1],
    [1, 1, 1],
    [1, 0, 1],
], dtype=float)

model_b_scores = np.array([
    [1, 0, 1],
    [0, 0, 0],
    [1, 0, 0],
    [1, 0, 1],
    [0, 0, 0],
    [1, 1, 0],
    [0, 0, 1],
    [0, 0, 0],
    [1, 0, 1],
    [1, 0, 0],
], dtype=float)


##########################################
# Compute z_p values from alpha and beta
#
# The paper uses:
# - z_{alpha/2}
# - z_beta, where power = 1 - beta
##########################################

def z_alpha_over_2(alpha: float) -> float:
    return NormalDist().inv_cdf(1 - alpha / 2)

def z_beta(beta: float) -> float:
    return NormalDist().inv_cdf(1 - beta)


#####################################
# Sample estimates of variance terms
#####################################

def question_means(score_matrix: np.ndarray) -> np.ndarray:
    """
    Estimate x_i for each question i by averaging over K samples.
    """
    return np.mean(score_matrix, axis=1)

def question_conditional_variances(score_matrix: np.ndarray) -> np.ndarray:
    """
    Estimate sigma_i^2 for each question i using the within-question
    sample variance across repeated samples. We cannot estimate this
    if we only have 1 resample (i.e., K = 1).
    """
    n, k = score_matrix.shape
    if k == 1:
        return np.zeros(n)
    return np.var(score_matrix, axis=1, ddof=1)

def estimate_sigma_squared(score_matrix: np.ndarray) -> float:
    """
    Estimate E[sigma_i^2] by averaging the within-question variances.
    """
    return float(np.mean(question_conditional_variances(score_matrix)))

def estimate_omega_squared(
    model_a_score_matrix: np.ndarray,
    model_b_score_matrix: np.ndarray,
) -> float:
    """
    Estimate:

        omega^2 = Var(x_A) + Var(x_B) - 2 Cov(x_A, x_B) = Var(x_A - x_B)

    where x_A and x_B are the question-level conditional means using sample
    variance and covariance across questions.
    """
    x_a = question_means(model_a_score_matrix)
    x_b = question_means(model_b_score_matrix)

    # compact form is Var(x_A - x_B)
    # equivalent to Var(x_A) + Var(x_B) - 2 Cov(x_A, x_B)
    """
    Expanded version:
    var_a = np.var(x_a, ddof=1)
    var_b = np.var(x_b, ddof=1)
    cov_ab = np.cov(x_a, x_b, ddof=1)[0, 1]
    return float(var_a + var_b - 2 * cov_ab)
    """
    diffs = x_a - x_b
    return float(np.var(diffs, ddof=1))

def estimate_power_analysis_variance_terms(
    model_a_score_matrix: np.ndarray,
    model_b_score_matrix: np.ndarray,
) -> dict:
    """
    Estimate all variance terms needed for sample-size formula in [1].
    """
    sigma_a_sq = estimate_sigma_squared(model_a_score_matrix)
    sigma_b_sq = estimate_sigma_squared(model_b_score_matrix)
    omega_sq = estimate_omega_squared(model_a_score_matrix, model_b_score_matrix)

    n_questions, k_a = model_a_score_matrix.shape
    n_questions_b, k_b = model_b_score_matrix.shape

    return {
        "n_questions": n_questions,
        "K_A": k_a,
        "K_B": k_b,
        "omega_sq": omega_sq,
        "sigma_a_sq": sigma_a_sq,
        "sigma_b_sq": sigma_b_sq,
    }


###############################
# Sample size formula from [1]
###############################

def required_sample_size(
    delta: float,
    alpha: float,
    beta: float,
    omega_sq: float,
    sigma_a_sq: float,
    sigma_b_sq: float,
    K_A: int = 1,
    K_B: int = 1,
) -> float:
    z_a2 = z_alpha_over_2(alpha)
    z_b = z_beta(beta)

    variance_term = omega_sq + sigma_a_sq / K_A + sigma_b_sq / K_B
    n = ((z_a2 + z_b) ** 2 * variance_term) / (delta ** 2)
    return float(n)


################################
# Compute Sample Size
################################

def main():
    terms = estimate_power_analysis_variance_terms(model_a_scores, model_b_scores)

    print("=" * 80)
    print("Estimated variance terms from fixed evaluation results")
    print("=" * 80)
    print(f"n_questions = {terms['n_questions']}")
    print(f"K_A = {terms['K_A']}")
    print(f"K_B = {terms['K_B']}")
    print(f"omega^2 = {terms['omega_sq']:.6f}")
    print(f"sigma_A^2 = {terms['sigma_a_sq']:.6f}")
    print(f"sigma_B^2 = {terms['sigma_b_sq']:.6f}")
    print()

    print("=" * 80)
    print("Critical z-values")
    print("=" * 80)
    print(f"alpha = {alpha:.3f}")
    print(f"beta  = {beta:.3f}")
    print(f"power = {1 - beta:.3f}")
    print(f"z_(alpha/2) = {z_alpha_over_2(alpha):.6f}")
    print(f"z_beta      = {z_beta(beta):.6f}")
    print()

    n_required = required_sample_size(
        delta=delta,
        alpha=alpha,
        beta=beta,
        omega_sq=terms["omega_sq"],
        sigma_a_sq=terms["sigma_a_sq"],
        sigma_b_sq=terms["sigma_b_sq"],
        K_A=terms["K_A"],
        K_B=terms["K_B"],
    )

    print("=" * 80)
    print("Required sample size")
    print("=" * 80)
    print(f"Target effect size delta = {delta:.4f}")
    print(f"Required n ≈ {n_required:.2f}")
    print()

main()

Don’t Use the CLT in LLM Evals With Fewer Than a Few Hundred Datapoints [2]

“Assumptions underlying the asymptotic, CLT-based approaches may not be suitable for LLM evals, at least in smaller data regimes. In that case, we expect to see broader failures of CLT-based confidence intervals.” - from [2]

In [2], authors extend the proposals from [1] by analyzing the effectiveness of the CLT in the small data regime (i.e., n <= 100). As we have learned, the CLT implies that sample means approach a normal distribution as the sample size n increases, but the value of n may have to be in the hundreds or thousands for this property to hold—the point at which n becomes “big enough” is difficult to determine a priori. The key insight from [2] is that the CLT underestimates uncertainty when there is limited evaluation data. This problem is worsened by the fact that LLM benchmarks are becoming increasingly specialized, leading many to the creation of many smaller benchmarks that capture performance on particular tasks (e.g., the popular SWE-Bench Verified benchmark contains only 500 questions).

CLT simulations. The shortcomings of the CLT are demonstrated in [2] via extensive simulation experiments with a known ground truth that permit directly verifying a confidence interval. From these simulations, we see that CLT-based methods consistently fail in small-data regimes by producing confidence intervals that are too narrow and overly-confident. Several scenarios are considered in the simulations in [2] that mostly align with evaluation setups from [1]:

• IID questions: model performance is measured on IID questions (assumed to be binary in [2]).

• Clustered questions: model performance is analyzed on questions that are not IID (i.e., the clustered setting from [1]).

• Unpaired model comparison: model performance is measured over separate question sets and compared between models.

• Paired model comparison: model performance is measured on an identical set of questions and compared between models.

Across all settings, evidence presented against CLT-based methods in the small n regime is clear—the CLT fails across all scenarios when n < 100. Such findings emphasize the fact that the CLT, despite being simple and powerful, makes underlying assumptions (i.e., IID variables, finite variance, and sufficiently large n) that degrade its effectiveness when violated. Authors in [2] do not recommend against using the CLT. Rather, they encourage awareness of these assumptions and limitations so that the CLT can be avoided in situations where it does not apply. Specifically, the key failure case for the CLT in [2] is when n < 100.

One other specific issue highlighted with the CLT in [2] is cases where models begin to achieve either perfect or zero accuracy. On especially small datasets, it is possible that an LLM either answers all questions correctly or—in the case of a tiny but non-trivial dataset like AIME—answers no questions correctly. In these cases, confidence intervals produced with the CLT will become overly narrow because all scores are the same, thus worsening issues with overconfidence.

Alternative approaches. Although the full details of these techniques are beyond the scope of this post, authors in [2] provide several possible alternative methods for computing confidence intervals. Most prominent among these techniques are Bayesian methods, which are less sensitive to the value of n and can provide narrower confidence intervals relative to the CLT. As shown above, Bayesian intervals are still relatively straightforward to compute and can be extended to handle important settings such as clustered questions or model comparisons. A brief overview of several alternative techniques—including Bayesian methods—alongside their benefits and drawbacks is provided below for reference.

Quantifying Variance in Evaluation Benchmarks [3]

“If we cannot trust our evaluation results or do not understand what improvements are statistically significant, we cannot make sound comparisons, making it more challenging to reliably use benchmarks.” - from [3]

As we have learned, most LLM evaluations just report a single deterministic score (e.g., an accuracy of 70% on MMLU) without explicitly accounting for variability. Small score differences are often used to claim superior performance of a model, but it is usually unclear whether a small difference is attributable to noise or actually a meaningful capability improvement. This issue causes misleading or even incorrect results on benchmarks, as well as poor decision making during the model development process. In [3], authors perform a deep dive into variance of LLM evaluation using 13 popular benchmarks and over 280 models.

Measuring variability. In order to perform a large-scale analysis of benchmark variance, a broad group of LLMs is curated in [3]. First, a group of seed models—all based upon Llama-2-7B—is created by training from scratch with different random initialization seeds on 210 billion tokens of data. Checkpoints are collected throughout training for each seed, resulting in 210 total snapshots that are used for evaluation. These seed models are then supplemented with an additional 41 checkpoints of Llama 1 and 2 from various training stages, as well as 32 other models across a variety of families (e.g., Gemma and Mistral). This group of models is then evaluated over a set of 13 benchmarks that cover a wide variety of domains like reasoning, math, general knowledge, and coding.

Variance metrics. To study the variability in evaluation results, three metrics are considered in [3]:

1. Seed variance measures the standard deviation of performance across models trained with different random seeds and is reported as an average over all training checkpoints.

2. Monotonicity8 measures the whether the sequence of evaluation scores for a model improve regularly throughout training.

3. The signal-to-noise ratio (SNR) of seed models is measured by dividing the mean benchmark score of the final model across different seeds by the standard deviation of scores across seeds.

Key findings. We learn in [1] that different benchmarks have drastically different variance characteristics; see below. For example, smaller benchmarks (e.g., COPA and HumanEval) are found to have higher seed variance and larger confidence intervals, emphasizing once again the need for evaluation datasets that are sufficiently large. Additionally, some benchmarks have random performance for smaller models, even after extensive training. These benchmarks may be too difficult for certain models, which reflects findings in current research showing that certain benchmarks may only be useful at a specific scale.

We also see in [3] that using a continuous evaluation formulation based upon token probabilities yields more reliable evaluation results compared to binary evaluations based upon correctness. Notably, this finding aligns with the variance reduction analysis provided in [1]. Specifically, authors in [3] compute continuous metrics using either:

• The probability of the correct answer token for multiple choice questions.

• The log likelihood of a reference answer—computed by summing the log probabilities for all tokens in a completion—for open-ended generations.

By using continuous evaluation metrics based upon these token probabilities, the SNR and monotonicity of evaluation benchmarks noticeably improve; see above. Based upon this observation, authors in [3] also reformulate the popular MMLU question-answering benchmark to be completion-based instead of using multiple choice questions—this new dataset is called MMLU-Cloze9. As shown in the figure below, this reformulation drastically reduces the variability of the benchmark, thus highlighting the benefit of using continuous metrics for LLM evaluation.

Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation [4]

During the LLM development process, we perform small-scale experiments to tune our training settings and rely upon evaluation results to determine the best setting. However, the results of small-scale experiments may not translate well to large-scale training runs, and noise in the evaluation process can lead to incorrect decisions. In [4], an SNR-based framework is proposed for assessing benchmark reliability and improving the predictive accuracy of evaluation across scales.

Assessing reliability. Evaluation datasets are analyzed in [4] using an SNR metric, but a specific definition of signal and noise is proposed. Assume scores is an array of scores for a benchmark, where each index stores the evaluation score for a different model. We define signal = [max(scores) - min(scores)] / mean(scores). In other words, the signal metric captures the relative spread of scores across models for a particular benchmark.

Noise measures variability in performance due to randomness in the training process. To compute noise, we could train multiple models with different random seeds and measure the variance in their evaluation scores. However, such an approach is computationally expensive. As a solution, noise is measured in [4] by:

• Considering the last n model checkpoints from the training process.

• Obtaining the evaluation result for each of these checkpoints, yielding a list of evaluation scores ckpt_scores.

• Computing noise = std(ckpt_scores) / mean(ckpt_scores).

We can then combine these metrics into a single SNR metric by taking the quotient of signal and noise. This SNR metric is helpful for analyzing benchmark reliability, as any evaluation dataset with high SNR is capable of distinguishing between different models and is relatively insensitive to training randomness.

Practical tips. The SNR metric is validated in a large-scale evaluation study in [4] that considers 465 LLMs and 30 evaluation benchmarks. Across all evaluation settings, we see that benchmarks with higher SNR provide more reliable model rankings. Specifically, the correlation between SNR and decision accuracy—meaning that the better model receives a higher evaluation score on a particular benchmark—is found to be quite high. Several practical tips for LLM evaluations are proposed in [4] based on these observations:

• For an evaluation benchmark, we can select specific sub-tasks with the highest SNR to improve reliability. For example, authors in [4] use SNR to select 16 (of 57 total) MMLU tasks for evaluation, which improves decision accuracy and drastically reduces evaluation costs.

• Instead of only evaluating the final model checkpoint, we can compute an average evaluation score across the last n model checkpoints to improve reliability and mitigate noise due to training randomness.

Authors in [4] also advocate for using continuous—rather than discrete—metrics for evaluation. Similarly to findings in [1, 3], we see in [4] that evaluating a model based upon the log likelihood of the correct completion improves the reliability of the evaluation process, as evidenced by a clearly-improved SNR.

“We calculate the bits-per-byte (BPB) using the correct continuations of each test set. The bits-per-byte is the negative log likelihood of the correct answer divided by the number of UTF-8 bytes in the answer string.” - from [4]

Key Takeaways

In this overview, we have learned a wide variety of tools for evaluating LLMs in an uncertainty-aware manner. To close, we will summarize what we’ve learned by outlining how each of these tools can be used when evaluating an LLM. In the simplest case, we can draw upon the CLT to derive a standard error and confidence interval along with our evaluation results. However, there are a few cases in which this approach will not yield valid results:

• If the value of n is small, then the CLT-based standard error expression is overly confident. We can solve this issue by evaluating over a larger dataset or using another approach (e.g., the Bayesian methods outlined in [2]) that is better equipped to deal with smaller n.

• If evaluation questions are not independent, then we can derive a cluster-adjusted standard error to account for the relationship between questions in our evaluation dataset.

When comparing models that are evaluated on the same questions (i.e., a paired setup), we can apply the same approaches over their question-level differences to provide a more statistically efficient estimate of which model performs better.

To reduce evaluation variance, we can use resampling, where K is selected such that E[σ_i^2 / K] ≪ Var(X). In some settings, token probabilities can be used to compute the expected score—or the probability of the ground truth answer— directly, thus reducing within-question variance. Such an approach has been shown in several concurrent works [1, 3, 4] to improve the stability of evaluation results. When creating an evaluation dataset, we can use power analysis—or just adopt the sample size formula from [1]—to determine the number of samples needed. We can also rearrange the sample size formula to find the minimum detectable effect δ that can be measured with a given dataset, which helps us to determine whether certain evaluations are even worth running at all.

New to the newsletter?

Hi! I’m Cameron R. Wolfe, Deep Learning Ph.D. and Senior Research Scientist at Netflix. This is the Deep (Learning) Focus newsletter, where I help readers better understand important topics in AI research. The newsletter will always be free and open to read. If you like the newsletter, please subscribe, consider a paid subscription, share it, or follow me on X and LinkedIn!

Subscribe now

Bibliography

[1] Miller, Evan. “Adding error bars to evals: A statistical approach to language model evaluations.” arXiv preprint arXiv:2411.00640 (2024).

[2] Bowyer, Sam, Laurence Aitchison, and Desi R. Ivanova. “Position: Don’t Use the CLT in LLM Evals With Fewer Than a Few Hundred Datapoints.” arXiv preprint arXiv:2503.01747 (2025).

[3] Madaan, Lovish, et al. “Quantifying variance in evaluation benchmarks, 2024.” URL https://arxiv. org/abs/2406.10229 (2024).

[4] Heineman, David, et al. “Signal and noise: A framework for reducing uncertainty in language model evaluation.” arXiv preprint arXiv:2508.13144 (2025).

Many of the recent capability gains in large language models (LLMs) have been a product of advancements in reinforcement learning (RL). In particular, RL with verifiable rewards (RLVR) has drastically improved LLM capabilities by using rules-based, deterministic correctness checks (e.g., passing the test cases for a coding problem) as a reward signal. Deterministic verifiers allow RLVR to provide a reliable reward signal that is more difficult to exploit compared to the neural reward models that were traditionally used for RL with LLMs. Such improved reliability has made stable RL training possible at scale, enabling the creation of powerful reasoning models with extensive RL training. Despite these benefits, verifiable rewards also have limitations—the same properties that make RLVR reliable confine it to domains with clean, automatically-checkable outcomes.

“While lots of efforts have been paid on RLVR, many high-value applications of LLMs, such as long-form question answering, general helpfulness, operate in inherently subjective domains where correctness cannot be sufficiently captured by binary signals.” - from [3]

Many important applications (e.g., creative writing or scientific reasoning) are not verifiable, making RLVR difficult to apply directly. To address this gap, we need reward signals that preserve RLVR’s scalability and reliability while still working in non-verifiable settings. Rubric-based rewards are a promising step in this direction: they decompose desired model behavior into structured, interpretable criteria that an LLM judge can evaluate and aggregate into a multi-dimensional reward. By creating prompt-specific rubrics that specify the evaluation process in detail, we can derive a more reliable reward signal from LLM judges and, therefore, use RL training to improve model capabilities even in highly subjective domains. For this reason, rubric-based RL training, which we will cover extensively in this overview, has become one of the most popular topics in current AI research.

From LLM-as-a-Judge to Rubrics

Before learning about how rubrics can be used for RL training, we need to build a background understanding of LLM-as-a-Judge and the different setups that can be used to evaluate open-ended problems with an LLM. At the end of the section, we will connect these ideas to rubrics and RL training by overviewing existing RL training techniques and how they are being extended to non-verifiable domains.

LLM-as-a-Judge

Prior to the LLM era, many evaluation metrics used for generative tasks (e.g., BLEU or ROUGE) were quite brittle. These metrics use n-gram matching (or embedding-based matching as in BERTScore) to compare a model’s output to a golden reference answer. Though this approach works relatively well, there are some fundamental problems that arise with reference-based metrics:

• We always require a reference answer in order to perform evaluation.

• Our output must be similar to this reference answer to perform well.

As we know, LLMs are capable of solving many different tasks, and most of these tasks are open-ended in nature. For example, we can use the same LLM to do creative writing or to answer medical questions. Although these problems are quite different, they do have a fundamental similarity: there are many ways to answer a question correctly. Traditional reference-based metrics struggle to handle such nuanced scenarios where divergence from a chosen reference answer does not imply that an output is bad. As a result, we have seen from several papers that reference-based metrics tend to correlate poorly with human preferences.

“LLM-as-a-judge is a scalable and explainable way to approximate human preferences, which are otherwise very expensive to obtain.” - from [7]

LLM-as-a-Judge is a reference-free metric that prompts a foundation model to perform evaluation based upon specified criteria. Although it has limitations, this technique shows high agreement in many settings with human preferences and is capable of evaluating open-ended tasks in a scalable manner (i.e., minimal implementation changes are required). To evaluate a new task, we simply need to create a new prompt that outlines the evaluation criteria for this task. LLM-as-a-Judge was originally proposed after the release of GPT-4. This metric quickly gained popularity due to its utility and simplicity, culminating in the publication of an in-depth technical report [7]. Today, LLM-as-a-Judge is a widely-used technique in LLM evaluation; e.g., AlpacaEval, Chatbot Arena, Arena-Hard, and more.

Scoring setups. When performing evaluation with an LLM, there are a few different scoring setups that are commonly used (shown above):

1. Pairwise (preference) scoring: the judge is presented with a prompt and two model responses and asked to identify the better response.

2. Direct assessment (pointwise) scoring: the judge is given a single response to a prompt and asked to assign a score; e.g., using a 1-5 Likert scale.

3. Reference-guided scoring: the judge is given a golden reference response in addition to the prompt and candidate response(s) to help with scoring.

This list of scoring setups is not exhaustive, but most scoring setups for LLM-as-a-Judge use some variant or combination of the above techniques. For example, we can derive a pairwise score by scoring two responses independently and comparing their scores. In most cases, we also pair LLM-as-a-Judge with chain-of-thought prompting by asking the model to explain its evaluation process before providing a final score. Not only do such explanations make the evaluation process more interpretable, but they also improve the scoring accuracy of the LLM. Practically, implementing this change can be as simple as adding “Please provide a step-by-step explanation prior to your final score” to your prompt.

“We identify biases and limitations of LLM judges. However, we… show the agreement between LLM judges and humans is high despite these limitations.” - from [7]

Biases of LLM-as-a-Judge. Despite the effectiveness of LLM-as-a-Judge, this technique has several limitations of which we need to be aware. Fundamentally, the LLM judge is an imperfect proxy for human evaluation. By using a model for evaluation, we introduce several sources of bias into the evaluation process:

1. Position bias: the judge may favor outputs based upon their position within the prompt (e.g., the first response in a pairwise prompt).

2. Verbosity bias: the judge may assign better scores to outputs based upon their length (i.e., longer responses receive higher scores).

3. Self-enhancement bias: the judge tends to favor responses that are generated by itself (e.g., GPT-5 can assign higher scores to its own outputs).

4. Capability bias: the judge struggles with evaluating responses to prompts that it cannot itself solve.

5. Distribution bias: the judge may be biased towards certain scores in its scoring range (e.g., on a 1-5 Likert scale the judge may output mostly 3’s).

In addition to these biases, LLM judges are generally sensitive to the details of their prompt. Therefore, we should not simply write a prompt and assume proper evaluation. We must calibrate our evaluation process, collect high-quality human labels, and tune our prompt to align well with human judgment; see here.

There are several techniques we can adopt to combat scoring bias; e.g., in-context learning to better calibrate the judge’s score distribution, randomizing position and sampling multiple scores (i.e., position switching), providing high-quality reference answers, or using a jury of multiple LLM judges. For further details on LLM-as-a-Judge, a full overview of the topic is available at the link below.

LLM Evaluation with Rubrics

The prompts used for LLM-as-a-Judge in the above section are quite simple. We just describe the evaluation task at a high level and let the LLM judge output a score. However, scoring with a single, general prompt is not always the best approach. Prior work [15] has shown that we can significantly improve the reliability of LLM evaluation by:

• Creating several per-criterion scoring prompts.

• Providing a step-by-step description of the evaluation process.

Put simply, providing a granular scoring prompt is beneficial, and we need not stop here. We can create judge prompts targeted to each domain, task, or instance. Increasing the granularity of LLM-as-a-Judge in this way is where the idea of a rubric arises. A rubric is just a scoring prompt that provides a detailed set of criteria by which a response is evaluated; see below. In many cases, rubrics are prompt (or instance)-specific, meaning that a tailored rubric is created for each prompt-response pair being evaluated. These prompt-specific rubrics are often synthetically generated with an LLM—potentially with human intervention.

As we can see above, rubrics are usually checklist-style and separated into a list of distinct criteria. Each of these criteria captures a single quality dimension that can be evaluated with an LLM judge. Additionally, in many setups, weights are defined for each criterion to simplify the aggregation of criteria-level scores. Given the similarity of rubrics and vanilla LLM-as-a-Judge, the emergence of rubrics is hard to attribute to a single paper. Rather, the use of rubrics was a slow transition that occurred over time as LLM-as-a-Judge prompts became more granular.

“HealthBench is a rubric evaluation. To grade open-ended model responses, we score them against a conversation-specific physician-written rubric composed of self-contained, objective criteria. Criteria capture attributes that a response should be rewarded or penalized for in the context of that conversation and their relative importance.” - from [16]

In recent work, prompt-specific rubrics have become heavily used for evaluation in expert domains. For example, HealthBench [16] evaluates the quality of medical conversations according to physician-written rubrics that are specific to each conversation; see below. These rubrics focus on detailed and objective criteria—each associated with a weight—that can be verified with an LLM to yield a binary (pass or fail) score. MultiChallenge [17]—a multi-turn chat benchmark focused on tough edge cases like iterative editing, self-coherence, and instruction retention—develops prompt-specific rubrics to improve benchmark reliability, finding that rubrics improve agreement between expert humans and LLM judges.

In this overview, we will go beyond the use of rubrics for evaluation and instead focus on the application of rubrics for deriving a reward signal in RL training. One of the biggest risks when using LLM-as-a-Judge-derived rewards for RL training is reward hacking—LLM judges have known biases that can be exploited. However, we see above that detailed rubrics help to make the evaluation process more reliable, thus reducing risks associated with reward hacking.

RL with Verifiable (and Non-Verifiable) Rewards

Though RL training has long been used for LLMs, the role of RL in LLM training pipelines has become more central with the recent advent of reasoning models. In general, there are two common RL paradigms used for LLMs:

• Reinforcement Learning from Human Feedback (RLHF) trains the LLM using RL with rewards derived from a reward model trained on human preferences.

• Reinforcement Learning with Verifiable Rewards (RLVR) trains the LLM using RL with rewards derived from rule-based or deterministic verifiers.

The main difference between RLHF and RLVR is how we assign rewards—RLHF uses a reward model, while RLVR uses verifiable rewards. Aside from this difference, both are online RL algorithms with a similar structure; see below. For details on the inner workings of RL optimizers, please see prior posts on PPO and GRPO.

Impact of RLVR. Recent progress in reasoning models has been driven largely by reinforcement learning with verifiable rewards (RLVR), which derives a reward signal during RL training from deterministic (or programmatic) rules that can be reliably checked (e.g., passing unit tests for code or matching a known numerical answer in math). Using rules-based rewards lowers our risk of reward hacking because we are using a hard rule to derive our reward rather than an LLM-based reward model. As a result, we can run larger-scale RL runs (i.e., over more data and for a larger number of iterations) with less risk of training instability.

On the other hand, the same property that makes RLVR so powerful—the dependence on reliable, rules-based rewards—limits its applicability. Practically, we can only use RLVR on tasks with clean ground-truth labels1 that can be checked automatically. Luckily, several important tasks fall into this category (e.g., math and coding). However, there are many other tasks we would like to solve but are subjective and difficult to verify. Due to this need for verification, we see that LLMs have advanced quickly in certain verifiable capabilities, while gains on non-verifiable tasks have been less uniform. To solve this issue, we need to develop an approach for extending recent advances in RL training to non-verifiable tasks.

“In RLVR, rewards are derived from deterministic, programmatically verifiable signals—such as passing unit tests in code generation or matching the correct numerical answer in mathematical reasoning. While effective, this requirement for unambiguous correctness largely confines RLVR to domains with clear, automatically checkable outcomes.” - from [2]

Open-ended domains. We typically turn to RLHF for training LLMs in open-ended settings. RLHF replaces deterministic verifiers with a learned reward model trained on preference data; see below. Preference data can be collected for any domain by simply sampling multiple completions for each prompt and having a human (or model) select the better of the two. We can drastically increase domain coverage by using RLHF. However, relying upon preference data and reward models introduces notable difficulties and failure modes:

• A large volume of preference data must be collected.

• We lose granular control over the alignment criteria—preferences are expressed in aggregate over a large volume of data rather than via explicit criteria.

• The reward model can overfit to artifacts (e.g., response length, formatting, etc.) and generally introduces more risk of reward hacking.

RLHF is a general technique, but it is usually used in practice for improving broad, subjective properties; e.g., helpfulness, harmlessness, or style. For complex, open-ended tasks, the reward signal tends to be multi-dimensional. Traditional reward modeling captures these quality dimensions via a single preference label, which eliminates our ability to specify quality dimensions at a more granular level. One could collect criterion-level preferences to solve this issue, but doing so requires training (and maintaining) separate reward models per criterion and increases the volume of data that must be collected. A natural alternative is to make evaluation dimensions explicit by using a rubric to ground the reward in structured, interpretable criteria rather than a single judgment.

Rubrics-as-Rewards. The idea of deriving a reward from a rubric-based LLM judge is one of the current frontiers of RL research—it presents an opportunity to extend RLVR to arbitrary open-ended tasks. Although this area of research is still nascent and evolving quickly, the idea of using rubrics for RL is not new! Similar ideas have already been proposed for better handling the safety alignment of LLMs. During LLM alignment, we have a detailed list of safety specifications that describe the desired behavior of the model. These specifications are changed frequently as new needs or failure cases arise in practice. The dynamic nature of safety criteria makes applying a standard RLHF approach difficult—the preference data must be adjusted or re-collected each time that our criteria change.

To avoid the need for constant data collection, methods like Constitutional AI [13] and Deliberative Alignment [14] show that a reliable reward signal can be derived directly from the safety specifications themselves. More specifically, we can provide safety criteria as input to a strong reasoning model that is used to generate data or evaluate model outputs according to these criteria. Due to the strong instruction following capabilities of frontier-level reasoning models, this approach is capable of providing a reliable reward signal for safety training.

This approach avoids the need to re-collect data as criteria change. Rather, we just maintain a clear, itemized list of safety criteria—basically a safety rubric—that can be provided as input to the alignment system. Instead of collecting data, we focus on creating a “constitution” that dictates the behavior of our model. Once this constitution is available, we rely upon an LLM judge to apply the necessary supervision for achieving this desired behavior. This approach is both dynamic and interpretable, but it can only be applied in domains where the LLM judge is known to perform well. Extending similar techniques to arbitrary domains, which we will explore for the remainder of this post, is a non-trivial research problem.

Using Rubrics for RL

We now have a detailed understanding of LLM-as-a-Judge, rubrics, and their application to RL training. Next, we will extend these ideas by overviewing a broad collection of recent papers that study the application of rubrics to RL training. Many papers have been written on this topic in quick succession. As we will see, however, much of this work shares a similar flavor. Slowly, rubric-based RL has become more effective across a wider variety of tasks, enabling powerful reasoning models to achieve impressive gains even in non-verifiable domains.

Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains [1]

“Rather than using rubrics only for evaluation, we treat them as checklist-style supervision that produces reward signals for on-policy RL. Each rubric is composed of modular, interpretable subgoals that provide automated feedback aligned with expert intent. By decomposing what makes a good response into tangible, human-interpretable criteria, rubrics offer a middle ground between binary correctness signals and coarse preference rankings.” - from [1]

RLVR is effective in verifiable domains with a clear correctness signal like math or coding, but there are many domains in the real world that are not strictly verifiable (e.g., science or health). For these domains, we need a more versatile reward mechanism—such as an LLM judge or reward model—that can handle open-ended problems that lack a clear or verifiable answer. Going beyond a vanilla LLM-as-a-Judge setup, we see in [1] that prompting the LLM judge with a rubric composed of structured, instance-specific—meaning unique to each prompt—criteria benefits the model’s performance in on-policy RL training.

Creating rubrics. Rubrics in [1] are checklist-style and cover multiple criteria that are specific to each prompt being scored. The checklist for a rubric contains K total criteria c_i, each with a corresponding weight w_i. A criterion is defined as a binary correctness check that can be validated using an LLM judge. We can also recover an RLVR setup by assuming K = 1 and letting c_1 be a deterministically verifiable reward signal with weight w_1 = 1.0.

We refer to this approach of using rubrics to generate a reward signal for RL as Rubrics-as-Rewards (RaR). There are two approaches we can use to evaluate a rubric and derive a reward for RL training (shown above):

• Explicit aggregation: each criterion is independently evaluated using an LLM judge, and the final reward is derived by summing and normalizing the weighted score of each criterion.

• Implicit aggregation: all criteria along with their weights are passed to an LLM judge, which is asked to derive a final reward that considers all information.

Explicit aggregation provides more granular control over the weight of each criterion, which can aid in interpretability but requires tuning and can be fragile. In contrast, the implicit aggregation approach delegates the reward aggregation process—including handling the weights of each criterion—to the LLM judge.

Generating rubrics. All instance-specific rubrics used in [1] are generated by an LLM; see above. When generating rubrics, guiding principles are provided to the model with respect to how rubrics should be created. Namely, rubrics must i) be grounded in guidance from human experts, ii) be comprehensive (i.e., span many dimensions of quality), iii) specify per-criterion importance (e.g., factuality is more important than style), and iv) use self-contained criteria (i.e., criteria should not depend on one another). Given these desiderata and a golden (expert-curated) reference answer for a prompt, the LLM then generates a rubric that includes:

• 7-20 self-contained criteria.

• A numeric or categorical (i.e., essential, pitfall, important, or optional2) weight for each of these criteria.

Numeric weights provide fine-grained control over criterion importance, but categorical weights, each of which are mapped to a numerical score, are more interpretable—both for humans and the LLM—which leads them to be used for experiments in [1]. Once generated, a rubric can be used as a reward function by passing it to an LLM judge and performing explicit or implicit aggregation.

“We generate rubrics using OpenAI’s o3-mini and GPT-4o, conditioning generation on reference answers from the underlying datasets to approximate expert grounding. The resulting collections—RaR-Medicine and RaR-Science—are released for public use.” - from [1]

Experimental settings. In [1], authors see rubrics as an opportunity to provide flexible, scalable, and interpretable reward signals for RL in real-world domains that go beyond verifiable problems like code and math. Moving in this direction, two non-verifiable domains are considered in [1]: medicine and science. Prompts and rubrics used for RL in [1] are sampled from a mixture of public datasets, such as NaturalReasoning, SCP-116K, and GeneralThought-430K. This data is further curated to create two datasets for RaR training in [1]:

• RaR-Medicine: ~20K prompts focused on medical reasoning with instance-specific rubrics generated with GPT-4o.

• RaR-Science: ~20K prompts curated to align with the problem categories from GPQA-Diamond with instance-specific rubrics generated by o3-mini.

All experiments use Qwen-2.5-7B as a base model and train with GRPO. Rewards are assigned using GPT-4o-mini with the instance-level rubrics described above. The proposed technique in [1], referred to as RaR-Implicit, uses LLM-generated, instance-specific rubrics with implicit aggregation as a reward signal. Several rubric-free and fixed-rubric baselines are also considered:

• Base models: Qwen-2.5-7B and Qwen-2.5-7B-Instruct models are evaluated with no additional training.

• Direct Assessment Judge: an LLM judge provides a direct assessment score for each response on a 10-point Likert scale—this is a standard LLM-as-a-Judge setup that does not use a granular, instance-specific rubric.

• Reference-Based Judge: same as above, but the LLM judge is given a golden reference answer as context when generating a score.

• RaR-Predefined: a fixed set of generic rubrics are used for all prompts with explicit aggregation and uniform criteria weights.

• RaR-Explicit: instance-specific rubrics are used, but all criteria receive fixed weights based on their categorical importance label.

All models are evaluated on the GPQA-Diamond (Science) and HealthBench (medicine) benchmarks. For some smaller ablation experiments, RL training is performed on the training set of HealthBench rather than RaR-Medicine.

Do rubrics provide useful rewards? Across all experiments in [1], we see that using structured, rubric-based rewards during RL training is beneficial. Rubric-based rewards are especially impactful when using smaller LLM judges for RL training and are found to reduce variance in reward signals across different sizes of LLM judges. As shown above, rubric-based approaches outperform all rubric-free methods aside from the reference-based LLM judge, relative to which we only see marginal gains from rubrics. However, rubrics are found to yield a more notable gain over reference-based LLM judge rewards in later experiments that train on HealthBench; see below. We also see that implicit aggregation tends to outperform explicit aggregation by a small (but consistent) margin.

“Rubric-guided training achieves strong performance across domains, significantly outperforming Likert-based baselines and matching or exceeding the performance of reference-based reward generation.” - from [1]

These experiments also highlight the necessity of expert-curated references for generating rubrics—performance noticeably deteriorates without references, indicating purely synthetic rubrics are suboptimal. Predefined or generic rubrics are also found to perform quite poorly, indicating that prompt-specific criteria are useful for deriving high-quality rubrics. These best practices for creating better rubrics are also evaluated beyond their impact on RL training. In [1], authors show that rubrics created via their proposed approach have noticeably higher levels of agreement with preference annotations from human experts; see below.

Reinforcement Learning with Rubric Anchors [2]

“The success / failure hinges tightly on the diversity, granularity, and quantity of the rubrics themselves, as well as on a proper training routine and meticulous data curation.” - from [1]

Authors in [2] continue studying the application of RL to open-ended tasks using rubric-based rewards. They scale the rubric creation process to produce a dataset of ~10K rubrics curated by humans, LLMs, or a combination of both. Building on this dataset, a practical exposition of rubric-based RL is provided, ultimately arriving at a functional RaR training framework called Rubicon. Interestingly, simply increasing the number of rubrics—whether generated synthetically or with human assistance—yields only marginal gains. Instead, we must carefully curate high-quality rubrics, suggesting that the success of RaR heavily depends upon both rubric quality and the quality of the underlying training dataset.

Rubric system. Instead of using strictly instance-level rubrics, multiple scopes are considered in [2], including instance, task, and dataset-level rubrics. When generating data, the system in [2] (shown below) starts by constructing the rubric first. Data is synthesized only after the rubric is created so that it explicitly matches the rubric. Then, the combination of rubric and data is used for both RL training and evaluation. Tasks in [2] are selected according to the asymmetry of verification—verifying a candidate output should be much easier than generating it.

To ensure rubric quality, authors run dedicated ablation experiments for every set of rubrics that is generated to measure their impact on the training process. Each rubric is comprised of K criteria C = {c_1, c_2, …, c_K}. An example of a rubric created for evaluating open-ended or creative tasks is provided below. After evaluating each of these criteria, we are left with a multi-dimensional reward vector that can be aggregated to yield a final reward.

As a baseline, criteria-level rewards can be aggregated via a weighted average, but non-linear dependencies may exist between criteria that make a weighted average suboptimal. For this reason, authors in [2] consider the following advanced strategies for criteria aggregation:

• Veto Mechanisms: failing on a critical dimension overrides any reward from other dimensions.

• Saturation-Aware Aggregation: over-performing on a single dimension yields diminishing returns relative to a balanced reward across dimensions.

• Pairwise Interaction Modeling: criteria are modeled together to capture inter-criteria relationships (i.e., synergistic or antagonistic effects).

• Targeted Reward Shaping: rewards in high-performance regions are amplified to better capture differentials and avoid scores becoming compressed.

Training strategy. The data used in [2] is derived from a proprietary post-training corpus with ~900K examples. Prior to any training, offline difficulty filtering is performed to remove any examples on which the base model performs too poorly or already performs well3. From here, RL training progresses in two phases, each with a different curriculum:

• The first phase focuses on instruction-following and programmatically-verifiable tasks to teach the LLM how to properly handle constraints.

• The second phase extends the training process to more open-ended and creative tasks with a higher level of subjectivity.

While the first phase primarily relies upon static rubrics and verifiers, we must use reference-based rubrics—often with instance-specific criteria—for the second phase. Granular rubrics help to provide a more reliable reward signal on tasks that are highly subjective. This multi-stage training framework aims to progressively cultivate the capabilities of the model. When training jointly on all tasks, authors observe a “seesaw effect”—joint training actually reduces model performance relative to forming a multi-stage curriculum; see below.

Reward hacking is one of the biggest risks in a RaR setup. Whereas verifiable rewards are deterministic, neural reward models can be exploited, and the likelihood of our policy finding such an exploit increases in large-scale RL runs4. The Rubicon approach proposed in [2] combats reward hacking by performing an offline analysis of rollout data. After the first phase of RL training, authors examine rollouts that yield abnormally high rewards and create a basic taxonomy of recurring reward hacking patterns that are discovered. From this taxonomy, a specific rubric is created for preventing reward hacking—this rubric can also be iteratively refined over time. The addition of a reward hacking rubric improves training stability (i.e., avoids collapse into a reward-hacked state) and allows RL training to be conducted for a much larger number of training steps.

Rubicon-preview is a Qwen-3-30B-A3B base model that is finetuned in [2] using the Rubicon framework. This model excels in open-ended and humanities-related benchmarks. For example, we see below that Rubicon-preview achieves an absolute improvement of 5.2% compared to the base model on various instruction following, emotional intelligence, and writing benchmarks. Notably, Rubicon-preview also outperforms DeepSeek-V3-671B on most of these tasks, where an especially significant performance boost is observed on writing tasks.

The performance benefits of Rubicon-preview are also achieved with shocking sample efficiency—the model is only trained on ~5K data samples. By using an RaR approach, authors are also able to granularly control the style or voice of the resulting model. More specifically, a few case studies are presented in [2] that demonstrate the use of rubrics to guide the LLM away from the didactic tone that is common of chatbots and towards a human-like tone with more emotion.

Going further, creatively-oriented RaR training does not seem to damage the LLM’s general capabilities. As shown above, Rubicon-preview performs on par with or better than the original base model across a wide scope of benchmarks. Such a result should not come as a surprise given the natural ability of RL to avoid forgetting and retain the prior knowledge or skills of an LLM; see here.

OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment [3]

We’ve seen several papers that study the use of rubrics for RL training, where rubrics are generated—possibly with human intervention—and evaluated by an off-the-shelf LLM. Instead of focusing upon the downstream application of rubrics in RL, authors in [3] specifically analyze the rubric generation and evaluation process. To facilitate this study, an open dataset of prompt-rubric pairs, called OpenRubrics, is created for training both rubric generation models and rubric-based reward models. As we learned in [2], RaR training is highly dependent upon rubric quality. Creating better rubrics—and reducing the amount of human supervision in this process—makes RaR training more scalable and effective.

The rubric structure used in [3] is consistent with prior work. Namely, each rubric is comprised of K criteria, where each criterion is a rubric description that specifies one aspect of response quality. Two types of criteria are considered:

1. Hard rules: explicit or objective constraints (e.g., length or correctness).

2. Principles: higher-level qualitative aspects (e.g., reasoning soundness, factuality, or stylistic coherence).

Unlike prior work, rubrics in [3] do not use per-criterion weights and are used for pairwise comparison of two completions—as opposed to direct assessment. For a rubric R = {c_1, …, c_K} and two responses y_1 and y_2 to the same prompt x, we want our rubric-based reward model to provide a binary preference label (i.e., y_1 > y_2 or y_1 < y_2) by reasoning over the rubric criteria.

Building OpenRubrics. The prompts and preference labels used for creating OpenRubrics are sourced from several public datasets (e.g., UltraFeedback, MegaScience, Medical-o1, instruction following data from Tulu-3, and more). For each of these datasets, preference data is obtained via domain-specific post-processing of the existing data. For example, the highest and lowest scoring responses form a preference pair for UltraFeedback, while for MegaScience and Medical-o1 completions are generated with a pool of LLMs and scored via a jury of different reward models to obtain preference pairs; see below.

Once this preference data is available, rubrics are generated using two key strategies proposed in [3] (shown above):

1. Contrastive Rubric Generation (CRG): an instruction-tuned LLM is provided both a prompt and a preference pair and asked to produce discriminative evaluation criteria by contrasting the chosen and rejected responses.

2. Rubric Filtering: rubrics are filtered by prompting an LLM to choose the preferred response given a preference pair and rubric as input and only retaining rubrics that yield agreement with human-provided preference labels (i.e., preference label consistency)5.

CRG and rubric filtering aim to create rubrics that are both prompt-specific and aligned with human preference examples, allowing them to serve as useful anchors for reward modeling. The result of this rubric generation and filtering approach is OpenRubrics, the key statistics of which are summarized in the plots below.

“After collecting the rubrics-based dataset, we proceed to develop a rubric generation model that outputs evaluation rubrics and a reward model Rubric-RM that generates final preference labels.” - from [3]

Rubric-RM. OpenRubrics provides a high-quality dataset of preference pairs and rubrics. In [3], this data is used to train two kinds of models (both of which are based upon Qwen-3-4B or Qwen-3-8B):

1. A rubric generation model—trained via SFT—that, given a prompt, can produce a discriminative rubric for predicting preference labels.

2. A reward model—also trained via SFT6—called Rubric-RM that can predict rubric-guided, pairwise preferences.

At inference time, these two models are used in tandem. Given a prompt, we first use the rubric generation model to produce our rubric. Then, Rubric-RM ingests this rubric, the prompt, and a pair of completions to generate a final preference prediction. We can also use majority voting (i.e., running this pipeline several times and taking the most frequently outputted score) to improve accuracy. Although using a two-stage pipeline increases inference costs, authors mention that costs can be decreased significantly by caching generated rubrics.

Comparison to other reward models. Rubric-RM is compared to a wide variety of other reward models and LLM-as-a-Judge approaches on several key evaluation benchmarks; see above. Rubric-RM tends to outperform similarly-sized baselines; e.g., the 8B variant gets 70.1% average accuracy, whereas the strongest 7B-scale reward model (RM-R1-7B) has an average accuracy of only 61.7%. These results are made even stronger with the use of majority voting. When comparing to the Qwen-3 base models, we see a noticeable uplift in preference scoring accuracy for Rubric-RM, highlighting the effectiveness of the finetuning strategy in [3].

“Rubric-RM excels on benchmarks requiring fine-grained instruction adherence… This demonstrates that rubrics capture nuanced constraints better than scalar reward models.” - from [3]

The gains from Rubric-RM are most pronounced on instruction-following tasks, which means that the rubrics in [3] work well for explicit evaluation criteria. On the other hand, this finding indicates less impact for subjective criteria, revealing that improving rubric supervision for open-ended tasks is still an open problem.

Application to post-training. Beyond evaluating Rubric-RM on reward modeling benchmarks, we can also measure the model’s downstream impact by using it as a reward signal in LLM post-training. Downstream evaluations in [3] only consider instruction following tasks (i.e., IFEval, InfoBench, and IFBench)—likely because this is the domain on which Rubric-RM excels—and use DPO for preference tuning. Rubric-RM is found to yield a boost over other reward models; see below.

DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research [4]

“Deep research (DR) models aim to produce in-depth, well-attributed answers to complex research tasks by planning, searching, and synthesizing information from diverse sources” - from [4]

Rubrics are studied in the context of deep research (DR) agents in [4]. A DR agent is an LLM that is taught to perform multi-step research and produce long-form answers—or surveys—that answer a query with detailed information and citations. This idea was popularized by Gemini DR and followed shortly after by DR agents from OpenAI, Anthropic, and more. Though many closed models support DR mode, open models are behind in this area: most open DR models are either prompt-based or trained on short-form, search-intensive QA tasks (i.e., not reflective of frontier DR agents) with RLVR. To solve this, authors in [4] train Dr. Tulu-8B—a fully-open7 LLM agent for long-form, open-ended DR tasks—using a novel online RL technique that evolves instance-level rubrics alongside the policy throughout training.

Definition of DR. Before describing Dr. Tulu, we need to understand the basic mechanics of DR agents. Details of closed DR agents are not publicly disclosed, but we can discern from using these agents that they:

1. Heavily rely on search tools to ground their answers in external knowledge.

2. Output long answers (i.e., basically survey papers) with many citations.

Authors in [4] use these observations to formalize an action space for DR agents; see below. In this formulation, a DR agent has the ability to i) think, ii) call a set of search tools, iii) provide a final answer, and iv) insert citations into the final answer. For all actions, any context that is output (e.g., thinking traces or tool outputs) is just concatenated to the sequence being processed by the DR agent. The DR agent itself is just an LLM that performs tool use in this action space.

Rubrics for DR. Evaluating a DR agent is a tough task. These agents generate lengthy outputs with detailed information, so there are many ways that an output could be good or bad—a static or predefined set of rubrics will not capture the detailed quality dimensions required for this task. Additionally, evaluation varies depending on the query (e.g., asking for a vacation plan versus an AI research survey).

Given that most DR queries are knowledge-intensive, we must also verify key information against known world knowledge. For this reason, synthetically generating instance-specific rubrics with an LLM—as in [1, 3]—is insufficient. This approach relies upon the parametric knowledge of the LLM rather than grounding on external knowledge that can be used to verify correctness. Ideally, we should ground the evaluation process in knowledge retrieved via search tools rather than relying on the (incomplete) parametric knowledge of an LLM.

Evolving rubrics. To address the unique considerations of DR tasks, Dr. Tulu is trained using a modified rubric-based RL technique, called Reinforcement Learning with Evolving Rubrics (RLER), that derives a reward from instance-specific rubrics that i) evolve alongside the policy during training and ii) are grounded in knowledge from the internet; see above. Similarly to prior work, rubrics are defined as a set of weighted criteria. Each of these criteria can be scored with a separate LLM judge to derive a final score as shown below. This formulation matches the explicit aggregation strategy proposed in [1].

During training, we have a buffer of rubrics for each prompt that stores a set of evolving rubrics specific to that prompt. Within this buffer, we designate certain rubrics as active, and these active rubrics are used for deriving the reward in the current training iterations. To initialize the buffer, we first create a set of search-based rubrics using an LLM with access to search tools. These initial rubrics are used persistently—meaning they are always included in the active set of rubrics—throughout training. At each training step, we prompt an LLM to generate a set of new (or evolving) rubrics given a prompt, a group of corresponding rollouts, and the set of active rubrics for that prompt as context; see below. Specifically, there are two types of rubrics that can be created by the LLM:

1. Positive Rubrics: capture strengths of new relevant knowledge explored by the current policy but not yet present in any rubric.

2. Negative Rubrics: address common undesirable behaviors of the current policy (e.g., reward hacking).

During RLER, the number of evolving rubrics can become large. To avoid this, we maintain a subset of active rubrics—always containing the initial persistent rubrics—via an explicit management strategy that filters and ranks rubrics based on their discriminative power. To measure a rubric’s discriminative power, we rely upon the group of completions created for advantage computation in GRPO. During each policy update, the group of completions for a given prompt is scored using all active rubrics for that prompt, and rubrics with zero reward variance (i.e., no discriminative value) are removed. Remaining rubrics are ranked in descending order based on the standard deviation of rewards across the group. Only rubrics with the top K standard deviations—and persistent rubrics—remain active.

The evolving rubrics in [4] are grounded in external knowledge and allow the reward for RL to adapt to the current state of our policy. As the model discovers new behaviors (e.g., a reward hack), these changes can be identified and captured in a new or modified rubric to maintain training fidelity. For this reason, we do not need to create a rubric a priori that exhaustively captures all desiderata for evaluation, which is difficult for DR tasks. Rather, this system can observe policy behavior and automatically incorporate key trends into new rubrics.

The rubric evolution process is found in [4] to have interesting characteristics, such as producing rubrics with measurably higher levels of specificity or even negative rubrics that penalize specific behaviors within the LLM; see above.

Dr. Tulu-8B is trained using a two-stage approach that includes a cold start SFT phase and online RL with GRPO. The Qwen-3-8B base model used in [4] does not yet possess the necessary atomic skillset (e.g., proper planning or citations) for solving DR tasks. If we were to begin RL training directly from this model, most rollouts would be of low quality, and the training process would likely struggle to efficiently discover high-reward solutions via exploration. To solve this issue, a cold start SFT phase is performed in [4] prior to RL training by sampling DR trajectories from a strong teacher model—in this case GPT-5 with a detailed system prompt describing the DR task—for supervised training. By finetuning the Qwen-3 base model on these trajectories, we allow the model to quickly learn a better initial policy for searching, planning, and citing sources prior to online RL. Given that most open DR agents are trained on short-form QA tasks, these supervised trajectories, which are openly available, are by themselves a useful artifact.

After cold start SFT, we perform online RLER using GRPO (with token-level loss aggregation) as the RL optimizer. Efficiently generating rollouts for online RL with a DR agent is a non-trivial systems problem due to output length and the frequency of tool calls. Rollouts are already the largest bottleneck in RL. Adding tool calls into the mix (i.e., “agentic” rollouts) makes this problem even worse. To improve efficiency, authors in [4] use one-step asynchronous RL training. Rollout generation and policy updates are performed at the same time, but policy updates are performed on rollouts from the prior training step. Additionally, tool calls are executed immediately to overlap generation and tool calling as much as possible.

“Tool requests are sent the second a given rollout triggers them, as opposed to waiting for the full batch to finish… Once a tool call is sent, we place that given generation request to sleep, allowing the inference engine to potentially continue to work on generating other responses while waiting for the tool response. This results in the generation and tool calling being overlapped wherever possible.” - from [4]

One other difficult aspect of RL training with a DR agent is the output lengths—generating long outputs (obviously) increases the time taken to produce a rollout. Plus, there can be high variance in output lengths. To mitigate this issue, sample packing is adopted during RL training, which improves efficiency by combining multiple outputs into a single, fixed length sequence. Finally, a few additional sources of heuristic rewards are used on top of RLER to encourage correct formatting and sufficient usage of search and citation tools by the agent.

Performance and efficiency. Dr. Tulu-8B is evaluated on several DR benchmarks (ScholarQA, HealthBench, ResearchQA, and DeepResearchBench), where we see that it substantially outperforms other open DR agents—even those that are larger (e.g., Tongyi-DR-30B-A3B)—and frequently matches the performance of the top proprietary systems. Additionally, Dr. Tulu-8B is smaller and cheaper compared to other systems. Notably, Dr. Tulu-8B is up to three orders of magnitude cheaper than OpenAI DR in some cases; e.g., costs are reduced from $1.80 per query to $0.0019 per query on ScholarQA8. Much of these savings come from the ability to call the correct tools and avoid excessive tool usage that drastically increases API costs. Not only does Dr. Tulu-8B generally make fewer tool calls, but authors observe in [4] that the model heavily calls free paper search tools for academic benchmarks while only using paid web search tools for more general queries.

Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Training [5]

Rubrics are helpful for performing granular evaluation, assuming that the rubric we are using is of high quality. To curate a high-quality rubric, we rely upon human annotators or synthetic generation. Relying on human oversight would make it difficult to scale rubric curation. On the other hand, synthetic rubrics are scalable, but static models are often used to generate and evaluate these rubrics, which limits adaptation to new domains. To make this process more dynamic, a joint training procedure for rubric generation and evaluation is proposed in [5].

“Rubric-ARM [is] a framework that jointly optimizes a rubric generator and a judge using RL from preference feedback. Unlike existing methods that rely on static rubrics or disjoint training pipelines, our approach treats rubric generation as a latent action learned to maximize judgment accuracy. We introduce an alternating optimization strategy to mitigate the non-stationarity of simultaneous updates.” - from [5]

Rubric-ARM. There are two models being trained in this framework: a rubric generator and an LLM judge. These models are trained with an alternating RL framework that switches between training each model. This approach, called Rubric-ARM, jointly optimizes the generator’s ability to create a rubric and the judge’s ability to predict human-aligned preference scores given a rubric as input. By learning these components together (i.e., instead of using separate training pipelines), we allow them to co-evolve and reinforce each other throughout training.

A rubric is defined in [5] as a set of evaluation criteria that are conditionally generated given a prompt as input—no explicit per-criterion weights are defined. Given a rubric sampled from the rubric generator, the objective—for both the rubric generator and the judge—is to maximize the preference accuracy of scores output by the judge. Notably, Rubric-ARM only considers preference data. The LLM judge is trained to predict a preference label (i.e., instead of performing direct assessment) given a prompt and two possible completions as input.

Training pipeline. Prior to RL, Rubric-ARM performs a cold-start SFT phase that trains both the rubric generator and the judge over a synthetic dataset curated from a variety of open data sources (e.g., UltraFeedback, Magpie, and more). From here, we begin the alternating RL procedure that switches between training the rubric generator or judge while keeping the other fixed. Alternating the learning process gives each component a clear training signal by keeping the other fixed.

At each training iteration t, we sample a batch of preference data. A rubric is then sampled—and cached for future use—with the rubric generator for each prompt in the batch. First, the rubric generator is kept fixed, and we perform RL training (with GRPO) to update the judge. The reward is defined as a sum of:

• Preference accuracy: a binary score indicating whether the predicted label matches the ground-truth label.

• Correct formatting: a heuristic that checks the judge’s trajectory for expected components (i.e., addressing each rubric criterion, providing per-criterion explanations, and finishing with an overall justification and decision).

Rubrics are generally sampled once and used for multiple judge optimization steps. After training the judge, we then freeze the judge’s weights and update the rubric generator. The rubrics used during this phase are cached, as the rubric generator was not trained during the prior phase. To train the rubric generator, we only use a preference accuracy reward based on whether the fixed judge is able to predict a correct preference label given the generated rubric. We learn from experiments in [5] that the optimization order is important. Training the rubric generator before the judge leads to noticeably degraded performance.

“Early-stage exploration by the rubric generator can dominate the learning dynamics. To mitigate this, we first stabilize the reward model under fixed rubrics before optimizing the rubric generator. This alternating schedule reduces variance and ensures robust optimization.” - from [5]

Application to post-training. The rubric generator and judge obtained from Rubric-ARM can also be applied to LLM post-training. Beginning with a set of prompts, we do the following:

1. Sample a rubric for each prompt with the rubric generator.

2. Sample two completions for each prompt using our current policy.

3. Score the completions using the judge with the above rubric9.

4. Perform DPO using preference data created with the above steps.

We are not restricted to offline training either! The above steps can easily be generalized to a semi-online DPO setup by regularly sampling new, on-policy completions and performing DPO training in phases to increase the freshness of preference data. We can even perform fully-online RL by modifying the above steps with a pairwise RL approach [6]. More specifically, we do the following for each prompt:

1. Sample a deterministic (baseline) completion with greedy decoding.

2. Sample a group of rollouts using a normal sampling procedure.

Once we have these completions, we use them to derive a direct assessment reward from the pairwise comparisons predicted by the LLM judge. To do this, Rubric-ARM creates preference pairs between each rollout in the group and the baseline completion. Then, our reward is defined as whether Rubric-ARM correctly predicts the greedy baseline as the rejected completion; see below.

“Rubric-ARM outperforms strong reasoning-based judges and prior rubric-based reward models, achieving a +4.7% average gain on reward-modeling benchmarks, and consistently improves downstream policy post-training when used as the reward signal.” - from [5]

How does this perform? Rubric-ARM is trained on the general-domain portion of OpenRubrics [3]. Both the rubric generator and LLM judge use Qwen-3-8B as a base model, and a two-stage rubric judging process—including generating and evaluating the rubric—is used at inference time. Rubric-ARM is compared to several open and closed LLM judges, as well as an SFT baseline trained on the same data (i.e., the Rubric-RM model [3]). Metrics on a wide variety of alignment-related reward modeling benchmarks are provided below. As we can see, Rubric-ARM outperforms all other open models and matches or exceeds the performance of most closed judges. Additionally, Rubric-ARM improves the performance of the SFT baseline by 4.8% absolute, indicating that alternating RL is helpful for discovering more discriminative rubrics and improving judge performance.

Rubric-ARM is also tested on WritingPreferenceBench, an out-of-distribution benchmark, where we see that the system generalizes well to other domains and continues to outperform baselines even on a very open-ended task (i.e., creative writing). Authors also run several ablation experiments, where we learn that:

• The optimization order for alternating RL is important; i.e., training the rubric generator first (instead of the judge) degrades preference accuracy by 2.4% with the largest regressions seen on instruction-following tasks.

• Removing the format reward used for the judge is harmful; i.e., LLM judges trained with only correctness rewards perform 2.2% worse than those trained on a combination of correctness and format rewards.

Similar results hold true when Rubric-ARM is used for LLM post-training. Rubric-ARM yields a boost in policy performance in both online and offline alignment scenarios, and policies trained with Rubric-ARM outperform those trained with other open models. Of the methods that are considered, iterative DPO with Rubric-ARM yields the best results, indicating that Rubric-ARM excels in creating high-quality preference data for LLM post-training; see below.

Further Reading

Although we have already covered a variety of papers, RaR is a particularly active and popular topic. To give a more comprehensive picture of the current research landscape, we close with high-level summaries of several more related works.

RL from Checklist Feedback (RLCF) [8] proposes a rubric-based approach for aligning language models to follow complex instructions. Instead of deriving rewards from a reward model trained on a static preference dataset, RLCF uses an LLM to generate instruction-specific checklists that outline the requirements of the instruction as a series of itemized steps. Each component of the checklist is an objective yes or no question that can be evaluated to derive a reward signal.

Rule-based rewards [9] propose an approach to LLM safety alignment that derives a reward signal from an explicit set of rules. Safety alignment is usually handled via RLHF-style preference tuning. However, this process requires collecting preference data, which is expensive, scales poorly as requirements evolve, and offers limited fine-grained control. As an alternative, the authors in [9] explore a hybrid setup in which an LLM evaluates responses against a specified set of safety rules, enabling fine-grained control over refusals and other safety-related behavior. This rule-based reward model is combined with a standard reward model for general helpfulness, allowing the model to undergo a standard alignment procedure with rule-based rewards guiding safety behavior.

Context-Aware Reward Modeling (CARMO) [10] attempts to mitigate problems with reward hacking in human preference alignment with RLHF. Going beyond static evaluation rubrics, an LLM first dynamically generates evaluation criteria for each prompt. Then, these criteria are used by the LLM to score the response, and the score can be directly used as a reward signal for preference alignment.

Reinforcement Learning with Adversarial Critic (RLAC) [11] proposes an adversarial approach for training LLMs on open-ended generation tasks. This framework has three components:

• Generator: the LLM being trained.

• Critic: another LLM that identifies potential failure modes.

• Validator: a domain-specific verification tool.

For each prompt, the generator produces multiple outputs, the critic proposes validation criteria—or a rubric—for each output, and the validator provides binary feedback based on correctness. Preference pairs can be formed between outputs that are validated and those that fail, naturally providing data to update the generator with DPO. At the same time, the critic is actively trained to identify criteria that the generator is unable to satisfy. This creates a dynamic in which the generator constantly improves its outputs as the critic finds weaknesses.

Auto-Rubric [12] aims to avoid the need for extensive preference data collection in LLM alignment by extracting generalizable evaluation rubrics from a minimal amount of data with a training-free approach. These rubrics are transparent and interpretable, unlike standard reward models that are trained over large volumes of preference data. To derive these rubrics, authors adopt a two-stage approach:

1. Query-Specific Rubric Generation focuses on creating rubrics that agree with observed preference data. After proposing an initial rubric set, we can check whether these rubrics yield correct preference scores and, if not, propose a set of revisions to derive an improved rubric set. This process repeats until the rubrics correctly predict human preference labels.

2. Query-Agnostic Rubric Aggregation eliminates redundancy and unnecessary complexity in the resulting rubric set. With an information-theoretic approach, the rubric set is narrowed to a subset of rubrics that maximize evaluation diversity without introducing redundancy.

Using this approach, Auto-Rubric can extract underlying general principles from preference data, allowing smaller LLMs to outperform large and specialized LLMs on reward modeling benchmarks with minimal training data.

Conclusion

Rubrics decompose desired LLM behavior into self-contained criteria that an LLM judge can score and then aggregate into an overall evaluation or reward. Put simply, rubrics are a practical middle ground between deterministic verifiers and preference labels that allow us to extend RLVR beyond verifiable domains while retaining granular control over output quality. The work we have studied suggests rubric rewards are most reliable when criteria are specific (often instance-level), grounded (via references or retrieval), and carefully curated (usually with human oversight). In more advanced setups, rubrics can also be updated based on on-policy behavior, allowing the rubric to adapt instead of becoming stale or exploitable. Despite promising results, key challenges remain; e.g., reducing reliance on human supervision and improving robustness in highly subjective domains. As reasoning models and LLM judges become more capable, however, rubric-based RL is becoming a viable and general tool across a wider variety of domains.

New to the newsletter?

Hi! I’m Cameron R. Wolfe, Deep Learning Ph.D. and Senior Research Scientist at Netflix. This is the Deep (Learning) Focus newsletter, where I help readers better understand important topics in AI research. The newsletter will always be free and open to read. If you like the newsletter, please subscribe, consider a paid subscription, share it, or follow me on X and LinkedIn!

Subscribe now

Bibliography

[1] Gunjal, Anisha, et al. “Rubrics as rewards: Reinforcement learning beyond verifiable domains.” arXiv preprint arXiv:2507.17746 (2025).

[2] Huang, Zenan, et al. “Reinforcement learning with rubric anchors.” arXiv preprint arXiv:2508.12790 (2025).

[3] Liu, Tianci, et al. “Openrubrics: Towards scalable synthetic rubric generation for reward modeling and llm alignment.” arXiv preprint arXiv:2510.07743 (2025).

[4] Shao, Rulin, et al. “Dr tulu: Reinforcement learning with evolving rubrics for deep research.” arXiv preprint arXiv:2511.19399 (2025).

[5] Xu, Ran, et al. “Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Training.” arXiv preprint arXiv:2602.01511 (2026).

[6] Xu, Wenyuan, et al. “A Unified Pairwise Framework for RLHF: Bridging Generative Reward Modeling and Policy Optimization.” arXiv preprint arXiv:2504.04950 (2025).

[7] Zheng, Lianmin, et al. “Judging llm-as-a-judge with mt-bench and chatbot arena.” Advances in neural information processing systems 36 (2023): 46595-46623.

[8] Viswanathan, Vijay, et al. “Checklists are better than reward models for aligning language models.” arXiv preprint arXiv:2507.18624 (2025).

[9] Mu, Tong, et al. “Rule based rewards for language model safety.” Advances in Neural Information Processing Systems 37 (2024): 108877-108901.

[10] Gupta, Taneesh, et al. “CARMO: Dynamic Criteria Generation for Context Aware Reward Modelling.” Findings of the Association for Computational Linguistics: ACL 2025. 2025.

[11] Wu, Mian, et al. “Rlac: Reinforcement learning with adversarial critic for free-form generation tasks.” arXiv preprint arXiv:2511.01758 (2025).

[12] Xie, Lipeng, et al. “Auto-rubric: Learning to extract generalizable criteria for reward modeling.” arXiv preprint arXiv:2510.17314 (2025).

[13] Bai, Yuntao, et al. “Constitutional ai: Harmlessness from ai feedback.” arXiv preprint arXiv:2212.08073 (2022).

[14] Guan, Melody Y., et al. “Deliberative alignment: Reasoning enables safer language models.” arXiv preprint arXiv:2412.16339 (2024).

[15] Liu, Yang, et al. “G-eval: NLG evaluation using gpt-4 with better human alignment.” arXiv preprint arXiv:2303.16634 (2023).

[16] Arora, Rahul K., et al. “Healthbench: Evaluating large language models towards improved human health.” arXiv preprint arXiv:2505.08775 (2025).

[17] Deshpande, Kaustubh, et al. “Multichallenge: A realistic multi-turn conversation evaluation benchmark challenging to frontier llms.” Findings of the Association for Computational Linguistics: ACL 2025. 2025.

Continual learning, which refers to the ability of an AI model to learn from new tasks and data over time, has become a popular topic in the discussion of Artificial General Intelligence (AGI). Put simply, general intelligence should be adaptable, which has led some to believe that continual learning abilities are a prerequisite for AGI. The reasoning behind this argument is clear—dynamically adapting to arbitrary tasks (i.e., “on-the-job” learning) is a common trait of humans—but rigorously studying this concept is hard. In the real world, continual learning is unstructured, noisy, and open-ended. In order to make meaningful progress, we must transform this complex process into a more structured empirical setting.

“LLMs don’t get better over time the way a human would. The lack of continual learning is a huge huge problem. The LLM baseline at many tasks might be higher than an average human’s. But there’s no way to give a model high level feedback. You’re stuck with the abilities you get out of the box.” - Dwarkesh Patel

To do this, we can pull from decades of prior research on the topic of continual learning for neural networks [10]. Although much of this work predates LLMs, such research provides a foundational understanding of continual learning and addresses key questions that are still relevant in the modern era:

• Why is continual learning difficult?

• How should we structure continual learning experiments?

• Which techniques are effective in practice?

In this overview, we will bridge decades of continual learning research with more recent work on LLMs to develop a comprehensive perspective on the topic. While core concepts (e.g., catastrophic forgetting, experimental frameworks, method categories, etc.) carry over directly, continual learning for LLMs is unique because of scale. Even simple techniques become complex systems problems when considering the vast data and prior knowledge of modern LLMs. As we will learn, however, continual learning is not disjoint from current LLM research. Rather, existing post-training techniques—especially on-policy reinforcement learning (RL)—can naturally mitigate catastrophic forgetting, providing hope that continual learning is within reach given the current trajectory of LLM research.

Basics of Continual Learning

The continual learning paradigm is starkly different from how neural networks are typically trained: for several epochs over a large, fixed dataset. Modern LLM training pipelines already include a mix of offline and more iterative components. Some stages (e.g., pretraining) closely resemble classical offline training, while others (e.g., iterative RLHF or online RL) begin to capture aspects of continual learning. In this section, we will develop a foundational understanding of continual learning—how it is studied, common experimental frameworks, and the major categories of methods proposed for both LLMs and neural networks more broadly.

Catastrophic Forgetting

Historically, the difficulty of continual learning does not stem from a model’s inability to learn new tasks, but rather its tendency to degrade in performance on old tasks when training on new data. For example, running supervised training of an LLM over a new dataset will quickly enhance its in-domain performance. But, the same model may significantly deteriorate in its performance across general benchmarks or tasks that were observed previously in the training process.

In continual learning research, this phenomenon is referred to as “catastrophic forgetting”1 [11]. Training our model over new data tends to come at the cost of a significant—or catastrophic—degradation in performance on other tasks. The goal of research in this area is, therefore, to mitigate catastrophic forgetting. The figure below helps us to better understand this phenomenon. Here, a model is initially trained on task A (grey) before being exposed to a new task (yellow).

The three arrows in the figure depict three possible solutions that can emerge when trying to solve this continual learning problem. The red arrow depicts a solution that performs well on both tasks, while the blue and green arrows perform well on only the new task or neither task, respectively. Put simply, the goal of continual learning is to develop techniques that reliably follow the red arrow. More specifically, an effective continual learning system should both:

1. Perform well on new tasks to which it is exposed.

2. Maintain comparable (or better) levels of performance on prior tasks.

As we will see throughout this overview, these two objectives are usually at odds—we are constantly balancing general capacities with new tasks. Simply specializing our model to each new incoming task is not a valid approach because new tasks will always continue to emerge in a real-world setting. We must maintain the model’s generality while maximizing adaptability to arbitrary future tasks.

Experimental Frameworks for Continual Learning

There are many continual learning variants that have been studied in the literature; e.g., continual learning, lifelong learning, incremental learning, streaming learning, and more. Despite the many variants of continual learning that exist, all of these variants share the same sequential nature of the training process—the model is exposed to new data over time and cannot return to data from the past (unless explicitly stored in a buffer) when learning from new data; see below.

Non-IID data. First, we must consider the kind of data being exposed to our model. If the incremental data over which the model is trained is sampled from the model’s training distribution, then training on this data is unlikely to cause forgetting. This setup resembles a continued training approach, which is used frequently for LLM pre and post-training. However, if the incremental data is non-IID—or sampled from a distribution that is new or different from the training data distribution—then catastrophic forgetting becomes very likely; see below.

For this reason, most experimental frameworks for continual learning assume the use of non-IID data. For example, when training an image classification model, we can derive incoming data from previously unseen classes. Similarly, we can continually train an LLM on an unseen task. In both cases, we expose the model to an unseen or different distribution of data that can induce catastrophic forgetting.

Data increments. We now need to understand the different approaches for exposing data to the model during continual learning. The most common sequential learning setup is a batch-incremental learning approach, where entire batches of data are passed to the model sequentially. These batches can be arbitrarily large (e.g., an entire new dataset or task) and the model usually trains on each batch of data before moving on to the next batch; see below.

Formally, we have a sequence of T tasks, each with an associated dataset or batch {D_1, D_2, …, D_T}. The model is sequentially trained on each task (i.e., one-by-one and in order), leading to a sequence of T models throughout the continual learning process. When training on a new task, we do not have access to prior tasks’ data. The simplest variant of batch-incremental learning is a domain adaptation setup where T = 1. For this setup, a pretrained model is trained on data from only a single new domain. The goal of continual learning in this scenario is the same, but the model only undergoes one stage of adaptation.

The batch-incremental framework may not always be realistic, as our model may receive data in much smaller increments. For these cases, a streaming learning setup may be more appropriate. Streaming learning uses brief, online updates (i.e., one or a few forward and backward passes) for each piece of incoming data, forcing learning of new data to happen in real-time; see below.

In contrast, batch-incremental learning setups usually perform a full, offline training procedure (i.e., several epochs of training) over each batch of incoming data. Although streaming and incremental learning setups are quite different, we can interpolate between these two approaches by:

• Changing the amount of data passed to the model at each phase of sequential learning (e.g., single example, batch of examples, entire dataset, etc.).

• Restricting the number of model updates at each sequential learning phase (e.g., single update, multi-update, full epoch, multi-epoch, etc.).

Multi-task learning. In order to determine if a continual learning technique is performing well, we need a baseline to which our models can be compared. A common baseline is joint (multi-task) training, where the model has access to all T tasks and can perform offline training over all of the data. Joint training over all data is the best possible training setup and allows us to understand the ceiling in performance that we are aiming to match via continual learning.

Which setup is best? In this overview, we will study a variety of continual learning papers in the LLM domain. Most of these papers adopt some variation of batch-incremental learning, where each batch is a new task that the LLM must learn. The domain-adaptation setup, in which a base LLM is trained over a single new task, is also common. These setups are useful for testing the tendency of LLMs to catastrophically forget, but one could argue that such a task-incremental setup does not reflect how LLMs would continually learn in the real world. For this reason, no one continual learning setup is the best. Rather, we should modify our experimental configuration within the frameworks outlined above such that the practical setting we are trying to test is most accurately reflected.

Common Techniques for Continual Learning

Now that we have a basic understanding of continual learning, we can overview some of the key categories of techniques for mitigating catastrophic forgetting. We will cover continual learning approaches in general, as well as highlight the methods that have been used in recent continual learning work with LLMs.

Replay mechanisms (depicted above) are a simple and effective technique for continual learning that maintain a buffer of prior data over which to train the model. Before being included in the replay buffer, samples usually undergo a selection process (e.g., based on importance or diversity) [14] to ensure that the buffer contains high-quality, representative samples and is not too large. The entire replay buffer can also be quantized or compressed to reduce memory [15]. In cases where data cannot be explicitly stored inside of a replay buffer, we can also train or maintain a generative model to replay synthetic examples [16, 17].

Although replay buffers are one of the most simple and effective techniques for continual learning, applying them in the LLM domain is less straightforward. Namely, LLMs have a vast amount of prior training data and, in many cases, this data is not openly available. Therefore, constructing a replay buffer that captures the general capabilities of an LLM is non-trivial. However, several works have recently explored the use of replay buffers for continual post-training. For example, instruction tuning data has a more manageable volume, allowing a replay buffer to be constructed by retaining the most important or informative data throughout the continual post-training process [30, 31]; see above.

Knowledge distillation [18] can be used to mitigate catastrophic forgetting by ensuring that a model’s representations do not drift during the continual learning process. In their simplest form, distillation-based continual learning techniques just combine the training loss on new data with a distillation loss with respect to prior model outputs [19]; see above. Many variants of this approach have been proposed [12, 20, 22]. We should also note that these techniques are not mutually exclusive; e.g., replay buffers can be combined with a distillation loss [13].

Regularization in various forms can be helpful for continual learning. In fact, knowledge distillation can even be considered a form of regularization. Researchers have explored constraining weight updates for subgroups of parameters—usually the most important parameters for a task [11, 21]—or increasing plasticity for select parameters [23]. We can also regularize the output distribution of the model by applying a KL divergence—similar to the use of KL to regularize the RL training objective—and even simple changes like lowering the learning rate have been found to reduce forgetting [2]. Model merging has also been applied in tandem with explicit regularization to reduce catastrophic forgetting in LLMs [29].

Architectural approaches have also been explored for continual learning that dynamically adapt the model’s architecture to handle incoming data. For example, new modules can be added to a neural network to handle new groups of data [24]; see above. Given the popularity of LoRA for LLMs, recent work has explored using LoRA modules as an architectural extension for learning new information during continual learning [26, 27]; see below. Mixture-of-Experts architectures for LLMs have also been shown to be better at avoiding catastrophic forgetting [28].

Further reading. We have now seen a comprehensive, high-level overview of the continual learning techniques that exist, but the literature is vast and dates all the way back to the 1980s (if not earlier)! The resources linked below will be helpful for developing a deeper understanding of continual learning research:

• A broad overview of the categories of continual learning techniques.

• A deep dive on streaming learning techniques.

• A survey on continual learning for modern generative models.

Continual Learning for LLMs

“Surprisingly, without any data replay, continual post-training with RFT can achieve comparable performance with that of multi-task training, which is not achievable even when equipping SFT with continual learning strategies.” - from [1]

We will now take a look at several papers that study the topic of continual learning in the context of LLMs. Instead of focusing on continual learning techniques, however, these papers adopt standard LLM training methodologies—supervised finetuning (SFT) and reinforcement learning (RL) in particular—and analyze their natural ability to avoid catastrophic forgetting. Although SFT tends to not perform well for continual learning, RL is found to be shockingly robust to forgetting, even without employing explicit continual learning techniques (e.g., replay buffers or regularization). Given the current popularity and impact of RL in training frontier models, this inherent ability to handle continual learning makes RL an important tool for the creation of generally intelligent systems.

More on SFT and RL

To understand the different behaviors of SFT and RL in the continual learning setting, we need to gain a deeper understanding of the learning mechanisms that underlie these algorithms. For a full overview of each technique, please see the following resources:

• Supervised Finetuning (SFT)

• Group Relative Policy Optimization (GRPO)

As we will see, all of the papers in this overview adopt a reinforcement learning with verifiable rewards (RLVR) setup with GRPO as the RL optimizer.

Training objectives. In SFT, we have a fixed dataset of supervised examples over which we are training our LLM. The training objective aims to minimize the model’s negative log-likelihood over this dataset, as shown below.

In contrast, RL uses the objective shown below, which focuses on maximizing the reward—such as a binary correctness signal in RLVR—of on-policy completions sampled for prompts taken from a fixed dataset. Optionally, we can include a KL divergence regularization term that penalizes the model for producing an output distribution that differs significantly from some reference model2.

Forward and reverse KL. One possible way to view the SFT and RL training objectives is through their relation to the KL divergence. Formally, the KL divergence is a measure for the divergence between two probability distributions; see here for full details. For two probability distributions P and Q, we can define the forward and reverse KL divergences as shown in the figure below.

In the LLM domain, these probability distributions are usually the next token distributions outputted by our LLM. A key difference between the forward and reverse KL divergence lies in the sampling—the distribution from which we sample in the above expectations changes. Specifically, we are either sampling from our dataset (offline) in SFT or from the LLM itself (online or on-policy) in RL.

SFT ≈ forward KL. Using these concepts, we can show that the training objective used by SFT is equal to the forward KL divergence up to a constant. Let’s call the optimal (or target) distribution for our dataset π_*. We can show the following for the relationship between this objective and the forward KL divergence, where H(π_*) denotes the entropy3 of the optimal distribution over the SFT dataset.

In the above expression, the entropy of the optimal distribution is a constant, so the forward KL and SFT training objective are equal up to a constant—minimizing forward KL is equivalent to minimizing the negative log-likelihood objective.

RL ≈ reverse KL. As mentioned previously, RL tries to maximize the reward of on-policy completions while minimizing KL divergence with respect to a reference policy. We can actually derive a closed-form expression for the optimal solution to the RL objective. The expression for the optimal policy is shown below, where Z(x) denotes the partition function. Notably, this optimal policy expression is also the first part of deriving the training loss for DPO!

If we assume that this optimal policy π_* is our target distribution, then we can show that maximizing the RL objective is equivalent to minimizing the reverse KL divergence between this target distribution and our policy π_θ.

As we can see, the first line of this equation computes the reverse KL divergence relative to the KL divergence used in our derivation for the SFT objective. In the final line, we have the negative of our RL objective (plus a scaling factor of 1/β and an additional constant). Therefore, minimizing this reverse KL divergence objective would be the same as maximizing the RL training objective.

What does this tell us? Now we understand the relation of SFT and RL to the forward and reverse KL divergence, respectively. But, what do these relationships actually tell us about the objectives? SFT minimizes negative log-likelihood over a dataset, which is equivalent to minimizing the forward KL divergence. This is a mode-covering objective. Our model is heavily penalized for assigning low probability to any completion that is found in the data—the model must “spread” its probability mass across all possible completions or modes in the data.

On the other hand, RL maximizes rewards of on-policy completions, which is equivalent to a reverse KL objective and is mode-seeking. Put differently, the model prioritizes high-reward outputs, even at the cost of ignoring output modes.

In SFT, the model’s loss increases exponentially if we assign near-zero probability to any completion in the dataset—this is due to the shape of the negative log-likelihood curve (shown above)! Such a property is not true of RL, as we are simply maximizing the reward of on-policy completions. Assigning near-zero probability to a completion will prevent this particular completion from being sampled during RL, but reward can still be maximized over the completions that are sampled. This is a fundamental property of RL that creates favorable behavior with respect to minimizing catastrophic forgetting during continual learning.

Reinforcement Finetuning Naturally Mitigates Forgetting in Continual Post-Training [1]

Continual learning can be viewed as a continued post-training process for an LLM. In this setup, the same base LLM undergoes extensive post-training over an evolving and expanding data stream, forcing the model to adapt to new requirements and learn new skills or knowledge without losing existing capabilities. However, avoiding catastrophic forgetting in this scenario is difficult. In [1], authors consider this continual post-training setup and analyze the best learning paradigm—either supervised finetuning (SFT) or reinforcement learning (RL)—for maximizing performance and minimizing forgetting.

Continual post-training. In the real world, continual learning is messy—the LLM will be constantly exposed to new data from various sources—but a more organized proxy setup is needed for research. A common way to simulate continual learning is via a sequential learning (or batch-incremental) setup, where the LLM is sequentially exposed to an ordered group of datasets. In [1], authors choose seven datasets that cover a wide scope of multi-modal (vision) use cases: ScienceQA, TextVQA, VizWiz, Geometry3K, GQA, PathVQA, and Super-CLEVR.

“A higher AvgAcc indicates better overall performance, while an FM closer to zero signifies less forgetting and better knowledge preservation.” - from [1]

Evaluation metrics. Our goal in continual post-training is to i) maximize the LLM’s performance on each new task and ii) avoid performance degradation—or catastrophic forgetting—on prior tasks. Assume that the LLM is evaluated on all tasks after each training round, yielding performance P_{t, j} on task j after learning for task t is complete. We can then capture key performance properties of continual post-training via the following two metrics:

1. Average accuracy (AvgAcc): the average accuracy of the model across all tasks after training on the final task T has completed.

2. Forgetting measure (FM): the average difference between the model’s final accuracy for a task and the best accuracy observed for that task throughout all T rounds of the training sequence.

After the end of the continual post-training process, the above metrics are computed over the test sets of all previously-encountered tasks. Going further, authors in [1] also measure performance on several general LLM benchmarks (i.e., MMMU, MMLU-Pro, and POPE) at the end of the continual post-training process to check for any impact on the model’s general capabilities.

SFT versus RL. Continual post-training experiments are performed in [1] using the Qwen-2.5-VL-7B-Instruct model, which is sequentially trained on data from each of the seven benchmarks. Notably, no replay buffer or data from prior tasks is used when training on new tasks, so the model’s ability to avoid forgetting is entirely dependent upon mechanics of the learning algorithm. As mentioned before, two types of learning algorithms are used:

1. Supervised Finetuning

2. Reinforcement Learning (GRPO, RLOO and ReMax)

For RL, we derive rewards using a standard reasoning model setup that combines the verifiable reward with a format reward that encourages the model to i) wrap its reasoning trace in <think> tokens and ii) mark its output with a \boxed{} label. Models output a reasoning trace prior to their final output, though tests are performed both with and without reasoning for all training setups.

RL forgets less. The results of the continual post-training experiments in [1] are depicted below. SFT clearly leads to catastrophic forgetting of previously-learned tasks, which gets worse as tasks move further into the past—forgetting is worst on initial tasks in the sequence. More specifically, we see an average accuracy of 54% with SFT, while multi-task training on all tasks reaches an average accuracy of 62.9%. Similarly, a FM of -10.4% is also observed for SFT, indicating that most tasks degrade noticeably in performance throughout continual post-training.

While SFT struggles to mitigate forgetting, RL naturally adapts well to new tasks. For GRPO, we observe an average accuracy of 60% (i.e., slightly below multi-task learning) and an FM of -2.3%. Additionally, the final accuracy on ScienceQA—the first task in the sequence—is 93%, compared to a peak accuracy of 95.6%. These results show that RL strikes a strong balance between learning and remembering.

“Without any data replay, continual post-training with RFT can achieve comparable performance with that of multi-task training, which is not achievable even when equipping SFT with continual learning strategies.” - from [1]

Influence on general capabilities. In the same vein, SFT-based continual post-training also degrades general model capabilities; see below. In contrast, we see in [1] that RL maintains—or even slightly enhances—performance on general benchmarks. For example, models sequentially trained with GRPO improve from an initial accuracy of 52.1% to a final accuracy of 54.2% on MMMU!

Such an ability to maintain performance on general benchmarks is a desirable aspect of continual learning. Ideally, we want the LLM to adapt to new tasks while maintaining its existing, foundational capabilities as much as possible.

Why does RL forget less? Given the above results, we might begin to wonder: Why does RL have the ability to naturally avoid catastrophic forgetting? Of course, it is possible that such continual learning abilities are directly attributable to RL itself. However, authors in [1] also consider two alternative explanations for the lack of catastrophic forgetting:

• The use of a KL divergence term in RL regularizes the training process and acts as a form of knowledge distillation that preserves prior knowledge.

• The use of long CoT reasoning in models trained with RL leads to a more robust knowledge base that is better protected from forgetting.

To test whether these factors help with avoiding catastrophic forgetting, three setups are tested that ablate the use of KL divergence and long CoT reasoning. Interestingly, we learn from testing these setups that removing KL divergence, despite degrading the stability of RL training4, does not lead to any degradation in performance metrics for continual post-training. Additionally, models that do not output a reasoning trace resist catastrophic forgetting similarly to those that do. Using CoT reasoning improves baseline model performance, but continually trained models in either setup see the same amount of catastrophic forgetting.

The results of these ablation experiments are outlined in the table above. The impressive performance of RL in continual post-training experiments does not seem to stem from the use of KL divergence or long CoT reasoning. Rather, the ability to perform continual learning seems to be an inherent property of RL training. Insight as to how RL avoids forgetting is provided by theory in [1] showing that RL naturally scales policy updates according to the variance of the reward signal, leading to more conservative updates for important or sensitive parameters.

“We offer a theoretical perspective suggesting that RFT’s updates are inherently more conservative in parameter subspaces sensitive to prior tasks. This conservatism is naturally scaled by the variance of the reward signal, creating a data-dependent regularization that dampens updates on uncertain samples, thus protecting established knowledge.” - from [1]

Retaining by Doing: The Role of On-Policy Data in Mitigating Forgetting [2]

Work in [2] shares a very similar focus to the paper above—trying to compare SFT and RL in the context of continual learning. However, a different experimental setup is used that considers three domains: instruction following (IFEval), general skills (MMLU), and arithmetic reasoning (Countdown). Beyond these target tasks that are used for training and evaluation, a few non-target tasks (i.e., MATH and two safety benchmarks) are included to provide a wider evaluation suite. We do not train the LLM over a sequence of tasks in [2]. Rather, the LLM is trained over one target task—a domain adaptation setup—and we measure performance via:

• The accuracy gain on that target task.

• The average accuracy drop across all non-target tasks.

Notably, the lack of multi-step sequential learning makes this setup less realistic. In [1], we see that the impact of catastrophic forgetting is greater after several training rounds. However, the domain adaptation setup in [2] does allow us to efficiently analyze the forgetting mechanics of different learning algorithms. The following learning algorithms are considered in [2]:

1. SFT training on responses from a teacher model (Llama-3.3-70B-Instruct).

2. Self-SFT training, which performs SFT-style training over responses from the initial policy (before training) or reference model.

3. RL training using GRPO with verifiable rewards—a standard RLVR setup.

Both SFT variants filter completions based on correctness, as determined by deterministic verifiers for each domain. Self-SFT is a rejection sampling setup (i.e., incorrect responses are rejected) that is used as a simple baseline, whereas the SFT setup performs offline knowledge distillation from a larger model. Self-SFT is an offline approach as well because completions are sampled from the initial model, rather than on-policy. The same verifiable correctness signal used for filtering completions in SFT variants is also used as the reward signal in RL.

RL forgets less (again). Experiments are performed in [2] using Qwen-2.5 and Llama-3 models with up to 8B parameters. As shown above, higher levels of forgetting—as measured via the average accuracy drop across non-target tasks—are observed with SFT compared to RL. In fact, Qwen-2.5 models see <1% average accuracy drop across all tasks and model scales for RL training, whereas the average accuracy drop with SFT reaches nearly 30% in some cases.

“RL leads to less forgetting than SFT while achieving comparable or higher target task performance… SFT suffers from severe forgetting, whereas RL can achieve high target task performance without substantial forgetting.” - from [2]

Despite the ability of RL to avoid catastrophic forgetting, the results with SFT are not actually bad—there is just a clear domain tradeoff. We can achieve performance improvements in the target domain via RL training, but models trained via SFT actually perform better. Unfortunately, the superior performance of SFT in the target domain comes at the cost of degraded performance on non-target tasks. For this reason, the comparison is not as simple as RL > SFT. Rather, RL and SFT lie at different points on the Pareto frontier of target and non-target task accuracy—better performance in one domain comes at the expense of the other.

Benefits of on-policy data. Similar to work in [1], authors in [2] show the lack of catastrophic forgetting in RL is not due to the inclusion of a KL divergence term in the objective; see above. Interestingly, the exact advantage formulation used by GRPO is also found to have little impact on continual learning capabilities—a naive REINFORCE-based RL setup is shown to mitigate forgetting to a similar extent. It is possible, however, that the continual learning abilities of RL stem from its use of on-policy samples—unlike the offline dataset used by SFT—during training. To test this theory, we consider the following training setups:

• On-policy SFT: running SFT using fully on-policy samples that are directly obtained from the RL training process.

• Iterative SFT: re-generating data for SFT after every epoch using the current policy (i.e., a partially on-policy approach).

Put simply, these approaches adapt SFT to use on-policy data, which allows us to decouple the impact of RL training and on-policy data. The use of iterative SFT also allows us to test a semi-on-policy scenario, which samples fresh on-policy data at the end of each epoch (i.e., instead of generating new samples during each training iteration). This coarse-grained approach to on-policy data has efficiency benefits—we can adjust the regularity with which we sample fresh on-policy data.

Experiments with these training algorithms provide empirical evidence that on-policy data is a key contributor to the success of RL in the continual learning domain. Specifically, models trained via on-policy SFT mitigate forgetting to a similar extent as those trained via RL. Additionally, the data used does not need to be fully on-policy—similar trends are observed with iterative SFT; see below.

Mode-seeking versus mode-covering. Intuitively, we might assume that the mode-covering nature of SFT would allow the model to maintain probability mass across all tasks and, therefore, avoid catastrophic forgetting. As we have seen, however, the opposite is true in practice. Such a finding is due to the fact that we are only training our model over a small subset of the model’s total data distribution in most of these experiments. Potentially our observations would be different if we were able to retain the LLM’s entire training dataset within a replay buffer, but implementing such an approach efficiently would be incredibly difficult.

In the standard LLM post-training setup, the mode-seeking behavior of RL is more robust to catastrophic forgetting. To explain this phenomenon, authors in [2] construct a simplified setting shown above, which illustrates the dependence of forgetting on the modality of the underlying target distribution. If our target distribution is multi-modal, which is likely to be true for an LLM, then the mode-seeking nature of RL actually leads to less forgetting relative to a mode-covering objective like SFT. The simplified distribution that is constructed in [2] has two modalities corresponding to old and new knowledge. For such a distribution, the forward KL objective yields noticeable forgetting while minimizing the reverse KL allows both modes of the target distribution to be properly captured.

RL’s Razor: Why Online RL Forgets Less [3]

As we know, SFT and RL achieve comparable performance when training on a new task but have drastically different forgetting dynamics. In most cases, gains on new tasks with SFT come at the cost of erasing prior knowledge, while RL is much better at protecting old capabilities; see below. By studying this gap in performance, authors in [3] identify a metric that reliably predicts the amount of forgetting that occurs for both SFT and RL: the distributional shift—measured via KL divergence—between the base and finetuned models on the target task.

RL’s Razor. In addition to discovering this relationship between the underlying distribution shift and forgetting, we see in [3] that the finetuned models from SFT and RL have unique properties:

• RL is biased towards solutions that minimize distributional shift.

• SFT can converge to solutions arbitrarily far away from the base model.

Such a property naturally implies the improved continual learning abilities of RL. By discovering a solution that minimizes distributional shift, we also minimize the amount of forgetting that occurs; see above. The bias of RL towards nearby solutions that minimize catastrophic forgetting is referred to in [3] as “RL’s Razor”5.

“RL’s Razor: among the many high-reward solutions for a new task, on-policy methods such as RL are inherently biased toward solutions that remain closer to the original policy in KL divergence… the KL divergence between the fine-tuned model and the base model, measured on the new task, reliably predicts… forgetting.” - from [3]

Distribution shift. In the LLM domain, we often measure the KL divergence between the next token distributions of two models. For example, the RL training objective has a KL divergence term that regularizes drift between the current and reference policy, where the KL divergence is computed using on-policy samples taken from the current policy during RL training. In [3], authors compute the KL divergence over data from the task on which our policy is being finetuned (i.e., the target task). We are restricted to using the target data because we rarely have access to the pretraining data (or any prior tasks) on which an LLM was trained.

This KL divergence between base and finetuned models on the target dataset can be viewed as capturing the distributional shift from training. We are computing the divergence between models before and after training over the training data itself. When measured in this way, the distributional shift is found to be consistently predictive of the amount of forgetting that occurs. Given that no prior data is used to compute this KL divergence, such a finding is highly non-trivial!

Experiments in [3] are performed using both vanilla SFT and RL with GRPO. The RL setup uses standard verifiable rewards and no KL divergence regularization. Similarly to [2], the base model (Qwen-2.5-3B-Instruct) is trained on one target task (i.e., Open-Reasoner-Zero, ToolAlpaca, or the Chemistry L-3 subset of SciKnowEval) and evaluated on both the target task and set of prior tasks (i.e., HellaSwag, TruthfulQA, MMLU, IFEval, WinoGrande, and HumanEval). Given that hyperparameter settings can massively impact results in a continual learning setup6, a wide variety of hyperparameters are tested for each task and results are visualized as a Pareto frontier constructed by all possible settings.

Lower KL leads to less forgetting. RL training improves target task performance while keeping performance on prior tasks stable. However, improvements in performance obtained via SFT come at the cost of noticeable forgetting. The deterioration in performance is most visible in the math domain; see below.

Identifying the cause of such forgetting is difficult due to the high computational cost of RL training—testing each hypothesis is quite expensive! To make this search more tractable, a toy setting is created based on the MNIST and FashionMNIST datasets for which RL training is much faster. Using this setting, a variety of candidate metrics are tested for a relationship to catastrophic forgetting:

• The magnitude of changes to model parameters.

• The sparsity of weight updates.

• The rank of policy gradients throughout training.

The only quantity that demonstrates a consistent relationship with the amount of catastrophic forgetting is the KL divergence between base and finetuned models over the target dataset; see below. The fact that the rank or sparsity of policy gradient updates is unrelated to forgetting is notable, as prior research [4] has shown that RL works surprisingly well even when using LoRA with a low rank. Such a finding indicates that the updates being produced by RL are potentially sparse or low rank, which could help to reduce forgetting. However, we see in [3] that the story is not this simple. Rather, the benefits of RL stem from an implicit KL regularization—or RL’s Razor—that minimizes distribution shift in training.

To further validate the relationship between the KL divergence and forgetting, authors create an “oracle” SFT distribution in their toy setting. Put simply, this experiment performs SFT on a dataset that has been analytically constructed to minimize the KL divergence between the base and finetuned models. As shown above, running SFT on this data yields an even better tradeoff than RL—the model performs better on the target task without sacrificing prior task performance.

“RL performs well because its on-policy updates bias the solution toward low-KL regions, but when SFT is explicitly guided to the KL-minimal distribution, it can surpass RL.” - from [3]

On-policy data. Beyond the toy example explained above, authors in [3] also run SFT training over on-policy data obtained during RL. The accuracy-forgetting tradeoff achieved by the resulting model matches that of those trained via RL, which aligns with prior work [2] and provides further evidence that on-policy data plays a key role in mitigating forgetting for RL. To better understand the impact of on-policy data, three different learning algorithms are tested (shown below):

• Standard GRPO.

• Standard SFT.

• 1-0 REINFORCE: an on-policy RL algorithm with a very simple advantage function (i.e., 1 if the answer is correct and zero otherwise).

• SimPO [5]: an offline preference tuning algorithm that simplifies DPO by directly using the log probability of a sequence as the implicit reward and, therefore, removing the need for a reference model.

As we can see in the left half of the above figure, these experiments ablate the use of negative examples and on-policy data within the training setup. Interestingly, the 0-1 REINFORCE algorithm performs similarly to GRPO, while results with SimPO resemble those of SFT. Such results indicate that the use of on-policy data is the key contributor to RL’s lack of forgetting. We also see above that the use of on-policy data leads to minimal KL divergence between the base and finetuned models over the target distribution. Such results indicate that the implicit bias of RL towards low KL solutions stems from the online nature of training. This empirical observation is also justified by further theoretical analysis in [3].

Entropy-Adaptive Fine-Tuning: Resolving Confident Conflicts to Mitigate Forgetting [6]

“While RL aligns with the model’s internal belief, SFT forces the model to fit external supervision. This mismatch often manifests as Confident Conflicts—tokens characterized by low probability but low entropy.” - from [6]

We have learned that RL avoids catastrophic forgetting much better than SFT due to its use of on-policy data, which allows for the discovery of a solution with minimal KL divergence between base and finetuned models on the target data. Although we know that these factors lead to less forgetting, we do not yet understand why this is the case. In [6], authors offer a new perspective on the forgetting properties of SFT and RL by analyzing token probabilities and entropy of models trained with these two approaches. When these two quantities are measured throughout the training process, we see that a clear gap exists:

• On-policy RL tends to cluster in regions of highly-confident and correct predictions—characterized by high probability and low entropy—or exploratory completions—characterized by high entropy.

• SFT has a significant cluster of tokens with both low entropy and low probability—these are referred to as “Confident Conflicts”.

To discover this distribution mismatch, token probability and predictive entropy is measured over both the SFT dataset and model-generated rollouts. This trend is visualized below, where we see that SFT data has a noticeable cluster of confident conflict tokens that does not exist when using on-policy data.

Why does this occur? We are using external supervision in SFT (i.e., an offline supervised dataset), whereas RL learns from on-policy or self-generated data. In some cases, training the model on external data forces it to mimic outputs that align poorly with its current next distribution—confident conflicts occur when external data has a strong conflict with the model’s prior. As a result, gradient updates can become large and destructive, leading to catastrophic forgetting.

Masking conflicts. To determine whether confident conflict tokens truly lead to forgetting, authors in [6] test simply masking the loss from such tokens during SFT. Interestingly, catastrophic forgetting is significantly reduced when these tokens are masked from the training loss, indicating that confident conflict tokens play a significant role in the tendency of SFT to damage prior knowledge; see below.

Extending this idea, a novel training algorithm, called Entropy Adaptive Finetuning (EAFT), is proposed in [6] that scales the token-level cross-entropy loss by a dynamic entropy factor. The new loss formulation is outlined below, which multiplies the supervised loss by the token’s normalized entropy7. By using this token-level entropy scaling factor, we can effectively mask the loss of low entropy tokens that lead to destructive gradient updates while maintaining the full update for high entropy tokens that are beneficial for exploration.

“EAFT employs a soft gating mechanism that dynamically modulates the training loss based on token-level entropy.” - from [6]

To improve the efficiency of EAFT, authors in [6] only compute entropy over the Top-K (where K = 20) tokens in the distribution. As shown in the figure below, this setting balances the tradeoff between compute and memory overhead and ensures that added computational overhead relative to vanilla SFT is minimal.

Results on Math. EAFT is validated in the Math domain using models across multiple families ranging from 4B to 32B parameters. Training prompts are sourced from NuminaMath, BigMathVerified, and Nemotron-CrossThink, while completions are sampled from Qwen-3-235B-A22B-Instruct. Both in-domain and general benchmarks are used for evaluation. Models trained with EAFT perform well in the target domain while maintaining performance on general benchmarks; see below. Additionally, EAFT is found to effectively filter confident conflict samples during the training process, as demonstrated by the visible reduction in gradient magnitude within the confident conflict zone of the below figure. These results are further validated in experiments in the medical and tool use domains.

Does RL Generalize Well?

So far, we have focused on retaining old skills while learning new ones. A closely related question is whether the same mechanisms that reduce forgetting also improve transfer and out-of-distribution generalization. The fact that RL performs well in a continual learning setting has important implications for its generalization properties. Put simply, RL training tends to benefit more than just the target domain. As we will see in the next few papers, there are many examples of RL training yielding cross-domain performance benefits or improving the generalization of an LLM to some other task. Much of this analysis is similar in nature to what we have seen for continual learning, but the emphasis shifts from remembering prior tasks to generalizing beyond the training distribution.

SFT Memorizes, RL Generalizes [7] performs a comparative post-training analysis between SFT and RL on both language-only and vision-language tasks. The main results of this analysis are depicted above, where we see that:

• Both SFT and RL improve in-domain performance.

• Only RL generalizes well to new tasks or data.

Experiments in [7] use Llama-3.2-Vision-11B as the base model and train over two synthetic tasks (shown below) that test distinct forms of generalization:

1. GeneralPoints: A card game that requires the model to create equations to reach a target number using four given cards. We can test rule-based generalization by changing the mapping of face cards to numbers.

2. V-IRL: A navigation task that has the model reach a destination using visual landmarks and spatial reasoning. We can test generalization by varying the available action space or visual context.

Each task can be setup as both a language-only and vision-language problem. In all experiments, RL tends to promote out-of-distribution generalization while SFT actually damages it. For example, the out-of-distribution performance of models trained with RL improves by 3.5% and 11.0% on language-only GP and V-IRL. For vision-language variants, this performance improvement is slightly less pronounced (i.e., 3.0% and 9.3% on GP and V-IRL) but still present. In stark contrast, SFT degrades out-of-distribution performance by as much as 79.5%.

As an interesting side note, authors in [7] also find that RL benefits the model’s underlying perception capabilities. Namely, the model actually improves in its ability to identify key vision features during training, indicating that RL is not just learning reasoning patterns but also improving fundamental abilities (i.e., perception).

“Analysis of the GP-VL task showed that RL improved the model's ability to correctly identify card values from images, suggesting that outcome-based rewards can refine perceptual processing beyond what supervised training achieves.” - from [7]

From Atomic to Composite [8] tests the generalization impact of RL training on problems that require complementary reasoning—the ability to integrate external context with the model’s parametric knowledge. To test this style of reasoning, a controlled synthetic dataset is created; see below. The dataset is based on a knowledge graph of human biographies with fixed relationships. Using this graph, we can construct multi-hop questions that test complementary reasoning by design. More specifically, questions are specifically constructed to test three levels of reasoning with increasing complexity (depicted below):

1. IID reasoning applies known patterns to new entities.

2. Compositional reasoning applies known relationships to new relational paths.

3. Zero-shot reasoning requires generalizing to unseen relations.

The training process in [8] starts with Qwen-2.5-1.5B, performs an initial SFT stage, then tests several combinations of SFT and RL (using GRPO with binary verifiable rewards) training. The main results of these experiments are below.

As we can see, this analysis shows us that RL is capable of synthesizing multiple atomic reasoning capabilities into higher-level (composite) reasoning patterns. However, this is only possible when the model is trained with SFT prior to RL. In contrast, pure SFT training yields high in-domain performance and poor out-of-domain generalization, which reflects findings in prior work. In other words, SFT tends to memorize reasoning patterns rather than learn them. When a model is first trained via SFT to acquire primitive reasoning capabilities, then RL serves as a “synthesizer” through which the model learns how to properly combine these capabilities for solving complex, compositional reasoning problems.

“[We demonstrate] that RL synthesizes novel reasoning strategies and enables robust zero-shot generalization when LLMs are first pre-trained on foundational atomic reasoning skills via Supervised Fine-Tuning.” - from [8]

Does math reasoning improve general capabilities? A large-scale empirical analysis is performed in [9] to determine whether math-oriented reasoning training is also helpful in other domains. This analysis includes both a wide audit of existing models across math reasoning, general reasoning, and non-reasoning benchmarks, as well as a comparison of SFT and RL-based finetuning on Math-only data (i.e., ~47K prompts sourced from DeepScaler and SimpleRL).

As shown in the above plot, SFT-trained models tend to have poor transferability to non-reasoning tasks, while models trained with RL generalize across both reasoning and non-reasoning tasks—RL models generalize well beyond math and naturally avoid catastrophic forgetting. Similar trends are observed when analyzing the transferability of other open SFT or reasoning models across reasoning and non-reasoning benchmarks; see below. Further analysis in [9] reveals that on-policy data—as we might expect from [2, 3]—and the presence of a negative gradient in the RL objective are key contributors to favorable generalization properties.

Conclusion

In continual learning, we want the model to learn new tasks quickly while preserving old capabilities. When studying recent work on continual learning for LLMs, a consistent pattern emerges: on-policy RL is naturally more robust to catastrophic forgetting relative to SFT, even without explicit mechanisms to aid the continual learning process. This advantage appears to stem from the online nature of RL, which biases learning toward low distribution shift (or low KL) solutions and avoids destructive updates induced by offline data. The natural continual learning abilities of RL have broader implications for the emergence of AGI, as adaptability is a key prerequisite for generally intelligent systems. The studies seen in this overview use only simple, structured proxies for continual learning in the real world, which will be much messier. However, these results show that RL—an already impactful training paradigm—is a promising starting point for building general systems that can adapt to any task. In this way, continuing the existing trajectory of LLM research may yield natural progress for continual learning.

New to the newsletter?

Hi! I’m Cameron R. Wolfe, Deep Learning Ph.D. and Senior Research Scientist at Netflix. This is the Deep (Learning) Focus newsletter, where I help readers better understand important topics in AI research. The newsletter will always be free and open to read. If you like the newsletter, please subscribe, consider a paid subscription, share it, or follow me on X and LinkedIn!

Subscribe now

Bibliography

[1] Lai, Song, et al. “Reinforcement fine-tuning naturally mitigates forgetting in continual post-training.” arXiv preprint arXiv:2507.05386 (2025).

[2] Chen, Howard, et al. “Retaining by doing: The role of on-policy data in mitigating forgetting.” arXiv preprint arXiv:2510.18874 (2025).

[3] Shenfeld, Idan, Jyothish Pari, and Pulkit Agrawal. “Rl’s razor: Why online reinforcement learning forgets less.” arXiv preprint arXiv:2509.04259 (2025).

[4] Lu, Kevin et al. “On-Policy Distillation.” https://thinkingmachines.ai/blog/on-policy-distillation/ (2025).

[5] Meng, Yu, Mengzhou Xia, and Danqi Chen. “Simpo: Simple preference optimization with a reference-free reward.” Advances in Neural Information Processing Systems 37 (2024): 124198-124235.

[6] Diao, Muxi, et al. “Entropy-Adaptive Fine-Tuning: Resolving Confident Conflicts to Mitigate Forgetting.” arXiv preprint arXiv:2601.02151 (2026).

[7] Chu, Tianzhe, et al. “Sft memorizes, rl generalizes: A comparative study of foundation model post-training.” arXiv preprint arXiv:2501.17161 (2025).

[8] Cheng, Sitao, et al. “From Atomic to Composite: Reinforcement Learning Enables Generalization in Complementary Reasoning.” arXiv preprint arXiv:2512.01970 (2025).

[9] Huan, Maggie, et al. “Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoning.” arXiv preprint arXiv:2507.00432 (2025).

[10] McCloskey, Michael, and Neal J. Cohen. “Catastrophic interference in connectionist networks: The sequential learning problem.” Psychology of learning and motivation. Vol. 24. Academic Press, 1989. 109-165.

[11] Kirkpatrick, James, et al. “Overcoming catastrophic forgetting in neural networks.” Proceedings of the national academy of sciences 114.13 (2017): 3521-3526.

[12] Rebuffi, Sylvestre-Alvise, et al. “icarl: Incremental classifier and representation learning.” Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. 2017.

[13] Castro, Francisco M., et al. “End-to-end incremental learning.” Proceedings of the European conference on computer vision (ECCV). 2018.

[14] Chaudhry, Arslan, et al. “On tiny episodic memories in continual learning.” arXiv preprint arXiv:1902.10486 (2019).

[15] Hayes, Tyler L., et al. “Remind your neural network to prevent catastrophic forgetting.” European conference on computer vision. Cham: Springer International Publishing, 2020.

[16] Rannen, Amal, et al. “Encoder based lifelong learning.” Proceedings of the IEEE international conference on computer vision. 2017.

[17] Shin, Hanul, et al. “Continual learning with deep generative replay.” Advances in neural information processing systems 30 (2017).

[18] Hinton, Geoffrey, Oriol Vinyals, and Jeff Dean. “Distilling the knowledge in a neural network.” arXiv preprint arXiv:1503.02531 (2015).

[19] Li, Zhizhong, and Derek Hoiem. “Learning without forgetting.” IEEE transactions on pattern analysis and machine intelligence 40.12 (2017): 2935-2947.

[20] Wu, Yue, et al. “Large scale incremental learning.” Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.

[21] Aljundi, Rahaf, et al. “Memory aware synapses: Learning what (not) to forget.” Proceedings of the European conference on computer vision (ECCV). 2018.

[22] Dhar, Prithviraj, et al. “Learning without memorizing.” Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.

[24] Rusu, Andrei A., et al. “Progressive neural networks.” arXiv preprint arXiv:1606.04671 (2016).

[25] Draelos, Timothy J., et al. “Neurogenesis deep learning: Extending deep networks to accommodate new classes.” 2017 international joint conference on neural networks (IJCNN). IEEE, 2017.

[26] Guo, Haiyang, et al. “Hide-llava: Hierarchical decoupling for continual instruction tuning of multimodal large language model.” arXiv preprint arXiv:2503.12941 (2025).

[27] Zhao, Hongbo, et al. “Mllm-cl: Continual learning for multimodal large language models.” arXiv preprint arXiv:2506.05453 (2025).

[28] Li, Hongbo, et al. “Theory on mixture-of-experts in continual learning.” arXiv preprint arXiv:2406.16437 (2024).

[29] Liu, Wenzhuo, et al. “LLaVA-c: Continual Improved Visual Instruction Tuning.” arXiv preprint arXiv:2506.08666 (2025).

[30] Maharana, Adyasha, et al. “Adapt-$\infty $: Scalable continual multimodal instruction tuning via dynamic data selection.” arXiv preprint arXiv:2410.10636 (2024).

[31] Lee, Minjae, et al. “OASIS: Online Sample Selection for Continual Visual Instruction Tuning.” arXiv preprint arXiv:2506.02011 (2025).

Recent research on large language models (LLMs) has been heavily focused on reasoning and reinforcement learning (RL). At the center of this research lies Group Relative Policy Optimization (GRPO) [13], the RL optimizer used to train most open-source reasoning models. The popularity of GRPO is enhanced by its conceptual simplicity and practical efficiency. However, the simplicity of GRPO can be deceptive—the vanilla GRPO algorithm has subtle issues that can hinder the RL training process, especially at scale. Solving the shortcomings of GRPO has become a popular research topic, leading to the proposal of many tricks, best practices, and techniques for getting the most out of RL training. In this overview, we will outline all of this work, arriving at a deeper practical understanding of how to modify and use GRPO for training high-quality reasoning models.

Background on Reasoning and RL

Prior to covering recent work on improving GRPO, we will spend this section building a basic understanding of the GRPO algorithm in its original form. We will also learn about Proximal Policy Optimization (PPO) [11], the predecessor to GRPO, and discuss how RL is used in the context of LLMs and reasoning models more generally. Notably, this discussion will assume basic knowledge of the problem setup and terminology used for RL training with LLMs. Those who are less familiar with RL basics can learn more at the following links:

• RL Problem Setup & Terminology [link]

• Different RL Formulations for LLMs [link]

• Policy Gradient Basics [link]

RL for Reasoning

“Inference scaling empowers LLMs with unprecedented reasoning ability, with RL as the core technique to elicit complex reasoning.” - from [1]

GRPO is the most common RL optimizer to use for training reasoning models. Before diving deeper into the details of GRPO, we need to build an understanding of how RL is actually used to train LLMs. In particular, there are two key types of RL training that are commonly used:

• Reinforcement Learning from Human Feedback (RLHF) trains the LLM using RL with rewards derived from a reward model trained on human preferences.

• Reinforcement Learning with Verifiable Rewards (RLVR) trains the LLM using RL with rewards derived from rule-based or deterministic verifiers.

The main difference between RLHF and RLVR is how we assign rewards—RLHF uses a reward model, while RLVR uses verifiable rewards. Aside from this difference, both are online RL algorithms with a similar structure; see below. GRPO is one possible RL optimizer that can be used to derive the policy update in this pipeline, though any RL optimizer (e.g., PPO or REINFORCE) can be used.

Given that RLHF focuses on aligning an LLM to human preferences, it is used more heavily for chat models and is less applicable to reasoning1. Most reasoning models are trained using RL in verifiable domains (e.g., math and coding), so we will primarily focus on the RLVR setup for the remainder of this post.

More on RLVR. To train an LLM with RLVR, we must select a domain that is verifiable in nature; e.g., math or coding. In other words, we need to create a dataset that has either i) a known ground truth answer or ii) some rule-based technique that can be used to verify the correctness of responses to the prompts in our dataset. For coding, we can create a sandbox for running LLM-generated code and use test cases to assess correctness. Similarly, we can evaluate math problems by performing basic string matching between the answer predicted by the LLM and a ground-truth answer for a problem; see below.

Usually, we must instruct the LLM to format its output so the final answer can be easily parsed. As an example, Math Verify is a popular package that was built for performing robust verification in the math domain. Even then, however, string matching is not always sufficient for evaluating correctness. In many cases, we can benefit from crafting validation logic that is more robust (e.g., asking an LLM to identify equivalent answers) and that captures variations in output.

“Math verification is determined by an LLM judge given the ground truth solution and DeepSeek-R1 solution attempt. We found that using an LLM judge instead of a stricter parsing engine (Math-Verify) for verification during data generation results in a higher yield and leads to higher performing downstream models.” - source

Reasoning models are structurally identical to a standard LLM. The key distinction between reasoning models and LLMs is the ability to “think” about a prompt prior to providing a final output. By increasing the length of this thinking process, reasoning models can use inference-time scaling—or simply spend more compute on generating a particular completion—to improve their performance.

As shown above, this thinking process occurs in the form of a free-text, long chain-of-thought (CoT)—also called a rationale or reasoning trajectory—generated by the LLM. Many closed reasoning models (though not all of them!) hide the raw reasoning trace from the user, providing instead only a truncated version or summary of the reasoning process along with the model’s final answer.

Learning to reason via RL. If we look at some examples of reasoning trajectories from open or closed reasoning models, we will notice that these models exhibit some sophisticated reasoning behaviors in their long CoT:

• Thinking through each part of a complex problem.

• Decomposing complex problems into smaller, solvable parts.

• Critiquing solutions and finding errors.

• Exploring many alternative solutions.

Such behavior goes beyond any previously-observed behavior with standard LLMs and chain of thought prompting. However, this behavior is not explicitly injected into the model—it is naturally developed via large-scale RL training!

During RLVR, the model undergoes a self-exploration process in which it learns how to properly use its long CoT to solve reasoning problems. As evidence of this self-evolution process, we commonly observe during RL training that the average length of the model’s completions increases over time; see below. The model naturally learns how to use more inference-time compute (by generating a longer reasoning trace) in order to solve difficult reasoning problems.

Training stages and Aha moments. As shown below, LLMs undergo training in several stages. However, reasoning models depart from the standard alignment procedure—including supervised finetuning (SFT) and RLHF—by adding an extra RLVR training stage. Additionally, it is even common in RL research to use an RL-Zero setup in which we directly train the pretrained base model with RLVR.

The RL-Zero setup was popularized by DeepSeek-R1 [2], which showed that reasoning capabilities can be instilled in an LLM via pure RL (using GRPO) even with no SFT. Most notably, DeepSeek-R1-Zero—the version of DeepSeek-R1 that is trained with an RL-Zero setup—is found in [2] to have an “Aha moment” in which it learns to invest additional reasoning effort into re-thinking or evaluating its own responses inside of the reasoning trace; see below. This behavior emerges at an intermediate point in RL training and is a classic example of how self-exploration via RL can naturally lead an LLM to develop sophisticated reasoning behavior.

Proximal Policy Optimization (PPO)

GRPO is based on the Proximal Policy Optimization (PPO) algorithm [11]. PPO was used in seminal work on RLHF and, as a result, was the default RL optimizer in the LLM domain for some time. Only after the advent of reasoning models did alternative algorithms like GRPO begin to gain traction for training LLMs. A full overview of PPO is linked below, but we will cover the key details in this section.

The structure of training with PPO is outlined below. As we can see, each training iteration of PPO goes through the following sequence of steps:

1. Sample a diverse batch of prompts.

2. Generate a completion from the policy for each prompt.

3. Compute advantage estimates for each completion.

4. Perform several policy updates over this sampled data.

Surrogate objective. In PPO, we formulate a loss function (also called the surrogate objective) that is optimized with respect to the parameters of our policy. The PPO loss function is based on the policy ratio (also called the importance ratio) between the current and “old” (i.e., before the first update in a training step) policies. The importance ratio stabilizes the training process by comparing the new policy’s token probabilities to the old policy and applying a weight (or importance) to training that helps to avoid drastic changes; see below.

To derive the surrogate objective for PPO, we begin with an unclipped objective that resembles the surrogate objective used in Trust Region Policy Optimization (TRPO); see below. Additionally, we introduce a clipped version of this objective by applying a clipping mechanism to the policy ratio r_t(θ). Clipping forces the policy ratio to fall in the range [1 - ε, 1 + ε]. In other words, we avoid the policy ratio becoming too large or too small, ensuring that the token probabilities produced by the current and old policies remain relatively similar.

The PPO surrogate objective is simply the minimum of clipped and unclipped objectives, which makes it a pessimistic (lower bound) estimate for the unclipped objective. The behavior of the clipping mechanism in the surrogate loss changes depending on the sign of the advantage. The possible cases are shown below.

As we can see, taking the minimum of clipped and unclipped terms in the surrogate objective causes clipping to be applied in only one direction. The surrogate objective can be arbitrarily decreased by moving the importance ratio away from one, but clipping prevents the objective from being increased beyond a certain point by limiting the importance ratio. In this way, the clipping in PPO disincentivizes large policy ratios and, in turn, maintains a trust region by preventing large policy updates that could potentially damage our policy.

“We only ignore the change in probability ratio when it would make the objective improve, and we include it when it makes the objective worse.” - from [11]

KL divergence. We often incorporate a KL divergence between the current policy and a reference policy—usually the model from the beginning of training—into RL. The KL divergence serves as a penalty that encourages similarity between the current and reference policies. We compute the KL divergence by comparing token distributions from the two LLMs for each token in a sequence. The easiest—and most common—way to approximate KL divergence [12] is via the difference in log probabilities between the policy and reference; see here.

After the KL divergence has been computed, there are two primary ways that it can be incorporated into the RL training process:

1. By directly subtracting the KL divergence from the reward.

2. By adding the KL divergence to the loss function as a penalty term.

PPO adopts the former option by subtracting the KL divergence directly from the reward signal used in RL training, as shown in the equation below.

Advantage estimation. The advantage function, a key part of PPO’s surrogate objective, is the difference between the action-value and value function: A(s, a) = Q(s, a) - V(s). The value function in PPO is estimated with a learned model called the value model (or critic). This critic is a separate copy of our policy, or—for better parameter efficiency—an added value head that shares weights with the policy. The critic takes a completion as input and predicts expected cumulative reward on a per-token basis using an architecture that is similar to that of a reward model (i.e., transformer with a regression head); see below.

The value function is on-policy—it depends on the current parameters of our policy. Unlike reward models, which are fixed at the beginning of RL training, the critic is trained alongside the LLM to keep its predictions on-policy—this is known as an actor-critic setup. To train the critic, we add an extra mean-squared error (MSE) loss term—between the rewards predicted by the critic and the actual rewards—to the PPO loss. Using the critic, we can estimate advantage using Generalized Advantage Estimation (GAE). The details of GAE are beyond the scope of this post, but a full explanation and implementation can be found here.

Group Relative Policy Optimization (GRPO)

Group Relative Policy Optimization (GRPO) [13] builds upon PPO by proposing a simpler technique for estimating the advantage. In particular, GRPO estimates the advantage by sampling multiple completions—or a “group” of completions—for each prompt and using the rewards of these completions to form a baseline. This group-derived baseline replaces the value function, which allows GRPO to forgo training a critic. Avoiding the critic drastically reduces GRPO’s memory and compute overhead compared to PPO. Additionally, since GRPO is commonly used for reasoning-oriented training, we typically pair it with verifiable rewards, which eliminates the need for a separate reward model.

“We introduce the Group Relative Policy Optimization (GRPO), a variant of Proximal Policy Optimization (PPO). GRPO foregoes the critic model, instead estimating the baseline from group scores, significantly reducing training resources.” - from [13]

Advantage estimation in GRPO is performed by sampling multiple completions for each prompt and using the formulation shown below. This approach is very simple compared to PPO, which uses a learned value model and GAE.

In GRPO, completions to the same prompt form a group, and we calculate the advantage relative to other rewards in the group—hence, the name “group relative” policy optimization! More specifically, the advantage for completion i is calculated by first subtracting the mean reward over the group from r_i, then dividing this difference by the standard deviation of rewards over the group. The GRPO loss is assigned on a per-token basis, but we should note that the above formulation assigns the same advantage to every token t in completion i. The per-token loss is therefore dictated by the policy ratio, which varies for each token.

“GRPO is often run with a far higher number of samples per prompt because the advantage is entirely about the relative value of a completion to its peers from that prompt.” - RLHF book

Because we compute the advantage in a relative manner (i.e., based on rewards in the group), the number of completions we sample per prompt must be high to obtain a stable policy gradient estimate. Unlike GRPO, PPO and REINFORCE typically sample a single completion per prompt. However, sampling multiple completions per prompt has been explored by prior RL optimizers like RLOO.

Surrogate loss. Despite estimating the advantage differently, GRPO uses a loss function that is nearly identical to that of PPO. As shown below, GRPO uses the same clipping mechanism that is used by PPO for the importance ratio. This expression assumes an MDP formulation and has been modified to explicitly aggregate the loss over multiple completions within a group.

KL divergence. One key difference between PPO and GRPO is the KL divergence term being added as a penalty term to the surrogate loss, rather than subtracted from the reward. However, we should note that the KL divergence is frequently omitted when training reasoning models. In the context of RLHF, KL divergence enables model alignment without diverging significantly from the initial model, but this approach makes less sense when training long CoT reasoning models. The model’s behavior may diverge significantly from the initial model as it develops the ability to perform long CoT reasoning. All of the work that we will study in this overview omits the KL divergence term during RL training.

Limitations of vanilla GRPO. For a full overview of GRPO, please see the above link. As we have seen, GRPO is a relatively simple algorithm. The popularity of GRPO was catalyzed by its use for training DeepSeek-R1 [2]. The openness of this work led GRPO to be adopted in open replications of reasoning models, as well as countless other research efforts. Despite its popularity, vanilla GRPO has several issues that become especially pronounced in large-scale RL training runs:

• Noise and instability during the training process.

• Excessive response lengths, especially in incorrect answers.

• Collapse of the LLM’s entropy (i.e., reduced exploration).

• Poor sample efficiency and slow learning.

Due to these issues, many open research efforts initially struggled to replicate the results reported by DeepSeek-R1 [1, 3], indicating that some details necessary to achieve peak performance with GRPO may have been omitted from [2]. This overview will study various works that have diagnosed such issues with GRPO, uncovering a set of practical tricks that can be used to train better reasoning models at scale.

Assessing the Health of RL Training

Despite the recent success of reasoning models, we must remember that training LLMs via RL is a complex process with many moving parts. We are working with multiple disjoint systems to train the model, each of which has unique settings that must be tuned. As described below, even simple changes to the RL training process can yield unexpected results or completely derail the model. When issues occur, it can be hard to know exactly what went wrong, and the high cost of RL training can make debugging these issues slower and more difficult. To quickly identify issues and iterate on our RL training setup, we need intermediate metrics that allow us to efficiently monitor the health of the training process.

Health checks. The key training and policy metrics that can be monitored to catch issues with our RL setup are as follows:

1. Response length should increase during reasoning RL as the policy learns how to effectively leverage its long CoT. Average response length is closely related to training stability, but response length does not always monotonically increase—it may stagnate or even decrease. Excessively long response lengths are also a symptom of a faulty RL setup.

2. Training reward should increase in a stable manner throughout training. A noisy or chaotic reward curve is a clear sign of an issue in our RL setup. However, training rewards do not always accurately reflect the model’s performance on held-out data—RL tends to overfit to the training set.

3. Entropy2 of the policy’s next token prediction distribution serves as a proxy for exploration during RL training. We want entropy to lie in a reasonable range—not too low and not too high. Low entropy means that the next token distribution is too sharp (i.e., all probability is assigned to a single token), which limits exploration. On the other hand, entropy that is too high may indicate the policy is just outputting gibberish. Similarly to entropy, we can also monitor the model’s generation probabilities during RL training.

4. Held-out evaluation should be performed to track our policy’s performance (e.g., average reward or accuracy) as training progresses. Performance should be monitored specifically on held-out validation data to ensure that no reward hacking is taking place. This validation set can be kept (relatively) small to avoid reducing the efficiency of the training process.

An example plot of these key intermediate metrics throughout the RL training process from DAPO [1] is shown below. To iterate upon our RL training setup, we should i) begin with a reasonable setup known to work well, ii) apply interventions to this setup, and iii) monitor these metrics for positive or negative impact. We will see many examples of such a workflow throughout this overview as we study various tweaks and improvements to the vanilla GRPO algorithm.

“We typically use length in conjunction with validation accuracy as indicators to assess whether an experiment is deteriorating… the trend of reward increase [should be] relatively stable and does not fluctuate or decline significantly due to adjustments in experimental settings… we find that maintaining a slow upward trend in entropy is conducive to the improvement of model performance.” - from [1]

A note on batching and data. Prior to making algorithmic changes to GRPO, we should focus on the correctness of our data. GRPO needs (relatively) large batch sizes to work well. Using a small batch size in GRPO is one of the most common mistakes in RL training. To avoid this mistake, we should begin with a reasonable batch and group size (e.g., Olmo 3 [5] uses a batch size of 512 with 64 prompts and 8 rollouts per prompt) and test how varying the batch and group sizes impacts the metrics discussed above. For example, if a larger batch size makes our reward curve much more stable, then our initial batch size was probably too small.

As shown in recent RL research [9, 10], curating the correct set of prompts is also essential. More specifically, we want our data to be diverse in terms of topic and difficulty. For example, Olmo 3 [5] incorporates several domains—math, coding, instruction following, and general chat—into RL training and uses offline difficulty filtering to filter out prompts that are too easy or too difficult. Using another LLM to gauge prompt difficulty by measuring Pass@K performance is also a common filtering approach [9]. We see each data point multiple times during RL training, so data curricula3 are less relevant. To make the most of our data, we simply want to ensure sufficient quality and diversity!

“Where algorithmic changes can make models more robust to less balanced data, a crucial part of the current RL training is to have a diversity of difficulties in your data. With large batch sizes, the model should have questions that are trivial, somewhat challenging, and nearly impossible in each batch.” - Nathan Lambert

As a final note, certain categories of questions—specifically those that are easily guessable without any true reasoning—can damage the fidelity of RL training. For example, multiple choice questions can easily be reward hacked if the policy randomly guesses an answer to each question. Therefore, removing this style of easily-guessable questions from RL training is a common practice.

Improving upon Vanilla GRPO

Now that we understand GRPO, we will learn about recent research that has identified (and solved) problems with the vanilla GRPO algorithm. Given the popularity of GRPO, many papers have been published on this topic. We will aim to review this work in a way that is both comprehensive and of sufficient depth. The section will begin with longer overviews of a few popular papers. After the longer overviews, we will provide a wider outline of the topic via shorter paper summaries and an exhaustive list of recent and notable publications.

DAPO: An Open-Source LLM Reinforcement Learning System at Scale [1]

Despite the impressive recent results achieved with reasoning models, many details needed to reproduce these results are concealed. In fact, even open models like DeepSeek-R1 [2] do not provide sufficient technical details to fully reproduce their results. A naive application of GRPO with Qwen-2.5-32B achieves a score of 30% on AIME 2024, underperforming the score of 47% achieved in the DeepSeek-R1 technical report. This difficulty in reproducing the results of DeepSeek-R1 hints at missing details that are necessary for stable, performant, and scalable RL.

“The broader community has encountered similar challenges in reproducing DeepSeek’s results suggesting that critical training details may have been omitted in the R1 paper that are required to develop an industry-level, large-scale, and reproducible RL system.” - from [1]

In [1], authors aim to discover these missing details, arriving at four key changes to the vanilla GRPO algorithm that—when applied in tandem—match and surpass results observed in [2]. The modified GRPO algorithm derived in [1] is called the Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO) algorithm. All code (based on verl) and data are openly released to support future research.

Vanilla GRPO. When running the vanilla GRPO algorithm, authors notice several issues in the training process, including:

• Entropy collapse: the entropy of the model’s next token distribution collapses during the training process. Probability mass is primarily assigned to a single token and outputs are more deterministic.

• Reward noise: the training reward is very noisy and does not steadily increase during the RL training process.

• Training instability: the training process is unstable and may diverge. We do not observe a steady increase in response length during training.

To mitigate these issues, authors propose the following four solutions in [1].

(1) Clip higher. As mentioned previously, authors in [1] observe entropy collapse when training models with vanilla GRPO; see below. When entropy declines, the next token distribution becomes concentrated on a single token, leading sampled responses in a group to be very similar. As a result, exploration becomes limited and the advantage computation in GRPO becomes less reliable—each sample in the group will tend to receive the same reward, making group normalization difficult.

Interestingly, we see in [1] that this entropy collapse is caused by the clipping operation in PPO and GRPO. To see why this occurs, let us consider two kinds of tokens to which clipping could be applied:

1. Exploitation token: a token that is already highly likely in the current policy.

2. Exploration token: a low probability token in the current policy.

Sampling lower probability tokens gives the model a chance to explore alternative tokens when searching for better completions. Clipping is applied to the policy ratio, or the ratio of a token’s probability after and before the policy update:

The policy ratio is constrained to a range of [1 - ε, 1 + ε]. This upper bound allows high probability (exploitation) tokens to become more probable, but it restricts increases in low probability (exploration) tokens. A concrete example of how the upper clipping bound can discourage exploration is explained below.

The “clip higher” approach, which decouples the lower and upper bound for clipping, is proposed as a solution to this problem. Specifically, we clip in the range [1 - ε_low, 1 + ε_high], where ε_low = 0.2 (default setting) and ε_high = 0.28 in [1]. As shown in the figure above, increasing ε_high prevents entropy collapse and improves GRPO performance. On the other hand, authors note that ε_low should not be increased, as this would suppress some tokens to a probability of zero and collapse the token sampling space.

(2) Dynamic Sampling. Throughout the course of RL training, the number of samples for which all completions in a group are correct naturally increases; see above. Although this trend indicates that the model is improving, prompts with perfect accuracy are problematic for GRPO. If all completions in a group are correct (i.e., reward of one), then the advantage for each completion in the group and the corresponding policy gradient are zero. As a result, our batch size effectively becomes smaller because there are many elements in the batch with zero gradient—leading to a noisier batch gradient and, in turn, degraded sample efficiency. To solve this issue, we can perform dynamic sampling, which simply:

1. Over-samples prompts for each batch.

2. Filters or removes all prompts with perfect accuracy.

The sampling cost per batch is dynamic—hence the name “dynamic sampling”—and we simply continue sampling and filtering until we have a full batch. However, this additional sampling cost is typically offset by the improved sample efficiency of the algorithm. Put differently, the model tends to converge much faster when we filter out prompts with perfect accuracy (i.e., dynamic sampling); see below.