Agentic World Models
Creating better language agents by teaching them to model their environment...
Reinforcement learning (RL) has played a key role in the expansion of capabilities for large language model (LLM) agents. Despite the many benefits of RL, however, this training strategy has a key limitation—the learning signal it uses is sparse. More specifically, most LLM agents are trained using outcome rewards, meaning that an entire agentic trajectory receives only a single reinforcement signal based on whether the trajectory was successful. Although we can partially solve this issue via the incorporation of process rewards, the fundamental problem of supervision sparsity in RL still exists, leading to downstream issues like sample inefficiency.
“Language agents are trained to act in interactive environments, but no language model has been explicitly trained to model the environments themselves — to predict what happens next given the current state and an agent’s action.” - from [4]
Although RL has a sparse reward signal, agent trajectories themselves are information rich. Every interaction contains both an action from the agent and the resulting environment observation, revealing how the environment responds to the agent's actions. Standard agentic RL typically ignores observations during training. However, recent work shows that these observations can instead provide a dense source of supervision by augmenting RL with a world modeling objective that trains the agent to predict observation tokens. In this overview, we will learn that such a hybrid objective improves agent capabilities by developing an internal model of environment dynamics. Put simply, instilling an accurate world model within an agent leads to improved learning efficiency, capabilities, and generalization.
From Agents to World Models
Several background concepts need to be covered in order to fully understand the papers that will be covered in this overview. First, we need to develop a practical understanding of both supervised training and RL, as well as how these forms of training can be combined together. Additionally, we must introduce the concept of an agent and a world model, allowing us to grasp the complementary purposes of these two types of models more deeply. Once these key ideas are clear, we can fuse them together by explaining—alongside a concrete implementation—how world modeling can be incorporated into the standard training process for an agent.
Supervised Finetuning (SFT)
Supervised training is one of the most straightforward approaches that can be used to train an LLM. Given a sequence of text, this approach trains the LLM with a next token prediction objective; see above. More specifically, we apply a cross entropy loss that trains the model to predict the ground-truth next token at every position in a sequence. This objective uses the data itself as the source of supervision—the ground truth next token is naturally present in the textual sequence being used for training. A concrete implementation of the supervised training objective in PyTorch is provided within the code block below.
import torch
import torch.nn as nn
import torch.nn.functional as F
# toy settings
vocab_size = 10
batch_size = 2
seq_len = 5
prompt_len = 2
# fake / random logits to simulate transformer output
logits = torch.randn(batch_size, seq_len, vocab_size, requires_grad=True)
# next token / target ids
targets = torch.randint(low=0, high=vocab_size, size=(batch_size, seq_len))
# (optional) mask out prompt tokens for SFT
loss_mask = targets.clone()
loss_mask[:, :prompt_len] = -100 # -100 is ignored by cross entropy loss
# shift labels to make sure we are predicting the next token
shift_logits = logits[..., :-1, :].contiguous()
shift_targets = loss_mask[..., 1:].contiguous()
# resize so that we can pass into cross entropy loss
flat_logits = shift_logits.view(-1, vocab_size)
flat_targets = shift_targets.view(-1)
# compute the loss
sft_loss = F.cross_entropy(flat_logits, flat_targets, ignore_index=-100)
print(f"SFT Loss: {sft_loss.item():.4f}")We can formulate this objective mathematically as shown below. As we can see, SFT is a negative log-likelihood objective that maximizes the probability assigned by our policy π_θ to the ground truth next token given prior tokens as context.
Using this supervised approach, we can directly train an LLM on raw textual data (e.g., curated expert examples or data from a web crawl). Because LLMs use causal self-attention in their decoder blocks, each prediction of the next token only considers the current token and those that come before it. The model cannot simply “cheat” by looking forward in the sequence to find the next token. Stated intuitively, supervised training teaches the LLM to mimic the data that is used for training—we are teaching the model to simulate the distribution of its training data.
“Causal attention is a masked form of self-attention in which each token is allowed to attend only to itself and to earlier tokens, but not to later ones. In other words, when a GPT-style model is predicting the token at position
t, it can use information from positions1throught, but it cannot look at positions aftert.” - source
The two most prominent forms of supervised training for LLMs are pretraining and supervised finetuning (SFT). These training stages each use the same exact supervised objective, but they serve a different purpose. Pretraining is the first—and most expensive—stage of training during which the LLM is trained from scratch on vast amounts of raw textual data, primarily sourced from the internet. The LLM learns most of its world knowledge in this stage, and we can improve model capabilities significantly by continuing to scale up pretraining.
In contrast, SFT is a post-training stage that is applied after pretraining and uses a lower volume of data. Most of the data used for SFT is either expert curated or generated synthetically with a strong teacher model. Whereas pretraining builds a wide knowledge base via large-scale supervised training, SFT is used to refine model behavior (e.g., improve style, preferences, or instruction following) and create a solid starting point for subsequent stages of RL training.
Reinforcement Learning (RL)
RL goes beyond simulation by training LLMs on their own completions—an on-policy learning approach. As depicted above, the RL training process alternates between two key steps:
Rollouts: given a set of prompts, sample multiple completions for each prompt using the current LLM (and compute the reward for each completion).
Policy updates: compute a weight update for the LLM using the sampled rollouts and the given objective function.
The structure of the policy update depends on the RL optimizer being used; e.g., GRPO, PPO, and REINFORCE are common choices. The papers covered in this post largely rely upon GRPO, so we will briefly cover the details of this algorithm.
Based upon the Proximal Policy Optimization (PPO) algorithm, Group Relative Policy Optimization (GRPO) is a lighter weight and extremely effective RL algorithm that was proposed by DeepSeekMath [5] and received widespread attention for its use in training the DeepSeek-R1 [6] reasoning model. The main change made by GRPO relative to PPO is in the advantage estimator.
Instead of using a value model and GAE to estimate advantages as in PPO, the GRPO advantage estimate samples multiple completions or rollouts per prompt —referred to as the “group” in GRPO—and uses the rewards of each rollout to form a baseline. This group-derived baseline replaces the value function, allowing GRPO to forego training a value model; see above. As a result, there is only one trainable model (i.e., the policy itself), which drastically reduces compute and memory costs of GRPO training relative to actor-critic algorithms like PPO.
Concretely, the advantage for completion i is computed by normalizing the reward for this completion r_i with the mean and standard deviation of rewards in the group; see above. The same advantage value is assigned to every token t in the completion. Intuitively, GRPO looks at differences in rewards among a group—completions with higher rewards relative to the group are assigned higher advantage.
Once the advantage has been computed, the GRPO loss function is similar to that of PPO; see above. For example, GRPO uses the same clipping mechanism and importance ratio from PPO. For a deeper dive on GRPO and its many variants, please see the related overviews linked below:
Deep Dive into the GRPO Algorithm [link]
GRPO Implementation in PyTorch [link]
Overview of GRPO Variants and Practical Tricks [link]
Agentic RL
We can use RL to train LLMs on any task, including multi-step (agentic) tasks. At their core, agent systems are built around an LLM backbone and are not much different from a standard LLM. We simply wrap the LLM in a harness that:
Provides instructions for solving a problem.
Presents all tools that are available.
Parses and executes tool calls.
Returns any tool outputs or observations to the agent.
Manages the visible context for the agent.
Runs the agent in a loop to solve multi-step problems.
Despite the additional complexities that arise when we go from a standard LLM to an LLM agent, we are still prompting an LLM and generating output tokens at the lowest level. Therefore, the manner in which we handle RL training does not change much—we can still compute the same token-level GRPO loss presented before.
Although the mechanics of RL training are similar for agents and LLMs, the main difference lies in how we sample rollouts (see above):
Standard LLMs typically use a single-turn setup that samples a textual rollout given a prompt as input.
Agents use a multi-turn setup in which multiple outputs are generated from the LLM across several environment steps within a single rollout.
Agentic rollouts are more complex than those of a standard LLM. These rollouts operate over a long time horizon, and each step of the rollout can interact with an external environment via tool calls. For this reason, sampling agentic rollouts is a difficult infrastructural problem. For example, certain rollouts will take much longer to complete than others, which can be addressed via asynchronous RL with disaggregated architecture. For more details on agentic RL and frameworks that have been proposed in this domain, please see the overview linked below.
Agentic RL: Frameworks and Best Practices
An in-depth survey that coversLLM agents, how they can be trained with reinforcement learning (RL), and the many framework that have been proposed in this space.
A key practical difference between standard and agentic RL is the standard loss masking approach. Similarly to how we can mask prompt tokens from the SFT objective, we usually mask prompt tokens in RL—the loss is only computed over agent-generated tokens. If we extend this idea to agentic RL, we should also remove environment tokens (i.e., tool outputs or observations) from the loss, ensuring that only LLM-generated tokens are considered in the objective. This action masking procedure is a common implementation choice in agentic RL; see below.
World Modeling for LLM Agents
Although LLMs are incredibly capable, their understanding of the world is arguably incomplete. A pure generative model has the ability to produce valid completions, but these completions do not automatically imply a detailed grasp of the world. For example, an LLM agent can output the text “Your plane ticket is booked!” without calling an API to actually book the ticket and confirming the ticket is booked in an external database. Similarly, a video generation model can create a realistic scene without fully understanding 3D structures or dynamics.
“Language models have given machines an extraordinary command of concepts, vocabulary, and reasoning, but the physical world runs on a different substrate. Where language models learn the statistical structure of text, world models learn the statistical structure of space and time: how light falls on a surface, how a garden looks from an angle no camera has captured, how objects respond to force.” - source
We refer to models that focus upon modeling such details and dynamics of the external environment as world models. However, this term is incredibly broad—there are many different ways in which we can try to model the world. To narrow this definition slightly, we can pull upon the generic Markov Decision Process (MDP) formulation of agents shown below. In this formulation, the environment state is a complete representation of the world. However, an agent usually does not have access to this entire state. Instead, the agent is given a partial view into reality via the observations (e.g., tool outputs or a single video frame) that it receives and learns to take actions within its environment based upon these observations.
As we can see, the definition of a world model is broad. For example, certain models (e.g., MuZero or Dreamer) focus upon simulating the actual state of the environment and how it evolves over time. Alternatively, other world models can be used to render observations for the agent; e.g., generative image and video models like Nano Banana and Seedance can be used to generate images or videos that serve as synthetic observations for an agent. For further details on the types of world models that exist, see the taxonomy proposed by Dr. Fei Fei Li below.
In the agent domain, most of the world models we will see focus upon predicting observations from the environment—or the outputs of tool calls. The outputs of the tools invoked by an agent are a lossy representations of the actual state of the environment. These tool outputs may contain information about container state, code that was executed, changes made to the environment, and much more. We can create a world model by training an LLM to predict these observations given the prior trajectory history as context. As we will see in this overview, such world models tend to follow two main patterns:
A separate world model trained specifically to predict observations.
A unified LLM that is trained to serve as both an agent—by taking actions—and a world model—by predicting the resulting observation from those actions.
In both cases, world models in the agent domain are primarily trained using a supervised objective—these are simulators that are trained to predict observations given the agent’s prior trajectory as context. Training a standalone world model is straightforward: we can gather datasets related to world modeling and train an LLM-based world model on this data similarly to any other LLM. To train a unified world model and agent, most papers use a hybrid objective that combines supervised learning with RL. Put simply, we train the agent to act with RL and to predict observations with supervised learning. Concretely, such an approach can be implemented by using a mask to distinguish tokens within a trajectory and applying a separate objective function to actions and observation tokens.
“Pretraining creates simulators, LLMs that can model their environment with high precision. RL only improves the model’s own generations. But true agents should do both at the same time: we don’t want a pure simulator, not do we want the LLM to only act, blind to how it will affect its environment. Instead, we want it to learn to predict its environment in response to its own actions, allowing it to more intelligently plan and respond to unexpected world dynamics.” - from [2]
To understand how such a hybrid objective can be implemented, we can first consider the token-level, advantage-weighted loss for RL formulated below. We will see this same loss formulation again when we cover the ECHO paper [1].
When looking at the loss above, we might notice that—aside from the advantage term—it is nearly identical to a supervised objective1. More specifically, if we assign a constant, positive value to the advantage, the loss function shown above becomes equivalent to a supervised next-token prediction objective up to a constant scaling factor. We can then create a hybrid RL and supervised loss by:
Computing the advantage normally for LLM-generated action tokens.
Assigning a constant positive advantage for observation tokens.
By doing this, we fuse RL and supervised objectives, training the agent to select actions via RL and to simulate its environment with a supervised objective.
“Environment observations are not merely context for future actions, but a dense, on-policy supervision signal already present in every rollout.” - from [1]
The constant positive advantage we select also becomes a weight that balances the importance of the two losses. A reference implementation is provided below.
import torch
import torch.nn.functional as F
# constant weight applied to observation tokens
supervised_weight = 0.05
# standard GRPO setup: sample G completions for B prompts + compute rewards
with torch.no_grad():
completions = LLM.generate(prompts) # (B*G, L)
rewards = compute_rewards(completions) # (B*G)
# create a padding mask from lengths of completions in batch
completion_mask = <... mask out padding tokens ...>
# create masks to identify action and observation tokens
action_mask = <... mask identifying action tokens ...>
observation_mask = <... mask identifying observation tokens ...>
# get per-token logprobs
llm_out = LLM(completions)
per_token_logps = F.log_softmax(llm_out, dim=-1) # (B*G, T, V)
# gather logprobs of sampled tokens
per_token_logps = per_token_logps.gather(
dim=-1,
index=completions.unsqueeze(-1),
).squeeze(-1) # (B*G, L)
# compute mean and std of grouped rewards
reward_mean = rewards.view(-1, G).mean(dim=1) # (B,)
reward_std = rewards.view(-1, G).std(dim=1) # (B,)
# compute advantage for GRPO
advantage = (rewards.view(-1, G) - reward_mean)
advantage /= (reward_std + 1e-8) # (B, G)
advantage = advantage.view(-1, 1) # (B*G, 1)
# create per-token weights
token_weight = (
action_mask * advantage +
observation_mask * supervised_weight
) # (B*G, L)
# hybrid supervised + RL loss
loss = -token_weight * per_token_logps # (B*G, L)
# aggregate the loss across tokens
loss = ((loss * completion_mask).sum(axis=-1) /
completion_mask.sum(axis=-1)).mean()
# perform update
optimizer.zero_grad()
loss.backward()
optimizer.step()Teaching Agents to be World Models
“World modeling aims to capture action-conditioned dynamics by predicting the environment response after an action.” - from [3]
We have now learned the basics of world models and how we can perform world modeling for LLM agents via a combination of supervised training and RL. In the next section, we will cover several papers that study the intersection of world modeling and LLM agents. Although the majority of these papers are based upon the same high-level idea, many variations of the hybrid agent and world modeling objective we saw before are proposed. The exact manner in which this objective is implemented has a noticeable impact on agent performance, but incorporating world modeling into agent training tends to yield capability improvements.
ECHO: Terminal Agents Learn World Models for Free [1]
A terminal agent is an LLM agent that interacts with a terminal environment. The agent outputs tool calls—in the form of terminal commands—that are executed in the terminal environment, and the terminal’s output (e.g., stdout, stderr, files, logs, etc.) is returned as an observation. When training a terminal agent with RL, the learning signal is sparse. We typically use outcome rewards, and only a small ratio of rollouts may actually solve a task while others yield little gradient signal. However, authors in [1] argue that even failed rollouts contain implicit, untapped learning signal. Each rollout contains observations from the terminal environment—such as failed tool calls or useful actions in an unsuccessful trajectory—that are informative and can be used to better understand the agent’s environment.
In a typical agentic RL setup, an action mask removes environment tokens from the training objective. However, these tokens still provide useful supervision for terminal agents. Environment Cross-Entropy Hybrid Objective (ECHO) [1] argues that agents should be trained on environment tokens, rather than treating them only as external context. The intuition is simple: an agent that can accurately predict the output of a terminal command should understand the dynamics of a terminal environment. Each terminal observation—whether it contains file contents, test failures, or other feedback—provides a partial and noisy view of the underlying environment state2. By learning to predict these observations, the terminal agent can develop a better model of how its actions impact the environment, as well as an improved ability to reason about the consequences of its actions.
“Taken together, these findings suggest agent training has been operating with a supervision source masked out: every policy action has a consequence in the environment, and that consequence is in the rollout already. ECHO shows that these consequences can be trained on directly, turning even failed interactions into signal for learning how the world responds.” - from [1]
With standard GRPO, failed trajectories receive little or no reward and contribute minimally to the learning signal. As a result, outcome-based rewards can produce a sparse learning signal, especially when a group contains primarily unsuccessful trajectories. By turning environment observations into a supervised learning signal, ECHO allows even unsuccessful rollouts to provide dense supervision, enabling the policy to continue improving even in the presence of sparse rewards.
Environment Cross-entropy Hybrid Objective (ECHO). As the terminal agent learns and visits new terminal states, it begins to create its own self-evolving curriculum during RL training. ECHO turns this evolving stream of observation tokens into a dense source of supervision. By doing this, the agent is explicitly trained to model the results of its actions and, in turn, form better priors over the commands that are likely to be beneficial when solving a task. Concretely, this additional supervision is integrated into the RL objective as shown below.
The ECHO objective is a hybrid objective that takes a weighted sum of the standard GRPO objective and a length-normalized cross entropy (or next token prediction) objective. This auxiliary cross entropy objective is referred to as the environment loss in [1]. The GRPO loss is applied to LLM-generated action tokens, while the environment loss is applied to a select set of terminal output tokens3. The environment loss is normalized by Z—the total number of terminal output tokens4. Notably, all environment tokens, not just those that are selected in the loss, are included in the normalization term, ensuring that the environment loss has comparable scale no matter how terminal output tokens are selected.
One of the most notable characteristics of ECHO is that the objective can be computed with no additional rollouts or forward passes. As shown above, the objective uses the same token-level log probabilities that are already computed during RL in order to derive the policy update. The main difference is that the environment loss used in ECHO gathers these log probabilities at a different set of positions—those corresponding to terminal outputs rather than agent actions—relative to the core objective for agentic RL. Practically, this is implemented via separate binary masks that index action and observation tokens in a sequence.
“The expensive attention and MLP computations already run over the full rollout to compute action-token log-probabilities for GRPO. ECHO simply gathers the already-computed logits at terminal-output positions and includes their cross-entropy in the same backward pass.” - from [1]
Given that all log probabilities used by the environment loss are already present during RL, the only additional work done by ECHO is to compute a masked sum of log probabilities over the sequence. This operation is inexpensive and adds negligible overhead to the RL training process. Notably, the ECHO objective is still strictly on-policy—we are computing the loss over rollouts from the current policy. Additionally, ECHO is complementary to many existing methods for agentic RL, as it modifies the training objective but not the underlying RL algorithm.
Mixing losses. The ECHO objective introduces an additional mixing weight λ between the two loss components. A hyperparameter sweep is performed in [1], which determines that the optimal value for λ lies in the range [0.01, 0.05]. Below this range, the environment loss is dominated by the GRPO objective. If the value of λ is too high, however, the environment objective competes with the policy loss, causing the learning process to plateau. For example, authors in [1] note that large values of λ can cause training to degenerate towards low-quality rollouts that receive little reward but contain tool calls with easily-predictable outputs.
Experimental setup. ECHO is tested using 2,700 curated terminal task samples from a variety of sources (e.g., Endless Terminals and OpenThoughts-Agent) that are filtered by domain. These tasks are then used to generate 6,170 new tasks with a modified version of the pipeline used in Endless Terminals, and any task that is solved by GPT-5 in at least one of 16 attempts is kept. The result of this process is 8,870 total tasks—100 of which are held out for testing—spanning data processing, system operations, development, and tooling domains. Agents are evaluated on three in-domain datasets and a more difficult Terminal-Bench 2.0 benchmark.
“At each turn, the policy conditions on its prior reasoning, commands, and command outputs, then emits a thinking block followed by Qwen XML-format bash commands or a task-done signal. A minimal training harness parses the first command or completion signal, executes it, and returns optional format warnings plus stdout/stderr and exit code as the next observation.” - from [1]
Each task uses a Docker task environment format that is modeled after Terminal-Bench. Agents use the Harbor harness and run for up to 16 environment turns, where at each turn the agent outputs a thinking trace and a Qwen XML format tool call (i.e., a bash command). After the episode, a set of verification tests run to check whether the task was solved. Timeouts of ten and two minutes are used for the agent and verification steps, respectively. Three different LLM backbones are used for testing ECHO:
All experiments use the same GRPO recipe5 for RL, but the OpenThinker-Agent starting policy—which is based upon Qwen3-8B—has been initially trained via SFT over ~15K expert trajectories in diverse terminal environments. By including this SFT-trained starting policy, we can test differences in performance between learning from pure RL and seeding the RL training process with expert data.
High-level results. ECHO improves the performance of every starting policy on every benchmark. Relative to GRPO, ECHO doubles the pass rates of Qwen3 models on Terminal-Bench 2.0; see above. To better understand the impact of ECHO on agent performance, we can take a deeper look at the learning process for each model. As shown below, 8B models trained with ECHO learn faster, reaching GRPO’s peak performance with 1.5-2.3× fewer training steps.
“At 8B, ECHO reaches the GRPO-only peak in 1.5–2.3× fewer training steps. At 14B, both runs peak at the same step, but ECHO reaches a higher plateau.” - from [1]
Larger models see a similar speed benefit on in-domain evaluations. On the more difficult Terminal-Bench evaluation, the learning process is not faster, but the agent reaches a noticeably higher plateau in performance relative to GRPO. We also see that agents trained with ECHO seem to use their inference budget more efficiently. On Terminal-Bench 2.0, ECHO reduces timeout frequency from 19.8% to 9% and the number of completion tokens by up to 30%—these agents tend to use both fewer tokens and fewer environment steps, leading to fewer timeouts overall.
Priming the terminal agent for RL with SFT-style training on high-quality expert demonstrations is a common approach. When comparing pure RL and RL with an SFT-trained starting policy, however, we see that ECHO reduces dependence on expert-generated data; see above. For example, the Qwen3-8B model matches the performance of the SFT-trained starting policy on in-domain evaluation sets after GRPO training, indicating ECHO can recover the benefits of SFT initialization on these in-domain evaluations. On Terminal-Bench 2.0, ECHO recovers roughly half of this benefit. Therefore, ECHO reduces—but does not fully eliminate, especially on tasks that are difficult and realistic—dependence on expert data.
“Much of what expert SFT provides is an interaction prior: how terminal agents inspect files, run tests, encounter errors, follow tracebacks, and expose useful state. ECHO does not imitate the expert’s action choices; instead, it learns from the terminal consequences of the base model’s own actions… ECHO does not make expert demonstrations obsolete. Rather, it suggests that one component of their value—familiarity with terminal feedback and state evolution—can be learned directly from interaction.” - from [1]
World modeling. Not surprisingly, agents trained with ECHO are also better at predicting terminal feedback. These abilities are also shown to generalize beyond a model’s own trajectories in [1]—terminal prediction accuracy is tested on rollouts generated from a completely different agent. On held-out trajectories, agents trained with ECHO demonstrate significantly lower token-level cross entropy on terminal feedback tokens; see below. In contrast, agents trained with GRPO have virtually no difference in prediction accuracy relative to the starting policy.
Verifier-free adaptation. As a final experiment, authors in [1] test whether ECHO can be used to improve the policy via the environment loss alone (i.e., without any outcome verification for the main policy update). In certain settings, verifier-free improvement is possible and the policy can learn from environment interaction alone. On in-distribution tasks, the policy can continue acting in the environment and learning purely from environment feedback for its own actions; see below.
However, this verifier-free approach requires clean environment interaction examples from which to learn. If we test the same approach on hard out-of-distribution tasks, the policy tends to produce noisy environment interactions (e.g., failed tool calls or unproductive loops). As a result, the environment loss focuses purely on modeling failure cases, leading the learning process to stall and policy performance to plateau. Interestingly, we see in [1] that this issue can be avoided by simply filtering out any trajectories with malformed tool calls.
True Agents Model the World [2]
Pretraining and RL have completely different learning mechanics: pretraining teaches the LLM to simulate or predict observed data, while RL allows the model to learn from its own generations via positive and negative reinforcement. An agent should be able to learn in both ways. Most agents are trained with RL, but developing the ability to simulate its environment allows the agent to understand the impact of its actions and, in turn, enables better decision making. As we have seen with ECHO [1], combining world modeling and RL allows an agent to simultaneously learn how to act and simulate its environment. Unlike pretraining or supervised training on synthetic completions, world modeling during RL trains the model on observations induced by its own actions—an inherently on-policy approach6.
In [2], authors make no new algorithmic proposals—the methodology used is identical to ECHO—but perform a comprehensive empirical analysis to better understand optimal settings for world modeling and its impact on agent behavior.
Augmenting RL. As mentioned before, the algorithm used in [2] is the same as ECHO [1]: we just add an SFT objective on tool response tokens to the existing RL objective. As shown above, this is accomplished via a hybrid objective that takes a weighted combination of an RL objective applied to LLM-generated action tokens and an SFT objective applied to observation tokens. Concretely, we can implement this at minimal additional cost to the RL training process by restating SFT as an RL objective with constant positive advantage; see below.
The constant positive advantage used in the world modeling (SFT) loss in [2] is denoted by α. Based on analysis from ECHO, authors hypothesize that world modeling should make learning during RL more efficient. However, the addition of the world modeling loss will likely help most on domains that are:
Less represented in the model’s pretraining process.
Ripe with knowledge that is useful for the agent to internalize.
Predictable without memorization (i.e., contain non-trivial tool responses that facilitate generalization to new domains).
One final implementation note mentioned in [2] is the fact that the SFT and RL losses must be normalized separately in ECHO. By doing this, we ensure that the loss containing the larger number of tokens does not dominate the objective.
Environments and experimental setup. The analysis performed in [2] considers two environments:
forth-lang: an environment for the under-resourced forth programming language that allows the model to run and submit code, as well as read documentation for the language.deepdive: a search-based QA environment used for training deep research agents that allows the model to search the web, scan pages, and read pages.
forth-lang is an unfamiliar domain for existing models, but it has predictable and complex dynamics governed by the details of the programming language. On the other hand, deepdive is retrieval-oriented. Therefore, the dynamics of this environment are simple and predictable, but tool outputs tend to generalize poorly between tasks—each query may ask about completely different information. Agents are evaluated on an in-domain test set for each environment, as well as additional wordle and general-agent domains that test logical, multi-turn reasoning, as well as tool-heavy problems that require both logic and retrieval.
For all tasks, the agent is not given access to the test cases that determine the reward in order to prevent reward hacking. Instead, rollouts are always run in dedicated sandboxes and test cases are uploaded after the rollout is finished.
Selecting tools for training. When applying ECHO in practice, we do not always need to train agents on the outputs of all tools. For example, experiments with forth-lang test two different world modeling strategies:
ECHO (all) trains on all tool outputs.
ECHO (code) only trains on code-based tool outputs (i.e., tools for running and submitting code, but not tools for looking up documentation).
Interestingly, training only on code-based tool outputs—the ECHO (code) approach—yields better results in the forth-lang environment. Intuitively, adopting ECHO (code) means that we train on the outputs of code execution but not on the documentation retrieved by the agent, which is more prone to memorization. However, removing tools from the world modeling objective is not always necessary or beneficial—experiments with deepdive train on all tool outputs.
Learning efficiency and overfitting. When ECHO (code) is run on the forth-lang environment, the training reward is similar between ECHO (code) and pure RL models, but ECHO (code) clearly improves performance and learning efficiency compared to pure RL; see below. Models trained with ECHO (code) also have lower error rates when submitting code, use fewer turns, and are truncated less frequently due to outputs that are too long. These observations match findings from [1] showing that ECHO agents tend to solve problems with fewer turns.
Despite these benefits, the training process for ECHO (code) collapses after ~500 steps. This crash, though possibly random, is most likely caused by overfitting. While overfitting is less of a risk for pure RL, the dense loss used by ECHO increases the likelihood of overfitting—an issue of which we must be aware when using this technique. The collapse only seems to occur in the forth-lang test set, while other domains are not impacted. Interestingly, this pattern of crashes in one domain not impacting others is also observed when training on deepdive.
As shown above, ECHO—with all tools included in the world modeling objective—continues to yield positive benefits in the deepdive environment, revealing that the benefits of ECHO generalize to diverse environments (i.e., both retrieval and logic-based tasks). However, signs of overfitting emerge even more quickly—after only a single training epoch—in this domain. Overfitting likely occurs more quickly in the deepdive domain due to the heavy use of web retrieval. Tool outputs in this domain are non-trivial, but web page contents are prone to being memorized.
“Overfitting in forth-lang came late or never, but in deepdive it comes after one epoch… tools whose outputs can be memorized are useful in single epoch training if their outputs are non-trivial, but will lead to heavy overfitting in a multi-epoch setting, as is typical in RL. Tools for which generalization is a more efficient strategy can be trained on for many epochs… Overfitting is a significant concern with ECHO, but for complex but predictable tools, it’s a far lesser issue.” - from [2]
Behavioral impact. ECHO (code) has a noticeable impact on agent behavior. Namely, agents trained with ECHO (code) use more tokens than those trained with pure RL, but a smaller ratio of these tokens are model-generated; see above. Agents trained with ECHO rely more heavily on tool use, and those tools often produce longer outputs. For these reasons, trajectories become longer and are dominated by tool-response tokens. As shown below, this increase in tool use is found to be driven primarily by more frequent documentation lookups.
For forth-lang, predicting tool outputs allows the agent to internalize details of the programming language. As these details become engrained in the agent, certain behaviors—such as looking up code documentation—become less necessary. For example, we see above that the frequency with which the agent searches documentation initially increases but eventually decreases later in training.
Similar patterns are observed in the deepdive environment. Agents trained with ECHO on deepdive in [2] are shown to have drastically improved perplexity metrics on data sampled from wikipedia; see above. In contrast, agents trained with pure RL see virtually no change in perplexity metrics. Such a finding shows that ECHO introduces a pretraining-like mechanism that continues to improve the agent's ability to simulate its environment naturally as part of RL training.
World modeling complements RL. Although predicting environment responses improves an agent’s simulation capabilities—as evidenced by reductions in perplexity on held-out observations—these gains alone do not translate to optimal agent performance. Rather, the best results are consistently achieved with a hybrid objective that includes both world modeling and RL, allowing the agent to simultaneously learn an accurate model of its environment while optimizing its actions through reward-driven feedback. Put simply, world modeling is a valuable auxiliary learning signal that makes RL more effective, not a replacement for RL.
Does ECHO generalize? Across the experiments performed with ECHO in [2], the amount of domain interference that occurs is variable. ECHO consistently benefits performance in the training domain, and the strongest results occur in domains that were previously poorly understood by the model. In some cases, gains in the training domain generalize to other domains, but this is not always the case. For example, we saw before that ECHO training on forth-lang leads to better performance in the deepdive and wordle environments but worse performance in the general-agent domain. To understand these trends in performance, we can analyze the behavioral patterns learned by the agent.
The key behavior learned when training on forth-lang is the heavy utilization of documentation lookup—agents trained with ECHO learn to search documentation early during training. This documentation retrieval behavior generalizes well to domains like deepdive due to an emphasis on web search and retrieval, leading to improved performance. On the other hand, the general-agent domain has a complex tool set and generally requires multiple tools to be used—not just a single retrieval tool—to properly solve a problem. For this reason, behaviors learned via ECHO generalize poorly to this domain, leading to a decrease in performance.
Policy and World Modeling Co-Training for Language Agents [3]
“Standard RL optimizes actions for reward maximization without learning their consequences, leaving agents brittle to invalid operations, irreversible state changes, and delayed failures in long-horizon tasks” - from [3]
Although ECHO shows that including an environment loss is helpful for terminal agents, authors in [3] study the impact of this approach in a more general agent context. When working with these agents, we would like to jointly learn world modeling capabilities and policy improvements as part of the same RL training pipeline (i.e., without requiring extra compute, models, or simulators). The core objective for RL training aims to maximize rewards, but maximizing rewards does not automatically imply that an agent understands how its actions impact the external environment. To build this understanding, we need to go beyond pure RL and directly learn from the actions and observations of an agent.
Rollout structure. To begin, we need to first define a formal structure for agentic rollouts—these definitions will simplify later explanations of the methodology proposed in [3]. Given a policy π_θ, we execute several environment interaction steps to derive the final agent trajectory τ. At each step, the agent receives an observation o_t and, from the task history, forms a context h_t that is used to generate the next action a_t ~ π_θ(·|h_t). This action (e.g., a tool call) is executed by the environment, yielding a reward r_t and the next observation o_t+1. The final trajectory contains this information over several turns t: τ = {o0, a0, r0, o1, …, o_T−1, a_T−1, r_T−1, o_T}. Our goal in RL is to maximize the return (i.e., the cumulative reward) over the trajectory. This setup is depicted below.
Next-observation prediction can allow the agent to learn a world model. At step t, we can aim to predict the next observation o_t+1 from the environment after taking action a_t, formulated as π_θ(o_t+1|h_t, a_t). Given that the ground truth o_t+1 is already present in the sampled rollout, we can directly train the model to predict this observation using a cross entropy loss. This prediction is conditioned on both the task history and the agent’s generated action—the agent is explicitly taught to model action-consequence relationships in its environment.
Policy and World (PaW) Model Co-Training. The training strategy proposed in [3] is similar to that of ECHO [1]; see above. Rich world modeling signals already exist in each rollout (i.e., the observations at each step). We can learn from these signals by going beyond the core policy update and teaching the agent to directly predict these observation tokens. By learning to predict observations, an agent can internalize environmental dynamics and better understand the consequences of its actions. Concretely, we train the agent to predict the next observation using a cross entropy loss, referred to as the world modeling (WM) loss in [3]. Similarly to ECHO, this approach does not require any additional rollouts to be generated—it uses log probabilities that are already available from computing the policy update.
“Each interaction step yields both policy supervision from the action and its advantage, and dynamics supervision from the resulting next observation, which reveals what the action causes. While standard RL uses only the former, we exploits the latter as dense action-conditioned next-observation supervision, without requiring additional rollouts.” - from [3]
Although outcome-based reward supervision is sparse, every interaction step naturally produces a next observation. Therefore, the WM loss is a dense source of supervision that can provide a useful learning signal even when the trajectory itself receives no reward. The PaW objective proposed in [3] is a hybrid, weighted objective that combines the core policy objective and the WM loss; see below.
This objective has two complementary learning signals in each interaction step: policy supervision from the chosen action and its advantage, and world-model supervision from the resulting environment observation. The policy objective focuses on refining an agent’s ability to act, while the WM loss trains the agent to predict environment feedback. Everything needed to compute this objective is available in a standard RL training pipeline. The only additional computation necessary is to compute the WM objective over selected observation tokens, which adds minimal overhead; see below. During inference, PaW agents behave identically to an agent trained with pure RL (i.e., no added inference steps are necessary).
Improving stability. The PaW objective does not work out-of-the-box—we need a few tricks to make the training process more stable and performant. If we consider a GRPO setup, each RL update takes a batch of tasks and samples multiple rollouts—or a GRPO group—for each task. Each of these rollouts has a full trajectory with step-level transitions for each action, and resulting observation, from the agent. When considered together, all of these steps form a “pool” of transitions across the entire GRPO batch, where each item in the pool has:
The current step’s context
h_t.The agent’s chosen action
a_t.The corresponding next observation
o_t+1.The reward
r_tfor the action.
We can also derive an advantage estimate from these step-level rewards. Notably, this item-level information is sufficient to compute the PaW objective. We can compute the policy component using context, actions, and advantages, while the WM loss depends upon the context, action, and next observation. Therefore, we can view each iteration of RL training as computing our objective over a pool of step-level transitions from all of the rollouts sampled in the current batch.
With this framing in mind, authors in [3] proposed three key interventions to improve the efficacy of PaW training:
Smart data selection for the WM loss.
Using a clipped mean absolute error (MAE) for the WM loss.
Dynamically adjusting
λ_WMthroughout training.
When training our agent to be a world model, not all observations are actually useful—certain transitions may have little information or be redundant and prone to memorization. To identify high-quality transition candidates in each training batch, we can look at entropy of the agent’s actions. A high-entropy action indicates that the policy is not deterministically selecting that action, but rather considering a diverse range of possible actions. For this reason, authors in [3] argue that high-entropy actions are more informative—and useful for the WM loss—because they correspond to decisions where the policy is most uncertain.
We can explicitly select these actions for training in PaW by computing action entropy for each item in the batch and selecting those with the highest entropy. Concretely, a ratio α ∈ (0, 1] is defined, where α = 0.75 in [3]. In each training iteration, the policy objective is computed over all actions, while the WM loss is computed only on the top-α ratio of transitions based on action entropy. Notably, the policy update is still computed over all actions in the batch, but the WM loss is only computed over the transitions that are selected based on action entropy.
PaW could directly use a cross entropy loss for world modeling, but this objective is not the best choice for fitting environment observations. As shown above, there is a lot of noise and stochasticity in certain observations—these tokens contain a lot of random information that is not helpful for learning environment dynamics. Most of these noisy tokens will be assigned low probability by the LLM because they are hard to predict. When we examine the gradient of the cross entropy loss, however, we see that the contribution of low probability tokens is amplified; see below.
To solve this issue, we can adopt an MAE-style loss, which produces bounded gradients that prevent rare observation tokens from dominating the gradient.
When using standard MAE, we continue optimizing token probabilities until they become arbitrarily close to a perfect probability of one—token probabilities are still optimized further even when they are already relatively high. However, such a property encourages the memorization of environment details, but memorization can be avoided via a confidence mask. Namely, we can mask the loss at the token level such that any token with probability greater than some confidence threshold ρ is considered sufficiently learned and dropped. This objective is referred to as the clipped MAE loss in [3]. Authors set ρ = 0.2 and tokens with probability beyond this threshold are considered sufficiently learned and no longer optimized.
“The need for auxiliary dynamics learning is task-dependent: low-performing rollout groups can benefit more from next-observation supervision, while high-performing groups should focus more on refining the policy objective.” - from [3]
In addition to smart data selection and the clipped MAE loss, PaW proposes a dynamic approach for selecting the value of λ_WM. Given that the world modeling loss is a dense supervision signal, the objective can be easily dominated by this term. We must set the value of λ_WM such that the core policy update is balanced properly with the world modeling loss. Authors in [3] argue that the optimal value for this weight depends upon the properties of the current batch—certain prompts benefit more from world modeling than others. Instead of selecting a single value for λ_WM, we can set this weight separately for each GRPO rollout group using the formula below. This dynamic weight creates two learning modes in PaW:
If the rollout group achieves near the maximum possible return, we focus mostly upon the core policy update, allowing the agent to learn from the returns achieved by the current rollout group.
If the rollout group achieves low average return, we upweight the world modeling loss, allowing the agent to learn from environment feedback.
“[PaW] follows the base RL algorithm for rollout collection and advantage computation, then co-trains the same model with globally selected observation supervision and group-specific auxiliary weighting… PaW requires no additional environment interaction, no separate model, and no separate training stage… the action entropy, action-token loss, and observation-token loss are computed from distributions already available during rollout generation or the masked training pass.” - from [3]
Putting it all together. As described above, the mechanics of PaW are very similar to standard RL training, requiring no additional rollouts, forward passes, or training stages. Similar to ECHO, PaW does not train a separate world model that is used to simulate environment observations. Instead, world modeling is used as a auxiliary source of supervision for the policy itself, thus encouraging the policy to internalize key environment dynamics. The model behaves identically to a standard agent at inference time—the only change made by PaW is incorporating the modified world modeling loss into the training process. The key components of the hybrid world modeling objective proposed by PaW are summarized below.
Experimental results. PaW is empirically validated on a mixture of interactive decision-making (i.e., ALFWorld and WebShop) and search-augmented QA (i.e., HotpotQA, TriviaQA, and more) tasks. Qwen-2.5 models of various sizes are used for RL training, and PaW is evaluated by comparing its performance to the same RL algorithm without PaW included. As shown below, PaW is found to enhance the performance of both GRPO and GIGPO across all models and domains that are considered in [3]. These results are extended to a wider set of RL optimizers (e.g., PPO and RLOO) and model families (e.g., larger Qwen-2.5 and Llama models) in [3], finding that the positive benefits of PaW generalize quite well.
PaW tends to be most helpful in sparse reward settings, where very few rollouts receive positive rewards due to task difficulty or base model quality. For example, a training trajectory with and without PaW on WebShop is shown below. In this challenging setting, vanilla GRPO rarely obtains positive rewards, causing the training process to stall. However, the dense WM supervision of PaW allows the agent to learn useful environment dynamics even when most trajectories yield zero reward. After several training steps, the PaW agent begins to generate some successful rollouts, providing meaningful signal for RL training to progress.
Such a result indicates that PaW can mitigate failures due to sparse rewards by providing a meaningful learning signal even when the ratio of successful rollouts is low. This additional source of supervision enables the agent to avoid long-term plateaus in the training process by continuing to learn and explore, eventually discovering a policy that can generate successful rollouts. Additionally, we see above that all components of PaW are necessary for optimal performance.
Qwen-AgentWorld: Language World Models for General Agents [4]
Qwen-AgentWorld [4] takes a slightly different approach from the work we have seen so far by training an explicit simulation model for agent environments. This simulation model is a text-based LLM—based upon Qwen3.5—that is trained to simulate dynamics in several agent interaction domains, including both:
Text-based environments (e.g., MCP servers, web search, terminals, and SWE).
GUI-based environments (e.g., web, android, and desktop OS).
GUI-based environments are represented with renderable code so that all world modeling is purely text-based. Unlike ECHO and PaW, Qwen-AgentWorld is trained as a separate, dedicated world model—we explicitly prompt and train this model to simulate the set of environments outlined above. As shown below, a unified environment trajectory schema is adopted that can be used to simulate a diverse range of environments by presenting a system message and several interaction turns to the world model. The system message is composed of:
A task description that tells the LLM to act as a world model and specifies the domain and simulation objective being considered.
An action space that describes the available tools and how to invoke them.
A description of the initial state (e.g., installed packages, file system details, UI screen state, etc.) before any interaction turns have been performed.
An (optional) set of few-shot examples or demonstrations that provide actions and their corresponding observations.
A set of simulation instructions that can be used to control granular details and format of the simulation results returned from the world model.
The LLM is then given a set of interaction turns—each containing an agent action and a resulting environment observation—ending with a final action, prompting the world model to predict the next observation that would result from this action.
The resulting Qwen-AgentWorld model is purely a simulation model. In contrast, ECHO and PaW focus on training general language agents and introduce world modeling for the sole purpose of improving the capabilities of the language agent—agents are not directly used to simulate the environment during training or inference.
Training Qwen-AgentWorld. The work in [4] aims to create a general, language-based world model that can be accurately used to simulate diverse environments. To accomplish this goal, this world model must be trained over a wide variety of agent trajectories from diverse environments, be capable of generalizing to new domains not considered in training, and possess sufficient world knowledge to accurately predict dynamics within complex, real-world environments. These requirements are addressed via a three-part training framework; see below.
Most of the training data used in [4] consists of agent-environment trajectories that are derived from a variety of sources:
Dedicated agent-environment backends created specifically for collecting relevant data across the domains considered in [4].
Public datasets containing relevant environment trajectories.
In-house datasets that contain environment trajectories collected during prior model development efforts.
The resulting trajectories are reformatted using the standard structure described above. Before training, data is filtered at a trajectory level to remove sequences with fewer than two turns, catastrophic environment failures, or calls to tools that are not within the defined action space. Data is also filtered at the turn level to remove empty turns, repetitive error cycles, and turns in GUI domains for which the action yields no change to the environment state—usually due to a system error.
The Continual Pretraining (CPT) phase focuses upon learning general world knowledge and environment dynamics. The data used during this phase is drawn from two sources:
Trajectories from dedicated agent environments and public datasets.
A world knowledge corpus that spans a wide range of topics (e.g., law, medicine, finance, manufacturing, cybersecurity, and more).
The world knowledge incorporates necessary information into the training process that cannot be obtained via environment interactions. For example, the world model must possess broad and up-to-date factual knowledge to predict web searches related to recent events. This specialized knowledge is purposely kept diverse to facilitate generalization to new environments. During CPT, the world model uses a non-thinking format where observation tokens are predicted directly—without first outputting a thinking trace—from the interaction history. A token-level cross-entropy loss is used, similarly to standard LLM pretraining.
As we learned in [3], many turns within a trajectory are uninformative. In [4], each turn is categorized using the four metrics outlined above that are computed for each action-observation pair. These metrics help to determine whether a turn is sufficiently informative to contribute useful supervision during training; e.g., an observation that simply echoes information specified in an action would score poorly on these metrics. Using these metrics, we can identify turns that should be excluded from the loss computation. Turns are assigned to the seven categories shown below, each of which has a corresponding keep ratio that is used during CPT to determine how many tokens from the observation are included in the loss.
The Supervised Finetuning (SFT) phase follows CPT but introduces thinking into the observation-prediction process. This phase still uses a standard token-level cross-entropy training objective. In an attempt to improve generalization to other prompt structures, diverse prompt templates are introduced into SFT by uniformly sampling the model’s system prompt from a set of ten template choices—trajectory information is kept the same but injected into a different prompt template.
The SFT phase sources data from the internal trajectory set, but these trajectories do not yet contain thinking traces. When predicting each observation, the world model is expected to output a thinking trace prior to its final output. To augment each trajectory with such a reasoning trace, the trajectory is converted into a query that prompts a general reasoning model to predict the next observation from prior context. Then, a rejection sampling approach is applied that:
Generates multiple rollouts from the general-purpose reasoning model.
Independently scores all rollout candidates with an LLM judge, where the real observation from the ground truth trajectory is used as a reference.
Compares rollouts in a pairwise fashion to find the best rollout.
Ensures the best rollout scores above a minimum threshold, otherwise the trajectory is discarded.
Starting from an initial set of 10,250 candidate trajectories, the rejection sampling pipeline generates a final set of 7,094 trajectories that are all augmented with thinking traces. CPT and SFT serve complementary purposes: CPT acquires world knowledge, while SFT teaches the model to use this knowledge during simulation. We need rejection sampling here because the real environment provides the final observation but no reasoning trace. By using rejection sampling, we can use a reasoning model to augment the observation with a full reasoning trace and filter any completions leading to observations that conflict with the ground truth.
Following the SFT phase, the model understands the environment and has learned useful reasoning patterns that help to accurately predict observations. Finally, an RL training phase is used to further refine simulation results across various environments in an on-policy manner. This phase draws data from the same set of internal trajectories used for SFT. Notably, world modeling is not a deterministically verifiable task—environment observations are often complex and open-ended. However, we can use rubric-based rewards to provide a learning signal by using an LLM judge to score the quality of each predicted observation given the ground truth next observation and a detailed scoring checklist.
In [4], each of the seven domains used to train the world model have their own custom rubrics. However, the same five criteria are considered for scoring in each domain: format, factuality, consistency, realism, and quality. Two example rubrics for the search and android domains are shown above. As we can see, the rubric for each domain may follow a different strategy, but each rubric always provides i) a clear definition of the scoring criteria and ii) a detailed explanation of how the final score should be derived. Each criteria is provided a Likert score in the range [1, 5]. The final score takes a mean of these criteria-level scores and multiplies the result by five, yielding a single score in the range [5, 25]. If the LLM judge fails to provide a valid score for the rubric, a default reward of zero is assigned.
“A subset of the data carries executable verifier code that produces a binary 0/1 correctness signal, scaled to [0, 25] to align with the rubric’s range. Rule-based rewards serve as an objective anchor, effectively mitigating reward hacking induced by open-ended rewards. We combine the two signals at 9:1 (rubric:rule), balancing multi-dimensional rubric feedback with strict binary correctness.” - from [4]
Given that LLM judges are biased and can be exploited, the reward signal for RL is not purely derived from rubric-based LLM judges—a hybrid approach is used that incorporates verifiable reward components. These reward signals are executable tests that check exact properties of the predicted observation. These verifiable rewards are scaled to the same range as the LLM judge reward, and the final reward is a weighted combination of these two components. Despite being assigned a lower weight relative to rubric rewards, the verifiable checks serve as an anchor for the RL process, helping to promote exact correctness and prevent reward hacking.
RL stability. Several additional tricks are used to ensure the stability of RL training in [4]. Interestingly, GSPO is adopted as the underlying RL optimizer, and we see that the rubric setup used for RL significantly impacts training results. Specifically, several simpler rubric settings are explored in [4], such as:
A reference-based reward that compares generated observations to the ground truth to assess their quality along a single dimension.
A binary reward that simply asks an LLM whether a predicted observation may have been derived from a real-world environment.
Although an LLM judge may be able to identify high-quality observations with these approaches, the reward is sparse—there is no feedback for why a particular observation is better or worse. In contrast, multi-dimensional rubrics provide a denser reward signal that captures prediction quality holistically. The model can perform well on certain dimensions while struggling with others. In fact, we see in [4] that the five quality dimensions improve at different rates during training.
“[All five rubric dimensions] improve at markedly different rates. Factuality shows the largest relative improvement (11.3%) yet remains the lowest-scoring dimension throughout, confirming that factual world knowledge is the hardest aspect of environment simulation.” - from [4]
Replicating the exact ground truth observation may not be possible for the world model. For example, the model may be asked to output a current timestamp or random file path. A naive LLM judge would score such cases as incorrect even if the predicted observation is valid. To avoid these false negatives, a taxonomy is introduced into the domain-specific rubrics used in [4]. More specifically, each rubric asks the LLM judge to classify the observation being scored into a set of possible categories—such as deterministic observations that require an exact match or runtime metadata where simply matching the structure of the observation is sufficient—and different scoring standards are defined for each of these categories.
World models also exhibit interesting forms of diversity collapse. Prior training stages could sample multiple training examples from one trajectory; see below.
[act_1, obs_1, act_2, obs_2, ..., act_N, obs_N] # full trajectory
[act_1] -> obs_1 # first training example
[act_1, obs_1, act_2] -> obs_2 # second training example
...
[act_1, obs_1, act_2, obs_2, ..., act_B] -> obs_N # N-th training exampleResampling trajectories in this way is found to cause a diversity collapse during the RL training process. These training examples share long prefixes and have highly-correlated rewards, especially when sampled together. Such correlated examples can potentially reduce reward variance, contributing to a diversity collapse similar to the echo trap behavior commonly observed in agentic RL.
World model benchmarking. To assess the performance of Qwen-AgentWorld models—or other language-based world models—AgentWorldBench is proposed. The benchmark contains 2,170 trajectories from the seven domains considered in [4], and queries are sampled from popular agent benchmarks like Terminal-Bench and ToolDecathalon. Models are evaluated by assessing the quality of simulated observations with the same rubric framework employed during RL training. Qwen-AgentWorld achieves the highest overall scores among the models that are considered, even outperforming recent Claude and GPT variants; see below. However, these baseline models are not specialized for world modeling applications.
World modeling for RL. Qwen-AgentWorld is an accurate world model, but how can we use this world model to help create better agents? In prior papers [1, 3], we usually avoid training a separate world model, instead choosing to incorporate world modeling components into the training process for the agent itself. There are two possible paradigms that can be following when using a world model to enhance the capability of an agent:
Decoupled: the world model can be kept separate from the policy and used as an environment simulator.
Unified: the policy and world model can be joined into a single model that predicts both actions and observations, similarly to ECHO [1] or PaW [3].
In the decoupled regime, the world model provides benefits in terms of control and scalability. Managing environments is a difficult infrastructure problem, but an accurate world model can be used in place of real environments during RL training—the policy predicts actions, and the world model simulates the response of the environment. Such an approach is referred to as a SimRL setup in [4].
From the empirical results obtained with this SimRL setup, we learn that:
Qwen-AgentWorld can generalize well to domains not included in training (e.g., OpenClaw environments). Even without domain-specific adaptation of the world model, we can make noticeable gains in policy performance for these held-out domains with SimRL training.
Replacing the Qwen-AgentWorld simulator with an off-the-shelf Qwen3.6 model yields degraded results in SimRL (i.e., the policy has no performance improvement), confirming that world model quality is important.
Control instructions, which inject perturbations (e.g., API errors, paginated responses that require follow-up calls, and other partial failures) that mimic real-world observations into the simulator, are necessary for SimRL to yield meaningful performance improvements; see above.
When directly comparing SimRL with real RL in the search domain, agents trained with SimRL match or exceed the performance of those trained using a real environment; see below.
In the unified paradigm, the world model and agent are combined into a single policy. The actual approach used in [4] simply applies world-model-style training to a Qwen3.5 model—thus training it to act as a world model—and directly uses this model as an agent with no additional post-training. Unlike ECHO and PaW, there is no joint training stage that incorporates policy updates and world modeling into the same objective. World modeling is just included as a separate training stage for the agent to develop an improved understanding of the environment.
Despite the simplicity of this approach, this additional phase of RL improves agent performance across several agent benchmarks; see above. Interestingly, policy performance even improves in environments that are not included in training. Such findings suggest that world-model training improves an agent’s ability to act reliably and solve problems in complex environments.
Important Takeaways
In this overview, we have studied several papers that adopt different approaches for language world modeling. ECHO [1] and PaW [3] integrate world modeling into the RL objective, while Qwen-AgentWorld [4] trains a dedicated simulator that can either replace real environments or serve as a foundation model for downstream agents. Despite their differences, these papers arrive at remarkably similar conclusions. Together, they suggest that world modeling should be viewed as a complementary component of agent training, providing dense supervision that supplements sparse reward signals and enables agents to develop a rich understanding of how their actions influence the external environment.
World modeling as a learning objective. The key insight that arises from this work is the fact that RL and world modeling are complementary. RL teaches the agent how to select actions that maximize reward, while world modeling teaches the agent how the environment responds to these actions. Rather than keeping these objectives independent, we can jointly optimize them by making simple modifications the standard RL training loop. This unified approach positively benefits sample efficiency, agent performance, and generalization.
Environment supervision. Environment observations are discarded in standard RL, but these observations actually provide a rich learning signal. Namely, each interaction step naturally describes the impact of an agent’s actions on the environment, allowing even unsuccessful trajectories that receive little reward to contribute useful supervision to the agent. However, not all observations are useful for training—we need some mechanism to identify those that are most useful and informative. Practically, we have seen several techniques in this overview that filter repetitive failures, invalid tool calls, and low-information observations from the world modeling objective. By doing this, we ensure that the observations used for training are likely to improve an agent’s understanding of the environment.
Overfitting. Although world modeling provides a valuable and dense source of supervision, we must be careful with how it is incorporated into the RL objective. If the world modeling loss is weighted too heavily, the agent may prioritize predicting environment observations over learning reward-maximizing behavior. Because world modeling provides dense supervised feedback at every interaction step, it may also overfit much more quickly than pure RL if not incorporated carefully. Overfitting is particularly pronounced in retrieval-heavy environments (e.g., web search), where world modeling may encourage memorization rather than learning generalizable dynamics. To solve these issues, we must:
Carefully balance the world modeling weight relative to RL.
Normalize the objectives separately to ensure comparable magnitude.
Use data filtering, tool selection, or early stopping to prevent overfitting.
As we see in [2], overfitting is a known risk when introducing world modeling into RL and usually arises later in the training process. Common symptoms of overfitting include a drastic decrease in held-out evaluation scores in tandem with a sudden boost in training rewards. When using techniques like ECHO [1] and PaW [3], our goal is not to maximize world modeling accuracy in isolation, but to learn environment representations that improve the policy itself.
Cost and scalability. World modeling is straightforward to implement and can be incorporated into existing agentic RL systems with relatively little overhead. As we have seen, unified training approaches for RL and world modeling typically reuse information that is already present in the rollouts for RL to compute the world modeling loss. The only additional cost is the added sum over observation token log probabilities to compute the supervised loss. Incorporating the world modeling loss into RL also tends to improve training efficiency. Beyond unified training strategies, we can even create separate world models for the purpose of simulating environment observations, which can reduce dependence on real environment infrastructure that is expensive and difficult to maintain.
Agent capabilities. World modeling improves more than benchmark scores. Agents trained with world modeling:
Learn to use tools more effectively.
Require fewer expert demonstrations.
Make better use of inference-time computation.
Generalize more reliably to unfamiliar environments.
These results suggest that world modeling provides a practical mechanism for improving agent training that is simple to implement and allows LLM agents to build a deeper understanding of the environments in which they operate.
New to the newsletter?
Hi! I’m Cameron R. Wolfe, Deep Learning Ph.D. and Staff Research Scientist at Netflix. This is the Deep (Learning) Focus newsletter, where I help readers better understand important topics in AI research. The newsletter will always be free and open to read. If you like the newsletter, please subscribe, consider a paid subscription, share it, or follow me on X, Medium, and LinkedIn!
Bibliography
[1] Shrivastava, Vaishnavi, et al. “ECHO: Terminal Agents Learn World Models for Free.” arXiv preprint arXiv:2605.24517 (2026).
[2] Prime Intellect. “True Agents Model the World.” https://www.primeintellect.ai/blog/true-agents-model-the-world (2026).
[3] Lu, Ning, et al. “Policy and World Modeling Co-Training for Language Agents.” arXiv preprint arXiv:2606.02388 (2026).
[4] Zuo, Yuxin, et al. “Qwen-AgentWorld: Language World Models for General Agents.” arXiv preprint arXiv:2606.24597 (2026).
[5] Shao, Zhihong, et al. “Deepseekmath: Pushing the limits of mathematical reasoning in open language models.” arXiv preprint arXiv:2402.03300 (2024).
[6] Guo, Daya, et al. “Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.” arXiv preprint arXiv:2501.12948 (2025).
The only difference here is the fact that the supervised objective is formulated as a objective to be minimized, whereas RL is formulated as an objective to be maximized. As a result, the supervised objective is negative, and the RL objective is positive.
This is exactly why ECHO is referred to as a world modeling approach. The agent is learning to directly predict the dynamics of its external environment. The terminal agent is a version of an embodied system that lives within the terminal—predicting the terminal corresponds to predicting the world for a terminal agent.
Specifically, authors in [1] choose to not include all terminal output tokens in the loss. For example, the harness used in [1] appends a rule-based WARNINGS:\n prefix to warnings, which is excluded from the loss function. Such tokens are specifically removed because they are deterministic and easily memorized.
ECHO specifically normalizes by this factor to ensure that experiments remain comparable when we mask tokens for inclusion in the supervised loss differently. If we change the number of tokens included in the supervised loss (e.g., remove a prefix that is commonly appended to observations), this could modify the scale of the environment loss. However, normalizing by the total number of observation tokens ensures that experiments that select observation tokens differently for training are comparable.
A standard GRPO loss with prompt-level advantage normalization, sequence-level loss aggregation, and no KL penalty. A batch size of 16 is used with 16 rollouts per prompt. The agent receives a reward of one if all tests pass and zero otherwise. Please see this overview of common GRPO tweaks if these settings are not familiar.
Interestingly, we see in [2] that this is a core reason why ECHO is so effective. For example, we see in the Cartridges paper that an effective way to internalize new info into an LLM is to reformulate it into QA pairs and train the LLM with this particular format of data. Similarly, we can internalize an environment by exposing it to the LLM in the format of on-policy tool calls and responses.
Fascinating direction. Selfishly the thing I keep bumping into is that world-model-style agents want to run a lot of inference per decision, and once you are doing many model calls per step the cost stops being about the model and starts being about how cheaply you can serve it. That pushed me toward running the cheap stuff locally and only reaching for a big model at the decision points.
Modeling future trajectories for RL reminds me of AlphaGo and MCMC. To what extent do you think its techniques are applicable here?