Fascinating direction. Selfishly the thing I keep bumping into is that world-model-style agents want to run a lot of inference per decision, and once you are doing many model calls per step the cost stops being about the model and starts being about how cheaply you can serve it. That pushed me toward running the cheap stuff locally and only reaching for a big model at the decision points.
FWIW world modeling does not add inference cost. In fact, it tends to decrease inference costs (i.e., agents make tool calls more efficiently leading to problems being solved in fewer turns).
Ah, that reframes it, thank you. I was picturing the world model as something the agent queries at inference, but your point is that it is a training objective. The observations standard RL throws away become a dense supervision signal, so the agent internalizes the dynamics and just needs fewer, better-aimed turns. That framing, that the trajectory is already information-rich and we are simply not using it, that is key. Fewer turns from a smarter policy beats anything I can squeeze out on the serving side.
Definitely related because MCMC + world modeling are both used to simulate trajectories. The main conceptual difference is here we are less focused (depending on the paper) on building a simulator and more focused on using a simulation-style objective to improve agent behavior. E.g., ECHO and PaW do not actually use the agent to simulate responses from the environment.
Fascinating direction. Selfishly the thing I keep bumping into is that world-model-style agents want to run a lot of inference per decision, and once you are doing many model calls per step the cost stops being about the model and starts being about how cheaply you can serve it. That pushed me toward running the cheap stuff locally and only reaching for a big model at the decision points.
FWIW world modeling does not add inference cost. In fact, it tends to decrease inference costs (i.e., agents make tool calls more efficiently leading to problems being solved in fewer turns).
Ah, that reframes it, thank you. I was picturing the world model as something the agent queries at inference, but your point is that it is a training objective. The observations standard RL throws away become a dense supervision signal, so the agent internalizes the dynamics and just needs fewer, better-aimed turns. That framing, that the trajectory is already information-rich and we are simply not using it, that is key. Fewer turns from a smarter policy beats anything I can squeeze out on the serving side.
Modeling future trajectories for RL reminds me of AlphaGo and MCMC. To what extent do you think its techniques are applicable here?
Definitely related because MCMC + world modeling are both used to simulate trajectories. The main conceptual difference is here we are less focused (depending on the paper) on building a simulator and more focused on using a simulation-style objective to improve agent behavior. E.g., ECHO and PaW do not actually use the agent to simulate responses from the environment.