In plain words
A world model lets a planner try out actions in an imagined future. But a model can make most frames look right and still get the reward or end of an episode wrong, changing which plan looks best.
The paper introduces a test that swaps selected imagined frame pairs for real ones and checks how much the return error changes. Across 44 reliable Atari model, collection pairs, swapping pairs around reward or ending events removed more error than swapping the same number of random pairs.
The research
If a planner chooses actions by comparing imagined returns, a few missed rewards or episode endings can distort its ranking even when the generated frames look convincing. The question is whether those decision-relevant errors are spread through the rollout or concentrated at particular transitions.
Liu and Tang’s transition-level substitution diagnostic swaps real frame pairs into imagined rollouts to locate which steps drive return error.
For each probe, the researchers compared a model’s imagined return with the return from an emulator replaying the same actions. They replaced event frame pairs or an equal number of random pairs, then measured the change in mean absolute return error across probes.
Event patches beat random patches in 39 of 44 reliable Atari model–collection pairs. Because event transitions did not have greater pixel error, the result suggests that visual fidelity alone can miss the steps that matter to return.
The attribution results cover only model–collection pairs whose frozen reward detector passed an independent-data reliability check.
In the repair comparison, the deterministic error model received more training updates than the diffusion model, and the reported control-task result includes only comparisons whose detector passed.
Analysis
Deep
- Original paper authors
- Haoyue Liu, Xiaoying Tang
- Submitted to arXiv
- License
- arXiv non-exclusive