A weak action chooser can still teach agents which steps help
Actor, Judge, or Teacher? Understanding Decision Models in Long-Horizon Agents · arXiv:2611.17167
In plain words
This paper asks whether a model that struggles to choose the right steps can still help train an agent. It studies models that act, judge recorded actions, or score actions to guide training.
After an agent finishes a task, JEV scores its actions using the goal and what happened. A small agent trained with these scores succeeded more often in household and shopping simulations than one trained using task outcomes alone. The tests covered two text-based settings.
The research
For agents facing tasks with ordered subgoals, choosing the right step before seeing its consequences is hard. A model’s difficulty acting need not settle whether its judgments can provide useful training feedback. That distinction affects whether action ability alone is a fair measure of its value for learning.
The study tests decision models as action judges and training teachers, asking whether seeing outcomes lets a weak actor give useful step-by-step feedback.
JEV scored actions in completed rollouts from a policy being trained, using the goal and full trajectory; its centered scores were added to outcome-based training feedback. Compared with previously reported outcome-only training, the 1.5B Qwen policy’s ALFWorld task success rose from 72.8% to 82.9%.
WebShop showed a similar increase, where success means buying an item that fully matches the request. Together, the results support the view that after-the-task action scores can help a policy learn which steps contribute.
The outcome-only comparison comes from prior work, and the cited setup does not state whether training began from the same checkpoint. The evaluation covers reported tasks in two text-based environments, not longer or more varied tasks.
What the evidence shows
- Acting makes order difficult
Decision models can choose an agent’s actions, judge actions already taken, or score actions to guide training. In ALFWorld, a text-based household simulation where success means completing a task, JEV succeeds on 70.8% of tasks with two requirements that can be done in either order, but at most 18.8% of tasks where one change must happen before placement.
This contrast motivates asking whether a model can recognize useful actions after their consequences appear.
- Later events change judgments
The study’s decision models can also judge actions in recorded trajectories. For the same actions and question, the researchers compared a view showing only what happened up to the action with one showing the whole trajectory.
On the Judge’s 0-to-1 contribution scale, JEV’s mean score was 0.41 with the earlier context and 0.59 with the full trajectory. All six Judges showed the same direction of change, suggesting later events help them recognize an action’s contribution.
- Ask about each action’s role
With JEV shown the full trajectory, the researchers compared asking whether the task would succeed after an action with asking whether that action helped achieve the goal.
On 462 ALFWorld trajectories containing both milestone actions and no-ops, the within-trajectory score gap was larger for the contribution question (0.36) than for success prediction (0.05), indicating it separated actions that changed progress from those that did not more clearly in this set.
- Action scores aid training
This comparison asks whether after-rollout action judgments add useful training feedback beyond task outcomes alone.
For the 1.5B Qwen policy on reported ALFWorld tasks, success averaged over three evaluation seeds was higher with JEV’s Hindsight + Progress scores (82.9%) than with outcome-only training (72.8%), supporting usefulness in this tested setup.
- The pattern spans two settings
The same training setup also improved the 1.5B policy’s task success in WebShop, a text-based shopping simulation where success requires buying an item that fully matches the request. The results support a distinction between choosing actions before their effects are known and judging them afterward.
That distinction matters to the paper’s claim: an Actor’s task performance alone may not capture whether its judgments can help train another policy.
Deep
- Original paper authors
- Haochen Liu, Qianhao Zhu, Zixuan Wang, Dingming Li, Tongbo Chen
- Submitted to arXiv
- License
- CC BY 4.0