Predicted program feedback helps agents search for solutions
MLE-World: Predicting Machine Learning Experiment Outcomes with a Learned World Model · arXiv:2610.37175
In plain words
When an agent writes a machine-learning program, it usually has to run the program to learn whether it works. Those runs take time. This paper introduces MLE-World, which predicts a program’s likely output so an agent can compare ideas before choosing one to run.
In tests on 17 familiar tasks, five search procedures using these predictions had higher average leaderboard scores than the agent without them. The test does not show how well the approach works on wholly unfamiliar tasks.
The research
Machine-learning agents learn whether a program works by running it, but each run trains and evaluates a model and takes time. How can an agent compare more candidate programs when real feedback is costly?
MLE-World predicts program execution feedback so machine-learning agents can compare candidates before choosing which ones to run.
The authors tested MLE-World’s predictions on held-out programs, then used them to guide five search procedures. On the prediction test, a judge found more terminal-output matches than for the strongest prompted baseline.
In agent trials, every guided procedure had a higher observed mean leaderboard percentile than Base AIDE. This supports using predicted feedback to guide search in the tested tasks, while the execution gate saved run time with results that varied by procedure.
The main prediction test used new programs from tasks included in training, so it does not establish accuracy in entirely unfamiliar task environments.
A small exploratory test on three excluded tasks suggests some transfer, but failed runs dominated its sample; predictions of successful training outcomes and timeouts remain less established.
What the evidence shows
- The cost of real feedback
Machine-learning engineering agents search for solutions by writing programs, running them, and using the results to revise their code. Because each real run trains and evaluates a model, feedback is costly.
MLE-World is a learned world model that predicts a candidate program’s execution feedback, giving search procedures information to compare ideas before deciding what to run.
- Checking its predictions
The authors first tested whether MLE-World could predict execution results for new programs. A Coder judge compared predicted and observed terminal outputs, and the reported accuracy is the share of programs judged to match.
It labeled 77.9% of MLE-World predictions as matches, compared with 41.1% for the strongest prompted baseline, DeepSeek. These were new programs from tasks also represented in training.
- From predictions to search
The next question was whether predictions help the agent find better solutions. Across 17 machine-learning competition tasks, five search procedures used MLE-World to compare candidate programs, while Base AIDE searched without a world model.
All five had higher observed mean leaderboard percentiles than Base AIDE. A percentile measures the share of leaderboard entries beaten; the authors averaged the scores across tasks and seeds.
- Using confidence to save runs
The authors also tested whether the agent could save time by relying on predictions when confidence was high. A gate requested real execution for low-confidence predictions and for candidates with a new best predicted validation score.
Across the five procedures, it saved 30.0% to 39.8% of real-execution time while broadly maintaining observed mean percentiles, though the UCB procedure’s gated mean was lower.
Deep
- Original paper authors
- Brian Ko, Han Wang, Hejia Geng, Li Erran Li, Sherry Yang
- Submitted to arXiv
- License
- CC BY 4.0