Higher reasoning effort can miss cheaper successes
What Effort Level Should I Choose? On the Economics of Reasoning Effort in LLM Agents · arXiv:2610.34491
In plain words
Language models can be set to spend different amounts of effort reasoning through a task. This paper checks whether more effort reliably means more success, and what it costs across eight models and four tests, including tasks where an agent uses tools over several steps.
More effort did not always keep the tasks solved at lower effort, and it often raised costs. A strategy that retries at high effort only after a failed correctness check kept or improved success in the evaluations, though it did not cut costs for every model and test.
The research
When a language model solves a task, users cannot tell whether the effort setting they chose was necessary. Extra effort can raise the bill, yet choosing less effort in advance could miss a solution.
The question is whether task-level evidence can show when the additional computation helps and whether a different way of allocating effort can reduce that uncertainty.
The authors compare effort settings task by task and test whether retrying at high effort only after a failed check preserves success at lower cost.
The authors compare effort settings within each model and test configuration, recording task outcomes and the full token cost of agent runs. They find that higher effort can miss tasks solved at lower effort, and that it costs more on tasks solved at every setting.
They also test starting low and retrying high after an external failure check. This laddering strategy kept or improved success in the evaluations, but its cost advantage depended on the test and model.
The agentic tasks in the main evaluation were run once per effort level, so observed reversals do not establish that higher effort lowers a task’s underlying chance of success. Laddering also depends on a reliable external correctness check and charges for a fresh retry.
What the evidence shows
- What effort changes
This paper is a task-level audit of how language-model reasoning effort affects task success, execution cost, and agent behavior. The authors compare model-specific effort settings across four tests: two use a single model call, while two involve agents acting over multiple turns.
The question is whether higher effort consistently preserves lower-effort successes and whether any added success justifies the extra cost.
- Totals hide reversals
Terminal-Bench 2.1 tests agents on tasks in a terminal environment; the reported scores here are counts of solved tasks. For GPT-5.6-sol, moving from high to xhigh effort lost eight tasks and gained five, bringing the total down from 76 to 73 of 89.
Each of the eight models had at least one task solved at its lowest effort that it missed at its highest. Each task was run once per setting, so these are observed reversals rather than evidence of lower success probability.
- Cost without added success
The authors also compare spending on tasks each model solved at every effort setting. In the original SWE-Bench Pro pilot, which tests agents on repository software tasks, GPT-5.6-sol’s mean API cost per task rose from $0.27 at low effort to $1.24 at xhigh.
Since the task set was restricted to those solved at every setting, the increase shows higher spending without added success on that shared set.
- Environment affects scores
A comparison of the original and corrected SWE-Bench Pro environments helps qualify the original scores. The corrected environment removes repository history and restricts network access; on the same 44 instances, GPT-5.6-sol’s solve rate at xhigh fell from 77.3% to 59.1%.
All eight models’ solve rates fell at their highest evaluated setting. Both history and network access changed, so this comparison does not identify either one as the cause on its own.
- Feedback guides retries
Finally, the authors test laddering: begin at low effort, then retry at high effort only after an external check reports failure. It matched or improved success over always using the highest setting across the evaluated models and tests, but its cost benefit varied.
On Terminal-Bench, which tests agents completing tasks in a terminal environment, laddering improved the share of tasks solved for all eight models; costs fell for four, while the other four gained success at increased cost. The approach also requires reliable external feedback and pays for a fresh retry.
Deep
- Original paper authors
- Kaiser Sun, Yongchan Kwon, Zain Hasan, Shang Zhu, Kaitlyn Zhou
- Submitted to arXiv
- License
- CC BY 4.0