In plain words
The paper introduces ARIA, a learned decision rule for tasks where a lower-cost model does the work and a stronger model can advise it. At recurring workflow points, ARIA either lets the task continue or asks for advice; the lower-cost model still chooses actions and delivers the result.
Across three tested settings, ARIA recovered part of the stronger model's task-success advantage for lower mean evaluation cost than letting the advisor do every task. On terminal workflows, its mean success matched a fixed advice schedule.
The research
In a long task, the value of stronger reasoning can change as new evidence, mistakes, and chances to recover appear.
It remains unclear whether a system can choose the moments when advice is likely to improve the final result enough to justify its cost, instead of relying on the executor's own confidence.
ARIA is a learned decision rule that uses workflow progress and visible execution evidence to choose when a stronger model advises the task executor.
The authors learned ARIA's decisions on development tasks, then tested it on separate software-engineering, terminal-workflow, and interactive customer-service tasks. Task success is the share of tasks completed; headline results average three evaluation seeds.
Against a lower-cost executor doing each task alone and a stronger advisor doing it alone, ARIA regained part of their task-success gap at lower mean evaluation cost than advisor-only execution. On terminal workflows, it tied a fixed advice schedule in mean success.
Headline results average three evaluation seeds, so small differences between nearby methods are descriptive, not evidence of statistical separation. The experiments use one fixed executor and advisor pair within Magentic-One; results for other pairs or orchestration frameworks are untested.
What the evidence shows
- Advice has changing value
ARIA (Allocation of Reasoning in Agentic systems) is a learned meta-controller: a separate rule that decides when a stronger advisor should give guidance in a task workflow. The lower-cost executor remains responsible for actions and final delivery.
The research question is whether advice can be placed where it changes task success enough to justify its cost, since useful moments may emerge as evidence and errors appear.
- Signals guide consultation
ARIA has opportunities to consult before the next instruction, when progress stalls and replanning begins, or just before delivery. Its decisions use coarse workflow progress, observable execution evidence, and whether prior advice is already in the task history, rather than the executor's self-reported confidence.
The advisor is read-only: it may shape what the executor does next, but does not take over tool use or submission.
- Three settings test the rule
Evaluation tests three held-out task sets: SWE-bench Verified for software-engineering tasks (50), Terminal-Bench 2.1 for terminal workflows (50), and tau-squared-bench Telecom for interactive customer-service tasks (40).
Task success is the share of tasks completed; each headline mean averages three evaluation seeds. Within each setting, the same held-out tasks are used across the compared methods.
- Success rises at lower cost
The main comparison uses Executor-only, where the lower-cost model does all the work, and Advisor-only, where the stronger model does it, as reference points. Performance gap recovered is the share of their task-success difference regained by ARIA; costs are ARIA's mean evaluation cost as a share of Advisor-only's. Across software, terminal, and telecom tasks, ARIA recovered 68.2%, 44.4%, and 80.6% of the gap at 48.1%, 40.9%, and 34.5% of Advisor-only cost, respectively.
On Terminal-Bench, ARIA tied the fixed advice schedule Heuristic at 34.7% mean success; mean costs were $24.93 versus $27.47 per run. With three seeds, nearby differences are descriptive.
- Similar volume, different results
To check whether advice placement mattered beyond roughly matching how often advice was given, the authors compared ARIA with Random, which makes independent chance-based requests at eligible points under the same four-call limit.
On SWE-bench Verified's 50 held-out software-engineering tasks, Random averaged 28.0% success with 2.53 consultations per task run; ARIA averaged 48.0% with 2.78. Random used 90.9% of ARIA's consultation volume, a close but not exact match. This supports placement's role in this setting.
- Advice can go unused
To identify what remained unresolved, a language-model judge labeled failed trajectories with the Multi-Agent System Failure Taxonomy, a scheme for classifying agent failure types. Among annotated failed SWE-bench trajectories with usable traces, 58% received “ignored agent input” as the primary label.
Human validation supports that relevant advice had reached the executor but was not acted on. Successful and untraceable episodes were excluded, so this describes failure composition rather than all tasks or the cause of any one failure.
Deep
- Original paper authors
- Pascal Sowietzki, Steffen Holter, Yannick Metz, Emil Kasper, Raphael Baur
- Submitted to arXiv
- License
- arXiv non-exclusive