CLI agents can finish tasks but miss collaboration checks
RealCLI: Toward a Unified Evaluation of Human-Agent Collaboration for CLI Agents · arXiv:2610.22507
In plain words
This paper introduces REALCLI, a set of command-line tasks for checking both whether software agents finish a task and how they work with a user. In one version, the agent gets all instructions at the start. In another, a scripted user adds details, corrections, and approval decisions along the way.
Among interactive runs that passed the task check, 91% still missed at least one collaboration check that applied. Finishing the task did not guarantee good collaboration in these tests.
The research
When an agent changes files or runs commands over a long task, a correct final result may not reveal whether it respected a user's changing requirements or sought approval before sensitive actions. Can evaluations that judge only the final result capture this part of working with an agent?
REALCLI pairs command-line task tests with scripted user interactions to measure whether agents finish tasks and collaborate well.
REALCLI runs the same command-line tasks with complete instructions upfront and with a scripted user who supplies information and corrections during execution. The task verifier checks the final environment state in both modes; COLLAB also receives a separate score for collaboration behavior.
Among runs that passed the task verifier in COLLAB, 91% missed at least one applicable collaboration scenario. Since the task result and collaboration checks capture different outcomes, this supports evaluating them separately.
The collaboration results come from scripted user interactions on 101 tasks, evaluated across 11 models and three agent scaffolds. They do not directly show how agents collaborate with real users or perform outside these tested tasks and configurations.
What the evidence shows
- What task scores can miss
REALCLI is a paired benchmark, or set of tests, for command-line agents. It measures task completion and collaboration with a user on realistic tasks.
The paper addresses a gap in evaluations that give agents complete, fixed instructions and judge mainly the final code or environment state. Those evaluations can miss whether an agent handles changing requirements, verifies its work, communicates risks, and preserves user control during execution.
- A paired test of collaboration
To make collaboration assessable, REALCLI defines five dimensions: understandability, steerability, observation and recovery, trust calibration, and safety and risk awareness. Scenarios spell out the behavior being assessed and the evidence needed.
For each task, the EXEC mode gives the full specification upfront; the paired COLLAB mode adds a scripted user who progressively supplies information and may correct or redirect the agent. Both modes share the task goal, environment, and task verifier.
- Two kinds of evidence
The paper separates task outcome from collaboration outcome in COLLAB. The task verifier checks the final environment state.
A separate checker assesses applicable collaboration checklist items using interaction and environment evidence. A task passes a scenario only when every applicable item passes; scenario results are then aggregated through the collaboration dimensions.
This lets the evaluation ask whether an agent both completed the task and handled the user interactions required along the way.
- Success did not ensure collaboration
Among interactive runs that passed the task verifier, 91% still missed at least one applicable collaboration scenario. The result shows that a successful final state can coexist with a missed requirement for authorization, conflict handling, or risk communication.
It measures the share of successful interactive runs with a missed applicable scenario, not the share of all runs that failed collaboration checks.
- Averages hide different patterns
Average task pass rates alone also obscure how interaction changed outcomes. Across 11 Claude Code configurations and 101 matched tasks, mean pass rates were 46.8% with user interaction and 47.4% with complete instructions upfront.
But 17.9% of matched runs changed from failure to success, while 18.5% changed from success to failure. Collaboration scores were also uneven: agents did relatively well on understandability and observation and recovery, and less well on steerability, trust calibration, and safety and risk awareness.
Deep
- Original paper authors
- Yunqi Liu, Peiyu Wang, Bingli Wang, Leo Luo, Xuanwu Song
- Submitted to arXiv
- License
- arXiv non-exclusive