A benchmark shows task completion can miss user needs
PersonaCUABench: Benchmarking Computer-Use Agents Serving Diverse Personas · arXiv:2610.22990
In plain words
Computer-use agents can finish a task as written while missing what the person asking actually wants. PersonaCUABench tests this by having simulated users phrase the same website task in their own way, then checking task requirements and asking those users how satisfied they are.
Across nine agents, average task completion was lower for persona requests than for unchanged instructions. Even when a persona task was fully completed, the simulated user did not always give the top satisfaction rating. The test covers website tasks and simulated users, not every kind of computer use or real people.
The research
When a computer-use agent completes a task, has it served the person who asked, or only matched a fixed instruction? Users may add constraints, leave needs unstated, or ask the agent to pause before consequential actions, so those two kinds of success can diverge.
PersonaCUABench compares agents on unchanged and persona-phrased website tasks, measuring both requirements met and simulated-user satisfaction.
PersonaCUABench compares the same website tasks under original instructions and requests phrased by simulated users. Nine computer-use-agent backbones had lower mean task completion on persona requests, where the score is the share of task requirements met.
The losses often followed attempts to handle a stated requirement, and some requested pauses counted as failures under the original checker.
Simulated users also did not always give fully successful persona tasks their top satisfaction rating, so task completion alone did not capture their judgments.
The results come from constructed personas and simulated users performing website information and action tasks. They do not establish how real people or agents in other settings would fare, and the profile and question interventions were tested only with GPT-6 Astra.
What the evidence shows
- A task can miss user needs
Computer-use agents are often judged by whether they complete a fixed task. But a person may leave needs unstated, add a personal constraint, or ask the agent to pause before acting.
PersonaCUABench is an evaluation set that asks whether agents handle the same website task when different simulated users express it in their own way. It also measures user satisfaction separately from task completion.
- Compare the same task
For each task, a persona agent uses a profile and the original instruction to phrase a request and answer the computer-use agent’s questions. The agent under evaluation sees only that dialogue, not the profile or original instruction.
Researchers compare it with performance on the unchanged request for the same task. Task checkers score requirements met; the persona agent separately rates satisfaction.
- Persona requests lower scores
Across 74 tasks and nine agent backbones, mean task completion was 0.689 for original requests and 0.441 for persona requests. The score is the share of requirements met, and persona results average four user groups.
Every backbone scored lower on persona requests. The paired result indicates that completing the fixed instruction did not reliably transfer to requests expressed by simulated users.
- Why scores and users differ
A diagnosis of the lower-scoring persona cases found that 59.9% were cases where the agent engaged with a raised requirement but still missed it. Another 11.2% were requested pauses that the original checker counted as failures.
Separately, among fully successful cases, simulated users gave the top satisfaction rating less often for persona requests than for original requests.
- More interaction did not fix it
Two interventions with GPT-6 Astra tested whether showing the profile or requiring a question would recover task completion. Neither recovered the persona-request loss.
Requiring a question also lowered completion on original-request action tasks, in part when the simulated user’s answer added a constraint absent from the instruction. These results suggest that more information or interaction alone was not enough in these tests.
Deep
- Original paper authors
- Haoming Li, Defu Cao, Xu Hu, Yingtao Luo, Yiyang Wang
- Submitted to arXiv
- License
- CC BY 4.0