OpenGCUA lets computer agents coordinate screen and shell actions in code

When frontier computer-use systems keep their prompts, device tools, and action records closed, researchers cannot readily reproduce their behavior or compare designs. Which choices help an agent complete varied computer tasks, and can those choices be studied with an open system?

OpenGCUA lets a model combine screen and shell operations in executable programs, then evaluates that design with existing computer-use models.

The main question is whether more open, flexible computer-use designs can be studied when key details of frontier systems are unavailable. OpenGCUA presents an agent design in which a model chooses what to observe, how to act, and how to organize its work, and evaluates it with existing models.

On OSWorld-v2, a test of computer tasks, evaluators assign partial-completion scores. Across eight models on its 82-task offline subset, the full design scores 19.6 points higher on average than batched screen-only actions.

Researchers average scores over tasks within each model, then weight models equally; the result supports a difference between these configurations, not a single-component explanation.

The main comparison tests the full OpenGCUA design against batched, screen-only actions, so it cannot show how much each design change contributed.

The real-device results cover four macOS tasks and one selected run per task; the authors say they characterize those configurations, not a general model ranking.