A customer-delivery benchmark finds agents struggle to build effective systems

When a customer’s request may describe the wrong problem, can an AI agent discover the constraints that matter and deliver a system that improves the customer’s business?

A benchmark can test that full delivery task by letting agents inspect customer materials, ask stakeholders questions, and measuring the resulting system against the customer’s own metric.

FDE-Gym tests agents on customer delivery tasks, finding that they struggle both to discover unstated constraints and to turn gathered facts into effective systems.

The test varies how much an agent is told, from a fully specified task to the customer sponsor’s original request, while keeping the workspace, stakeholders, and grading criteria the same.

On the same 20 original tasks, three leading models’ average score fell by 0.28 when rules and constraints had to be found.

In the primary setting, the top configuration averaged 0.56 across 95 tasks, and resolved 14. The score compares business improvement with an unchanged system and an expert solution, while accounting for incidents and stakeholder contact costs.

Even when one model obtained every tracked key fact on a subset of tasks, high business uplift remained uncommon.

These results describe tested models in one minimal agent framework and simulated customer settings. There is no human baseline, and scripted stakeholder replies and expert-set contact costs do not measure how real customer interactions or their costs would affect delivery.