A new physics test finds research remains hard for AI agents
FrontierPhysics: Evaluating Agents for End-to-End Physics Research · arXiv:2611.06910
In plain words
FrontierPhysics is a set of physics research tasks for checking whether AI agents can do more than answer questions with known solutions. Each task starts with a broad prompt: the agent must investigate, plan, do the work and turn in checkable numbers and a scientific paper.
On this test, Claude Opus 5.5 passed 46 of 168 attempts. A pass meant its numbers cleared the checks and its work met every essential rule, so the result reflects the whole research task in this benchmark.
The research
A score on fixed physics questions can hide whether an agent can carry a research project from an open prompt to checked results and a defensible explanation.
FrontierPhysics makes that gap visible, so claims about research ability can be judged against the whole task, including numerical correctness and the quality of the work.
FrontierPhysics is an end-to-end physics research benchmark that asks agents to investigate open prompts and submit checkable results plus a scientific manuscript.
For each task, agents received a broad prompt and had to investigate, plan and do the research, then submit numerical outputs and a manuscript.
A verifier checked outputs against reference values, while a reviewer agent graded the work against task rubrics; a pass required both checks and every essential blocker criterion.
Across the 56 tasks, Claude Opus 5.5 passed 46 of 168 attempts (27.4%). That rate shows how rarely this configuration met the benchmark's combined correctness and essential-quality bar on these tasks.
The account of why agents struggled comes from 168 GPT-6 Astra xhigh attempts, not from every tested model. The pass result is for Claude Opus 5.5 on FrontierPhysics v0.1 and its 56 tasks.
What the evidence shows
- Beyond fixed questions
FrontierPhysics is an end-to-end benchmark for evaluating whether AI agents can conduct physics research. Its tasks address a gap the authors identify in existing tests: many focus on fixed questions, isolated subtasks or reproducing work with solutions provided.
Those tests can miss whether an agent can handle an open-ended project while producing both correct results and sound research.
- From prompt to paper
Each task gives an agent a high-level prompt without spelling out a method or solution. The agent investigates, plans and conducts the work, then submits numerical outputs that can be checked and a scientific manuscript.
This makes the test about carrying research through from an open prompt to evidence and an explanation, rather than completing one isolated step.
- One combined pass bar
Because passing required both correct numerical outputs and every essential rubric criterion, the rate captures attempts meeting the combined bar. It shows that completing the full research task remained difficult for this tested configuration.
- What review suggests
A separate manual review examined 168 GPT-6 Astra xhigh attempts. Reviewers most often noted insufficiently rigorous reasoning and missing expert intuition, with literature synthesis also a recurring difficulty.
The authors interpret these patterns as weaknesses in reasoning and expert judgment rather than physics knowledge or execution, but the diagnosis describes this reviewed cohort, not every model tested.
Deep
- Original paper authors
- Bingran You, Wenjun Ke, Ria Rosenauer, Bo Liu, Wenbo Chen
- Submitted to arXiv
- License
- CC BY 4.0