BenchMeUp turns research specifications into runnable benchmarks
BenchMeUp: Building Software Engineering Benchmarks from Natural-Language Specifications · arXiv:2610.37283
In plain words
Researchers often spend a lot of effort assembling software tasks for testing techniques: they find examples, prepare the code and setup, and decide what a correct result looks like.
BenchMeUp is an automated approach that starts from a researcher's written task description and builds a collection of runnable examples with a shared way to use them.
In four tested tasks, more BenchMeUp examples than examples from a general coding agent met the authors' checks for real sources, required files, working builds, and expected results. The checks were limited to those tasks and specifications.
The research
Can researchers build a useful software-engineering benchmark from a written task description without manually assembling and checking every case?
That matters because benchmark construction requires choosing suitable examples, preparing their code and environments, defining expected outcomes, and verifying them, work that can make new tasks or languages costly to study.
BenchMeUp turns a researcher's task specification into shared search rules and a construction plan, then coordinates agents that build, replay-check, and package benchmark instances.
For four task specifications, the authors compared instances delivered by BenchMeUp with those from Codex using task-adapted scripts for seven checks, from real sources and required files to working builds and expected outcomes. All seven checks were met by 180 of 198 BenchMeUp instances and 10 of 200 Codex instances.
The authors link BenchMeUp's stronger results on uniqueness, completeness, and buildability to its shared construction plan and replay checks. The comparison measures these delivered instances under the authors' assessment; it does not isolate the effect of any one design choice.
The comparison covers four task specifications and does not isolate which design choice caused the difference.
Replay confirms that instances meet their planned checks, but it does not show that those checks faithfully represent the task; the evaluation also used GitHub data and does not establish results for tasks using other sources.
What the evidence shows
- A costly construction task
BenchMeUp is an automated approach for turning a researcher's written software-engineering task specification into a collection of runnable benchmark instances with a common command-line interface.
The question is whether this work, which usually involves selecting examples, preparing their environments, and defining and checking outcomes, can be coordinated from that specification.
- Shared rules guide agents
The approach first generates shared search rules and a construction plan from the specification. Search agents use the rules to find candidates, while construction agents follow the plan to prepare each instance.
This makes decisions about selection and required artifacts common across agents, although agents still assess individual candidates and carry out the steps for them.
- Replay catches execution failures
During construction, the framework runs submitted scripts in fresh containers and compares their results with what the plan expects. Failed checks go back to the construction agents for correction before they can proceed.
Of 198 constructed instances across four tasks, 108 initially had at least one failed replay check, showing that the process surfaced execution problems that needed attention.
- Quality checks favor BenchMeUp
The authors compared delivered instances with Codex, a general-purpose coding agent, using seven task-adapted checks for source validity, specification fit, uniqueness, completeness, buildability, interface function, and expected results.
All seven were met by 180 of 198 BenchMeUp instances and 10 of 200 Codex instances. The authors associate BenchMeUp's stronger results on construction-related checks with its plan and replay process, while noting that remaining issues often involved interpreting the specification.
Deep
- Original paper authors
- Islem Bouzenia, Michael Pradel
- Submitted to arXiv
- License
- arXiv non-exclusive