A repair pipeline recovers software issue tasks across 13 languages
Longtail-SWE: High-Quality SWE Tasks for Underrepresented Languages · arXiv:2611.10304
In plain words
Longtail-SWE is a set of software issue-resolution tasks: agents try to fix real problems in code repositories. A judge checks whether a failed attempt reflects a bad solution or a flaw in the task or its tests, and a repair agent can revise tests or clarify the task. The set covers 202 tasks from 13 underrepresented programming languages.
From the same starting pool, the full Longtail-SWE construction path kept 202 tasks after validation, compared with 68 through SWE-rebench V2. Several stages differed, so this comparison cannot show which one led to the difference.
The research
Software-fix scores are useful only when tests accept valid ways to solve an issue and catch incorrect ones.
The findings make it reasonable to expect that reviewing repairable tasks, rather than discarding them at the first unclear description or test, can widen language coverage; the full-path comparison does not pinpoint which step produced the difference.
Longtail-SWE is a 202-task software issue-resolution benchmark that uses solver attempts and task repair to retain and evaluate issues across 13 underrepresented languages.
The construction paths began from the same 3,261 candidates, but differed in screening, repair, selection, and later review. After post-evaluation validation, Longtail-SWE retained 202 tasks and the SWE-rebench V2 path retained 68.
This supports the narrower conclusion that the combined process can preserve more tasks through construction; it does not isolate any single stage.
The full construction paths differed at several stages, and the paired test comparison covered only 63 released tasks whose descriptions stayed unchanged; it isolates test changes for that subset only.
What the evidence shows
- The coverage problem
Longtail-SWE adds tasks from programming-language ecosystems that repository-level evaluations often leave sparsely represented. The paper asks how to retain repairable issues when unclear descriptions or flawed tests might otherwise exclude them.
- Diagnose before discarding
The construction process has a solver attempt each task, then a judge review the attempt, patch, test outcomes, description, reference patch, and tests. The judge distinguishes solver mistakes from defects in the task, tests, or environment.
A separate repair agent can revise tests or add a clarification, so candidates with repairable problems can be checked again rather than filtered out immediately.
- More tasks made it through
The larger final set suggests that retaining candidates with repairable defects and reviewing them later can preserve more tasks through construction. Since the paths differ at several stages, the final gap cannot show which stage made the difference.
- Tests changed the verdict
Because the same solutions and unchanged descriptions were tested both ways, the higher scores indicate revised tests changed which solutions counted as successful on this subset. This isolates the test change for these tasks, not the effects of description repair or the full construction process.
- A comparison across models
The released tasks also served to compare nine model configurations, each making three attempts per task; success required passing all designated tests. Opus 5.5 (high) had the highest reported Pass@1 point estimate, at 81.52%.
The result demonstrates a use for the task set in comparing these configurations under a shared test protocol, while remaining specific to the evaluated models and tasks.
Deep
- Original paper authors
- Vladimir Dobrovolskii, Ibragim Badertdinov, Alexander Golubev
- Submitted to arXiv
- License
- CC BY 4.0