Learned agent skills added little test success and used more tokens
Does Your Generalist Skill Optimizer Really Incentivize Agentic Tasks? Rethinking Skill Optimization via Bipartite Attribution · arXiv:2610.37201
In plain words
A skill is a set of instructions an agent receives before trying a long task. Skill optimizers edit those instructions, keeping changes that score well on a small development set.
BiPA is a measurement method that checks whether those scores could come from selection noise and tracks how edits affect task success and the text read across turns.
Across 16 learned skills, average test success was only 0.1 percentage points above the shared starting skill, while input-token use was higher. That is an overall average, not a result for every skill.
The research
A skill that wins on a small development set can look more useful while quietly making each task more expensive. The key decision is whether a development-set win is enough to keep an edit when test success stays near the seed and the agent reads more text. BiPA makes that tradeoff visible for the optimizers tested.
BiPA measures whether skill edits improve test success, how much selection noise can explain development gains, and which behaviors add input-token cost.
BiPA replays each optimizer’s acceptance rule with unchanged candidates, then compares learned skills with the seed on test tasks and traces input tokens across turns. Unchanged-candidate replays often matched DeepSeek development gains, while pooled test success was only 0.1 percentage points above the seed and input-token use averaged 1.16 times as much. Together, these results point to weak overall test gains alongside higher reading cost.
BiPA analyzes recorded episodes, so it cannot show that a particular rule caused an observed behavior or predict its cost. The study used one shared hand-written starting skill, two models in one harness without history compaction, and three long-task collections.
What the evidence shows
- Measure noisy gains and costs
BiPA (Bipartite Attribution) is a post-hoc measurement method for skill optimizers. It addresses a problem with long tasks: a small development set and variable success can make a skill edit look helpful, while a final success score hides the extra text an agent reads over its turns.
BiPA measures both the selection effect and the recorded token costs without changing the optimizer.
- A dev score can flatter
The method replays an optimizer’s acceptance rule with candidates left unchanged, asking how often selection alone can produce a gain like the reported development-set gain. In DeepSeek-V4.1-Flash SkillOpt and GEPA runs, that happened in 0.56 to 0.94 of replays.
This makes the development gains weak evidence of improvement in those runs, though the replay does not establish that selection explains every gain.
- Test success stayed near level
The decisive test comparison is learned skills against their shared hand-written seed on the same test tasks. Across all 16 skills, mean test success was only 0.1 percentage points above the seed.
This pooled average indicates little overall change, not that every skill stayed level; task coverage also shows gains and losses rather than a consistent expansion.
- More instructions cost more
The same task-by-task comparison found that learned skills used an average of 1.16 times the seed’s input tokens. Because each turn resends conversation history in this harness, extra turns can add reading cost even when success does not rise.
The cost result changes the tradeoff: a development-set gain alone does not show that an edit is worthwhile on test.
- Transfer remained weak
The gains also did not reliably carry across the tested settings. Under GPT-6-Luna, skills learned for simulated customer-service tasks scored 3.6 to 6.2 percentage points below the seed.
That transfer result narrows the conclusion: the measured skills’ near-level pooled test average does not imply their performance carries to another model.