Similar coding skills can hide the behavior users installed
In plain words
Coding agents can use added skills: sets of instructions for doing a job. The authors studied what happens when a user’s installed skill is joined by another skill that does the same job.
They tested coding tasks with one skill available and with both available, checking task completion and whether the installed skill’s distinctive requirements were met.
In Claude Code, tasks still got completed, but the distinctive requirements were fulfilled less often when the similar skill was also available. A passing task result alone could therefore miss a change in which instructions shaped the work.
The research
When a coding agent completes a task, how can a user tell whether it followed the distinctive instructions of the skill they installed? Similar skills that do the same job may be present together, so task completion alone may not answer that question.
The authors compare runs with one installed skill against runs with a similar skill added, measuring whether the first skill’s distinct instructions are still followed.
In Claude Code, the authors ran the same requests and starting repositories with skill A alone and with A plus similar skill B, across 312 pairs and three models. They checked task completion separately from how many of A’s applicable core functions that B lacked appeared in the result.
The pooled comparison found 5.6 percentage points lower fulfillment of those exclusive functions with B present, while estimated task completion did not fall. A passing task result can therefore miss a change in which installed instructions shaped the work.
The full design was run on Claude Code, with most model, configuration, and pair combinations run once despite some repeat checks. The first-read comparison was observational, and the Codex check covered only 193 pairs, so these results do not establish the same effect across platforms or runs.
Analysis
Deep
- Authors
- Chaoliang Yan, Zihao Xu, Yuekang Li, Shangzhi Xu, Yi Liu
- License
- arXiv non-exclusive