Checked Skills can still combine into policy violations
CoSA: The Conflict of Skills Attack in Real-World LLM Agent Ecosystems · arXiv:2610.35082
In plain words
An agent can use Skills, which provide instructions or operations for a task. Even when each Skill passes its own security check, information or decisions passed between Skills can still contribute to a policy violation.
The paper introduces CoSA, a method that adds a limited set of checked Skills designed to interact with existing ones. Across fixed tasks, violations were more common with these additions than with original Skill pools, averaged over the tested models and agent setups.
The research
If every Skill passes its own package-level security checks, can a workflow still violate its task policy when one Skill passes information or decisions to another?
The paper asks whether such cross-Skill reuse can contribute to concrete violations, and whether carefully selected additions can expose that risk when agents choose Skills autonomously.
CoSA constructs a limited set of checked Skills to expose risky interactions; paired task runs show higher judged violation rates with augmented pools.
CoSA adds a limited number of Skills that pass package-level checks but are selected to interact with operations in an existing pool.
To test whether those additions change outcomes when agents choose Skills themselves, the authors compared original and augmented pools on the same fixed requests and initial states.
Across 4,320 task runs per condition, an LLM judge found concrete policy violations in 35.02% of runs with original pools and 58.45% with CoSA additions.
This pooled increase supports the claim that the additions raised violations in the tested grid; it does not measure the chance of failure on repeated runs of a single task.
The main result is an aggregate across fixed tasks, five models, and three agent harnesses, with each task run once per condition. It does not show how often agents would select these additions in changing real-world settings, and the OpenClaw guard result covers only recruiting tasks.
What the evidence shows
- Where individual checks stop
The Conflict of Skills Attack (CoSA) is a method for constructing limited additions to an agent’s Skill pool. It addresses a specific gap: package-level checks assess Skills individually, but a task can involve one Skill passing information or a decision to another.
The paper counts a conflict only when evidence links reuse across at least two admitted Skills to a concrete policy violation; a sequence of Skills alone is not enough.
- Patterns in Skill interactions
To identify patterns for constructing additions, the authors first ran controlled executions using real-world Skills and matched single-Skill controls. They report four recurring mechanisms: security context can be lost, interfaces can connect through meaning, assessment signals can appear authoritative, and target-action bindings can drift.
These patterns inform CoSA’s candidate construction, but their discovery does not by itself establish how often they occur when agents select Skills autonomously.
- From patterns to additions
CoSA proposes additions that either supply a missing business operation or reshape how existing operations exchange information, reuse decisions, or bind actions to targets. It checks candidate operations and handoffs against documented contracts, requires candidates to pass package-level checks, and selects a limited set to cover validated opportunities.
In a separate diagnostic, selected multi-Skill compositions produced violations in 41 of 60 requests, versus 5 of 60 matched single-Skill controls; both completed 55 tasks.
- Testing autonomous Skill choice
The main evaluation used CoSA-Bench, a set of 288 fixed business tasks across four domains and three policy-risk categories. Five language models and three agent harnesses ran each task once with the original pool and once with the augmented pool, while choosing Skills and their order autonomously.
An LLM judge counted a violation only when the execution record showed a concrete forbidden effect. Across this evaluation grid, the judged violation rate rose from 35.02% to 58.45%.
- Controls and a guard
OpenClaw controls help bound what explains the increase: random benign additions had a 26.39% violation rate, random selection from CoSA’s checked candidates had 43.40%, and full CoSA had 54.24%. These controls suggest that pool size or random selection alone does not reproduce the full result, though each random baseline used one sample.
A separate recruiting evaluation found fewer violations with CoSA-Guard, a harness-level check, while some violations remained.
Deep
- Original paper authors
- Yujia Zheng, Tianhao Li, Shangzhi Xu, Yuekai Huang, Ziyou Jiang
- Submitted to arXiv
- License
- CC BY 4.0