Code context helps some vulnerability-finding models more
How Much Does the Harness Matter, and for Whom? A Multi-Model Study of Context in LLM-Driven Vulnerability Discovery · arXiv:2610.28802
In plain words
The paper studies whether the tools and code information given to AI agents change how well they find security problems in real software. DeepSec first picks files to inspect, then has an agent investigate them and review its own reports. Researchers also check retained findings.
Across 24 model-and-tool setups, no single setup found all the confirmed problems seen across the study.
In a separate four-model test, combining an engineered prompt with code context helped some models more than others compared with a minimal prompt; the strongest model’s average finding count did not rise.
The research
When an AI agent searches real software for security flaws, how much do its instructions and added code information matter, and does the answer depend on which model is doing the search? This matters to interpreting findings because results can reflect the model and the surrounding tools together.
The authors compare DeepSec configurations and prompts, finding that engineered guidance plus code context raises confirmed security yield for some models but not others.
The authors compared each of four selected models using a minimal prompt, an engineered security prompt, and the engineered prompt plus CodeQL-derived code context. They kept the file-screening and review stages fixed and averaged two runs per condition.
The combined prompt-and-context condition raised mean confirmed security findings for three models and left GPT-5.5 unchanged. A staged comparison showed that the prompt alone improved yield for all four, while adding context helped only Gemma.
This indicates that the combined gain depends on the model, not that context alone helps every model.
The context study tested four models with two runs per condition, so its effect estimates are approximate. The headline comparison changes both the prompt and added code context; the staged comparison shows that context alone raised yield only for Gemma in this experiment.
The main evaluation also centers on one codebase, with a separate test offering limited cross-language corroboration.
What the evidence shows
- The question and system
DeepSec is an agent-based vulnerability-discovery harness: it screens files, has an AI agent investigate candidates, and asks the same model to recheck its reports. Researchers then group matching reports and review retained findings.
The study asks how model-and-harness configurations perform on production code with vulnerabilities that were not public at scan time, and whether adding code context changes results differently across models.
- One setup leaves gaps
In the multi-model study, 24 model–runner configurations shared the DeepSec pipeline and target codebase, while their base harnesses and runners differed. Those differences confound cross-configuration comparisons.
Researchers counted 265 distinct confirmed security root causes observed across the configurations’ two runs. The strongest two-run configuration found 160, or 60.4%, of that observed union, while 38 root causes appeared in exactly one model’s set.
This means the union included findings missed by the strongest configuration; it does not count every vulnerability in the codebase.
- Context effects vary
The context study compared four selected models against their own minimal-prompt baselines, using two-run means. The condition combining DeepSec’s engineered security prompt with CodeQL-derived code context raised average confirmed findings for Gemma, GLM-5.1, and Composer-2.5.
GPT-5.5’s average was unchanged. A severity-weighted score also rose more for Gemma, GLM-5.1, and Composer-2.5, while GPT-5.5’s small change was within run-to-run noise.
- Separating the changes
A staged comparison helps interpret the combined result. Adding the engineered prompt alone raised mean security yield over the minimal prompt for all four models.
Adding CodeQL context on top of that prompt raised yield only for Gemma and lowered it for the other three in this experiment. The headline comparison therefore cannot assign its changes to context alone.
The authors suggest context may supply useful code-path information to some models, but this proposed explanation was not directly tested.
- A second code setting
The authors checked the direction of the context-associated change on 21 known-vulnerable C/C++ file instances from CyberGym Level-0, a test of whether systems find known upstream vulnerabilities in their vulnerable files.
All four models found the vulnerability in more instances with context than with the base harness. This offers corroboration in another language and codebase, but the test uses a public, patched corpus and does not establish performance on undisclosed flaws generally.
Deep
- Original paper authors
- Ganni Galea Curmi, Ben Wilde
- Submitted to arXiv
- License
- CC BY 4.0