A probe reads vulnerability signals inside language models
Reva: Real-Time Vulnerability Detection through Contextual Probing of Large Language Models · arXiv:2610.35829
In plain words
Reva is a method for finding vulnerable software functions. It gives a language model the function and prepared security context, then uses a small trained probe to read the model’s internal signals. The model does not have to write a yes-or-no answer.
On a test of paired vulnerable and benign C/C++ functions, Reva matched one multi-step detector and did better than another using the same model. Results varied across the three tested models.
The research
When software is generated quickly, a detector that takes minutes per function may struggle to keep pace, while faster approaches are described as less accurate. Can prepared security context and a different way of reading a language model retain useful vulnerability signals in a single pass?
Reva prepares security context and uses a trained probe to read a language model’s internal signals, matching or exceeding multi-call detectors on PrimeVul tests.
On PrimeVul’s held-out test set, Reva and two multi-call detectors were compared using the same named language model. Reva labeled 60.0% of functions correctly with GLM-5.3-Flash, close to VulTriage’s 59.2% and above VulAgent’s 52.8%; the small gap from VulTriage is uncertain. With two other tested models, Reva was more accurate than both detectors.
Because the model generated no verdict for Reva, a validation comparison tested whether its own Yes/No answer could provide the same signal.
On Muse-Glimmer-30B, that direct answer reached chance-level accuracy while the probe did better, supporting the interpretation that useful vulnerability information can be present in internal representations.
These results come from PrimeVul, a dataset of paired vulnerable and benign C/C++ functions. The paper does not establish how Reva performs on other datasets or on vulnerabilities that depend on interactions between functions.
What the evidence shows
- The speed and accuracy problem
Reva (REal-time Vulnerability Analysis) is a method for detecting vulnerable software functions. The paper addresses a speed and accuracy tension: detectors that gather context through multiple model calls can take minutes per function, while faster approaches are described as less accurate.
Reva’s central move is to prepare security context in advance and read a model’s internal representations instead of waiting for a generated verdict.
- A different way to read code
Reva combines each function with a project threat model and static-analysis findings, adding repository information when available. The language model processes this prompt once; a trained probe reads its internal representations and assigns a risk score.
On PrimeVul, a dataset of vulnerable C/C++ functions paired with benign versions, the held-out test set contains 435 pairs. Accuracy measures the share of all functions labeled correctly, with each function scored independently.
- How the comparisons came out
The authors compare Reva with VulTriage and VulAgent on the same named language model and the same held-out functions. With GLM-5.3-Flash, Reva labels 60.0% correctly, versus 59.2% for VulTriage and 52.8% for VulAgent.
The small difference from VulTriage is uncertain because its confidence interval includes zero; Reva’s 7.2-point lead over VulAgent does not. With two other tested models, Reva exceeds both detectors by 5.1 to 6.7 points, with reported intervals excluding zero.
- Testing the source of the signal
A validation test with Muse-Glimmer-30B helps distinguish the probe’s signal from the model’s own answer. Given the same prompt, asking the model directly for a Yes/No verdict yields chance-level accuracy, while the probe performs better.
The authors interpret this as evidence that the model’s internal representations can carry vulnerability information that its generated verdict does not express. This diagnostic covers one model and the validation set, so it supports a bounded interpretation.
- What remains unestablished
The evidence supports Reva’s performance on PrimeVul and the three tested open-weight models. In a separate cross-validation check, no project appeared in both training and evaluation folds, and accuracy fell by only 1.3 to 2.7 points; the authors interpret this as evidence that most of what the probe learns transfers to unseen projects.
This check does not erase the six-point gap between GLM-5.3-Flash’s cross-validation and chronological test-set accuracy, which the authors suggest may reflect newer code in the test split. Performance on other datasets and on interprocedural vulnerabilities, which can depend on interactions between functions, remains unestablished; the authors list both as future work.
Deep
- Original paper authors
- Amine Lbath, Massih-Reza Amini
- Submitted to arXiv
- License
- CC BY 4.0