In plain words
A retrieval-augmented generation (RAG) system looks up documents and gives them to a language model to help answer a question. A document could contain hidden instructions that try to change what the model does, while similar wording might be ordinary guidance.
This paper introduces a small local checker that reads the question and document together. In tests with the same command placed in harmless and attack contexts, it handled both sides correctly more often than a checker that saw only the document. That is evidence for these cases, not every possible attack.
The research
When a system fetches documents to answer a question, those documents can contain instructions that try to redirect it. Yet instructional sentences also appear in harmless manuals.
The unresolved issue is whether a small local detector can use the user’s request to distinguish the two, rather than flagging by wording alone.
The paper introduces a small local detector that judges retrieved instructions against the user’s query before a RAG system generates an answer.
The student was trained on examples checked by an independent judge. On 659 held-out records, it caught 99.4% of malicious examples, compared with 11.6% for the same small model without training, while falsely flagging 2.6% of benign examples.
This measures detection, not what a later answer generator would do. A separate test reused each sentence across benign and injected contexts.
The detector got both labels right in 75% of 120 pairs, versus 54.2% for a classifier reading context alone.
Most results come from synthetic, same-domain tests; adaptive attacks and downstream answer generation were not tested. On PoisonedRAG, a test of passages with fabricated facts, the tuned detector flagged only 0.7% of malicious pairs, so it does not address factual poisoning.
Analysis
Deep
- Original paper authors
- Abhishek Rai
- Submitted to arXiv
- License
- CC BY 4.0