Copied news can sway some language-model judges
Copies Are Not Confirmations: How News Duplication Fools LLM Judges and Continual Learners · arXiv:2611.09203
In plain words
A claim may appear in many news articles even though they all trace back to one report. The paper measures this copying, tests whether four language models mistake copies for separate support, and tries ways to keep copies from influencing a learning system.
In a simulated stream, showing a judge one report from each detected copy group reduced accepted copied lies for all four models. A separate test could catch many model-written rewrites, but it missed most natural articles derived from the same source.
The research
Repeated coverage can make one report look like several confirmations. The results matter for deciding how much weight to give repeated articles: in this study, some language-model judges accepted more invented lies when shown identical reports, and filtering detected copies reduced that effect in a simulated learning stream.
The paper measures news copying, tests how it sways language-model judges, and finds that removing detected copies can reduce accepted lies in a simulated learning stream.
The authors first measured cross-outlet near-copies, then compared judges’ verdicts when they saw one report or three identical copies.
Two judges accepted substantially more invented lies with three reports; responses differed across models, and Gemini Flash already accepted most lies from one report.
In a simulated learning stream, showing one report per detected copy group reduced accepted copied lies for every tested judge. The results support copy removal as a way to limit this form of influence, while showing that simple report-count rules can also reject many true claims.
The learning results come from simulated streams with modeled detector errors, not a pipeline processing real news articles. The entity test caught few natural same-source derivatives and does not detect coordinated sources that avoid copying.
What the evidence shows
- Copies can look like support
The paper’s contribution is a measurement of news copying, controlled tests of four language-model judges, and a simulated test of copy removal in continual learning. The concern is that many articles can repeat a claim while deriving from one report, so counting articles as separate support may give a judge or learner a misleading impression of confirmation. Copying alone does not show whether a claim is true or whether outlets are independent.
- Repeated reports sway some judges
The same repeated lies about invented people affected judges differently: in prompts using invented outlets, Llama 3.2 and phi4 accepted them more often when shown three identical reports, while Gemini Flash already accepted nearly all with one report.
Gemini Pro’s increase was not statistically significant, so this controlled test does not show that copies sway every judge.
- Remove copies before judging
Compared with showing every report, limiting judges to the earliest report from each detected copy group reduced accepted copied lies across all four judges, with a small average cost in accepted truths.
A simple three-report threshold performed poorly: it accepted every copied lie but only a quarter of true claims.
- A second signal catches rewrites
Text similarity caught edited reprints but missed the tested model-written rewrites, motivating a test based on how many names and quotations a report adds.
That test distinguished phi4 rewrites from independent reports of the same event well, but caught only 17% of natural reports derived from the same source.
Deep
- Original paper authors
- Abdelkader Metales, Miriam Grabenheinrich, Aicha Metales
- Submitted to arXiv
- License
- Not specified by source