A text-only test traces watermarked training into student models
Watermark Basins: Black-Box Detection of Distilled LLMs · arXiv:2610.37266
In plain words
A language model can be trained on another model’s answers. If those answers carry a hidden watermark, traces of it may remain in the student model, but existing text checks can miss it when they depend on repeated word patterns.
WM-Basin compares answers from the suspect and watermarked teacher, using a clean model to find shared patterns to remove. In one math-reasoning test, it detected more of the distilled models than the existing text check did; this result applies to the tested models and conditions.
The research
When a model is trained on a teacher’s watermarked answers, can a verifier detect that connection just by querying the model?
Existing text checks depend on watermark-favored tokens appearing in recurring contexts, and the paper reports that this signal weakens with larger context windows and lower-temperature sampling.
WM-Basin detects a watermarked teacher’s influence by comparing probe-model gradients from suspect and teacher text after removing patterns shared by clean models.
The verifier queries the suspect on test prompts and a watermarked teacher on separate build prompts. A fixed clean probe turns both sets of answers into gradients; the method removes patterns shared by clean models and ranks the claimed key against decoys.
In math-reasoning distillation from one watermarked teacher, it detected 45% of 1,200 models pooled across 24 student architectures at a nominal 0.1% false-positive rate and a three-token window; the token detector found 1.3%. This supports stronger detection in that tested pool.
WM-Basin tests text from the domain being audited. A low score on another domain does not show that a model is clean, and continued training on clean data can weaken the signal in its answers even while an open-weight test still finds it in the model’s parameters.
What the evidence shows
- A text-only detection method
Watermark Basin Alignment (WM-Basin) is a black-box method for detecting whether a suspect language model was distilled from a teacher’s watermarked text. It uses answers obtained through the suspect’s API, so the verifier does not need the suspect’s weights or internal scores.
The question it addresses is whether a trace of the teacher’s watermark can be found when token checks that rely on repeated contexts lose their signal.
- Separating task from watermark
The authors first examined the signal in model weights. They compared students trained from the same base model, teacher, prompts, and recipe, with watermarked or clean teacher traces.
Using the clean student’s update as the task direction, they found that detection evidence fell to the detector’s floor after only a few steps along the watermark direction, but persisted over much larger steps along the task direction. This suggests the watermark signal is narrower than ordinary task learning.
- Comparing the two detectors
WM-Basin represents suspect answers and keyed teacher answers as gradients from one fixed clean probe model. It removes directions shared by clean models, then compares the remaining alignment for the claimed key against decoy keys.
In a math-reasoning evaluation pooling 1,200 models across 24 student architectures, it detected 45% at a three-token window and a nominal 0.1% false-positive rate; the token detector detected 1.3%. The result is an aggregate for that tested pool.
- A tougher sampling condition
The low-temperature comparison tested up to 100 Qwen2.5-3B students per condition, distilled from the same teacher at different sampling temperatures. At temperature 0.5, the reference bank remained at temperature 1.0; with a three-token window and a 1% false-positive rate, WM-Basin flagged about half the students while the token detector flagged none.
The result shows that WM-Basin retained evidence in this mismatch condition, not that it will do so for every setting.
- Where the evidence stops
Audits on math, code, and chat showed that detection followed the domain used for distillation: low scores in another domain did not establish that a model was clean. The authors also found that continued training on clean data could weaken the watermark signal in generated answers even when an open-weight detector still found it in the parameters. These findings bound what a text-only audit can establish.
Deep
- Original paper authors
- Hongyan Chang, Hady Elsahar, Tom Sander, Pierre Fernandez, Saeed Mahloujifar
- Submitted to arXiv
- License
- CC BY 4.0