EN
Gradient filtering removes risky training examples
Follow the Gradient: Tracing Safety Degradation Back to Training Data · arXiv:2610.37481
In plain words
Some instruction examples can look harmless but still weaken a language model’s safety after training. This paper introduces SimpleLoRIF, a method that compares the model updates an example would cause with updates linked to unsafe answers.
The authors used it to remove the 20% of Tulu 3 examples most aligned with unsafe responses, creating SafeTulu.
In tests of three model families, training on SafeTulu led to fewer harmful answers to one kind of optimized attack than training on the full dataset. Results on capability tests were usually close, but varied by test and model.
The research
When instruction examples look harmless, what can reveal that some may still weaken a model’s safety? The authors ask whether the model updates those examples induce can identify a connection to unsafe behavior, offering a way to select training data by its effect rather than its wording.
SimpleLoRIF ranks training examples by their gradient similarity to unsafe responses, and the authors use it to filter Tulu 3 into SafeTulu.
The authors tested whether examples’ effects on model updates could reveal links to unsafe behavior even when the examples looked benign.
In an insecure-code analysis, scores separated matched insecure and secure examples above chance for both tested model families, though the signal varied by family and model state.
They used the method to remove the 20% of Tulu 3 examples most aligned with unsafe SafeRLHF responses. Across three model families, mean success on an optimized harmful-prompt attack fell compared with full-data training, while capability results were usually close. This supports the filtering approach in those evaluations.
The SafeTulu tests cover general safety degradation using unsafe responses from one target set, SafeRLHF; they do not test specific behaviors such as sycophancy or other target sets. The observed patterns in removed examples are correlations, not proof that those features cause safety degradation.
What the evidence shows
- Look beyond example wording
SimpleLoRIF is a gradient embedding method: it represents a training example by a compact approximation of the model update that example would cause. The authors use similarity to unsafe responses to rank examples for removal.
This addresses a problem that content filters can miss: safety-degrading examples may look benign even when their effect on the model is not.
- Test the signal on code
The authors first tested whether gradient similarity separated insecure-code examples from closely matched secure-code examples. They used responses from fine-tuned Qwen and Mistral models that Claude 5 Opus judged to show broad misaligned behavior.
The authors’ ROC-AUC measure asks how often a randomly selected insecure-code example scores above a randomly selected secure-code example; results were above chance for both families, and stronger for Mistral.
- Check state and transfer
A control found that the Mistral score was 0.431 before insecure-code training and 0.865 afterward. A transfer test also showed an uneven result: Qwen embeddings separated the code examples less well when compared with unsafe Mistral responses instead of Qwen responses.
These comparisons qualify the diagnostic: the signal depends on model state and does not transfer equally across families.
- Evaluate the filtered data
The authors then used SimpleLoRIF to create SafeTulu by removing 20% of Tulu 3 examples most aligned with unsafe SafeRLHF responses. They trained three model families on the filtered data and compared them with models trained on full Tulu 3.
In GCG tests, where an optimized text suffix attempts to elicit unsafe answers, mean attack success was lower for all three families; capability results were usually close, with some differences by test and model.
- Bound the conclusion
Together, the tests support using gradient similarity to find examples associated with safety degradation and show that filtering can reduce measured attack success in the tested models. The authors also found recurring patterns among removed examples, including tool-execution failures and narrative arithmetic, but describe these as suggestive correlations rather than causes. The evaluation covers general safety with one target set, not every harmful behavior or target set.
Deep
- Original paper authors
- Saket Reddy, Amro Abbas, Anshuman Suri, Ari Morcos, David Schwab
- Submitted to arXiv
- License
- arXiv non-exclusive