Small image changes can steer preference training

Can preference training be redirected by examples that still look correctly labeled to a human reviewer?

The paper investigates whether an attacker can alter a vision-language model's learned responses by changing only contributed training images, leaving prompts, responses, and preference annotations intact.

DPOison-VL changes only contributed training images, using gradient-guided crafting and ranking to steer vision-language preference alignment.

The researchers tested whether this attack could change a model's answers on images kept out of the crafting process. They trained four VLMs on three misidentification tasks, then used a language-model judge to assess generated responses on held-out target images.

For Qwen3-VL-2B on a traffic-light task, the requested wrong color appeared for 98.0% of held-out target images with 2% poisoned preference data, averaged across three runs.

This shows that image-only changes can steer responses in a tested setting while the contributed preference labels remain correct.

The experiments cover four vision-language models, three misidentification tasks, and standard DPO training. The theoretical result depends on smoothness and gradient-descent assumptions, so neither the experiments nor the theorem establish the same effect for other training methods or settings.