A text-guided localizer can route attention by stereotypes
When Pointing Becomes Profiling: Social Bias and Refusal in Text-Steerable Visual Encoders · arXiv:2610.35017
In plain words
The paper studies text-guided image localizers, which point to people or objects named in a prompt. It asks whether prompts can steer a model by perceived race or gender when the picture gives no reason to choose, and whether the model can decline requests the image cannot answer.
The authors found that the released SteerViT localizer sometimes did this in tests. Their SafeSteer method added an abstention choice and reduced crime-linked selection of Black people in synthetic four-person scenes, though some such routing remained in crowded real photographs.
The research
If a model uses a prompt to point at a person, a request about criminality or another non-visual trait can force a choice the image cannot justify. Can the same system still locate visible people and roles, yet abstain when a prompt asks it to infer something the scene does not show?
The authors audit demographic routing in text-guided localization, then train SafeSteer to abstain on unanswerable prompts and share attention evenly when no person is visually favored.
The authors compared which person the released SteerViT pointed to in synthetic four-person mosaics, averaging rotated layouts. A Black person was its choice in a 9.2-percentage-point greater share of mosaics for crime than occupation prompts.
SafeSteer trains for abstention and evenly shared pointing when a scene does not distinguish candidates. It reduced that gap to 1.4 percentage points and almost always ranked should-refuse prompts above should-answer prompts, supporting both aims on these tests.
The routing reductions are reported on synthetic mosaics, and the paper does not state whether SafeSteer and released SteerViT share the same adapter initialization.
Some crime-to-Black routing remained in fully cluttered PHASE scenes, and the tested never-seen nationality and migration prompts were not refused.
What the evidence shows
- A protocol for visual routing
The paper’s counterfactual audit protocol for text-steerable localization is a test procedure for checking where a model points when language names a person or trait the image may not establish. It measures heatmap mass on people in synthetic mosaics and real multi-person photographs, then separately tests prompts that should be answered or refused.
Its race and gender labels describe perceived or dataset-assigned categories, not people’s self-identification.
- Prompts changed who was selected
SocialCounterfactuals is a synthetic set used to make four-person mosaics for comparing choices across prompts. In those mosaics, the released SteerViT text-guided image localizer selected a Black person 9.2 percentage points more often for crime than occupation prompts.
In separate four-women mosaics, “the man” selected the Black woman in 75.6% of sets, versus a 25% equal-selection chance rate.
- Controls locate the prompt effect
To check whether language associations alone explained the crime result, the authors compared frozen text embeddings for crime and occupation words against Black references. The standardized difference was d = −0.04, essentially no extra association for crime words.
They then held the image and model weights fixed while changing the text given to attention blocks that pass prompt information into image features. Giving the query to blocks 9 and 11 reproduced nearly all measured routing; substituting “the person” there removed most of it. This locates the effect in prompt processing by those steering layers in this setup.
- The released model could not abstain
The separate refusal test asked whether a score could distinguish prompt–mosaic examples that should be answered from those that should be refused. Its AUROC measures how often a randomly chosen should-refuse prompt–mosaic example receives a higher score than a should-answer example.
Released SteerViT’s heatmap-flatness score, one minus the largest share given to any person, scored 0.517, near chance. Because its heatmap must assign mass to image patches, the released model has no explicit way to abstain.
- SafeSteer addresses both failures
SafeSteer is the paper’s training method that adds an explicit abstention output to text-guided localization. It trains the model to spread attention evenly when an image gives no reason to prefer someone, to abstain for absent objects or non-visual attributes, and to preserve referring queries by learning from the released model’s heatmap.
Across three training seeds, its refusal score reached AUROC 0.997 ± 0.001; on synthetic mosaics, the crime-versus-occupation gap in Black-person selections fell from 9.2 to 1.4 percentage points.
- Transfer has clear boundaries
A refusal-only ablation performed well on refusal but left the measured routing near released-model levels. Since the paper does not establish that the ablation shared the same adapter initialization or all other training conditions, this is not a complete one-factor test.
On real PHASE photographs, SafeSteer distinguished non-visual identity prompts from visible-person prompts at AUROC 0.935. At a 0.5 threshold it refused 49% of identity photo-prompt pairs and 1.2% of groundable pairs; some tested unseen nationality and migration terms were still not refused.
Deep
- Original paper authors
- Seyedarmin Azizi, Parsa Razmara, Sajjad Ghiasvand, Ardalan Aryashad, Tina Khezresmaeilzadeh
- Submitted to arXiv
- License
- CC BY 4.0