Text personas rarely make vision models answer like viewers
In plain words
A vision-language model can answer questions about pictures. This paper tests whether adding a short description of a viewer, such as their eyesight or hearing, makes the model answer like that person.
Researchers compare those answers with recorded people’s answers and with simpler rules based on facts in the description.
Across reading, colour and subtitle tasks, personas often failed to match individual or group differences; describing blur also did not reduce object recognition as a blurred picture did.
The research
A persona can sound informed about a viewer while still giving the model its usual answer. That matters because a match on average can hide differences between people, and a guessed group answer cannot show which individuals struggle.
In several measured tasks here, persona answers failed those checks, so they cannot be treated as a substitute for those people’s answers.
The evaluation separates persona answers from altered-image answers and group estimates, then checks whether personas match measured viewers and task-specific rules.
In the reading task, three frontier models used each reader’s persona to copy sentences at different print sizes.
Their predicted reading limits did no better than copying the stated letter-chart acuity, while a regression using three stated facts tracked reader differences; that gap suggests personas were not using those facts to simulate individual reading.
Other comparisons tested colour-sorting rules, subtitle choices and responses to changed images. Together, they show that a description often supplies facts or a group-level expectation without reproducing how a particular viewer responds.
The studies used short template personas and, in reading and subtitle tasks, model inputs that differed from what people saw. The blur task was a controlled proxy with four sighted observers, not a clinical low-vision test.
What the evidence shows
- Separate answers from estimates
The paper’s evaluation method for vision-language model personas separates three things: a model answering with a viewer description, answering after the image itself is changed, and estimating how a group answered. Researchers compare persona answers with measured people and with task-specific rules-based only on facts in the persona.
This distinction matters because a model can repeat what it knows about a group without answering as an individual in that group.
- Reading facts did not simulate readers
The reading comparison asks whether personas capture how individual readers depart from their stated letter-chart acuity. Model predictions tracked those departures weakly (correlations at most 0.20, with intervals including zero), while a regression using the persona’s three stated facts reached 0.38.
This suggests the models mostly used the stated acuity but did not reflect how individual readers’ reading limits departed from it.
- Colour rules beat personas
The colour-sorting comparison asks whether personas can reproduce the cap-order pattern associated with each colour-vision type. Persona answers fell short of the textbook rule based on the stated type, while changing the image colours to match that type produced much stronger results.
The published data provide group scores, so the comparison does not show how individual viewers sorted the caps.
- Subtitle differences disappeared
The subtitle comparison tests both whether a persona brings a model closer to one viewer’s line-break choices and whether it preserves differences between viewer groups. Adding a persona barely changed agreement with individual choices, and the model answers erased the measured preference difference between deaf or hard-of-hearing and English viewers.
The results show that these personas did not preserve individual or group differences in the tested subtitle tasks.
- Blur and illusion tests separate abilities
The blur comparison helps explain the gap: describing blur left recognition unchanged in all 14 tested models, while applying blur to photos lowered accuracy. Yet on visual illusions, models’ estimates of adults’ answers captured more of the human illusion effect than their own answers did.
That contrast shows why a group estimate and an answer as an individual viewer must be judged separately.
Deep
- Original paper authors
- Jiawei Chen
- Submitted to arXiv
- License
- CC BY 4.0