Identity matching weakens when names switch scripts

If a model is asked whether two names refer to the same person, should its answer change just because the names are written in different scripts? The paper tests this directly by supplying both names and holding the people, relation and question fixed while changing their script-associated forms.

The authors compare fixed identity pairs across scripts, then test how Qwen’s answer cutoff and internal states relate to its missed matches.

Using 601 audited name pairs, the authors compared Qwen’s answers when the same identities appeared in Devanagari or Latin, keeping the people, question and Marathi prompt fixed.

Same-person accuracy was 8.5% for two Devanagari names and 35.1% for two Latin names, while different-person accuracy stayed at least 98.6%.

On held-out Marathi examples, adding a constant to Qwen’s answer margin raised balanced accuracy, which weights both pair types equally, from 0.525 to 0.879; a separate probe reached 0.933.

This suggests cutoff placement explains much of the failure, though the probe’s remaining advantage leaves other factors open.

The detailed causal analysis is for Qwen, and the patching comparison rests on 12 held-out Marathi items selected because each failed in Dev–Dev but succeeded with Latin-bearing donor names across paraphrases and both answer-key orders. The intervention controls reuse the same test split, so they are not independent replications.