In plain words
Entity alignment systems decide whether entries in two knowledge graphs refer to the same thing. Checking those decisions by hand takes expert time, so this paper tests whether language models can judge them reliably.
The authors found that when judges saw a system’s answer, their scores often failed to separate correct decisions from incorrect ones. Changing the displayed answer while keeping the evidence fixed often changed the score, too. These results cover three tested language models and entity alignment data.
The research
When an entity alignment system says two entries match, an evaluator must decide whether the evidence supports that decision.
Expert checking is expensive, so language model judging is attractive; the unresolved question is whether a judge can assess the evidence independently when it can see the system’s answer.
The authors test language model judges for entity alignment and find that visible system labels can distort scores of whether decisions are justified.
The authors compared three ways to evaluate entity alignment decisions: two prompts showed the judge the system’s decision, while a third asked the language model to predict alignment without seeing it.
They tested three judges across entity alignment datasets and systems, including a biomedical dataset they release.
On DBpedia–Wikidata pairs, scores under the label-visible prompts often failed to rank correct decisions above incorrect ones. J-ROC-AUC measures the probability of that ranking, and the tested values ranged from 0.12 to 0.53.
This supports the concern that visible decisions can undermine judging; label-free prediction worked strongly for some systems and datasets, but not all.
The results cover three closed-source language models and entity alignment tasks. The label-free prompt also changes the task from judging a system decision to predicting alignment, so that comparison cannot isolate label removal alone.