OpenTWBench tests Taiwan knowledge and exposes score effects

How can an evaluation score reflect models’ knowledge of Taiwan’s local exams and everyday life when many tests are translated from English or built for Simplified Chinese, and Taiwan-native tests are small and increasingly saturated? The paper also asks how much of a reported score comes from the way answers are measured.

OpenTWBench pairs Taiwan national-exam and everyday-knowledge tests, while measuring how answer-format rules can change reported scores.

OpenTWBench pairs national-exam questions with checked questions about everyday and contemporary Taiwan. For the latter, a model drafts questions from a community corpus, and a judge checks each keyed answer against source text.

In one civics evaluation, the same model and questions scored 20.8% when software searched free-form replies for an option letter, versus 93.9% when asked for a clearly marked answer and scored from that mark. The gap shows that reported accuracy can reflect how answers are read as well as what the model knows.

The transfer check used one model outside the selection pool, and the difficulty band was calibrated to seven models frozen in August 2026. The paper reports that later frontier models exceeded the band’s upper range.