Chiosa
Machine Learning

Jailbreak rankings shift with evaluators and attempt settings

2 min
Updated

In plain words

This paper studies why tests of jailbreaks, which try to get a language model to answer harmful requests, can give different results. The authors tested eight methods and compared three ways to judge answers.

One rule looked only for refusal wording; another checked whether an answer refused, was specific, and was convincing.

For one attack, the first rule counted almost all tested cases as successes, while the more detailed evaluator counted very few. The study shows that rankings depend on how success is judged and on how many chances the model gets to answer. It does not establish which evaluator is right.

The research

When one jailbreak attack is said to work better than another, what does that ranking actually tell us? A test counts requests according to a success rule and the answers generated under a particular protocol, so changing either could change the result. This study measures how large those shifts can be in its tested cases.

The authors compare jailbreak evaluations and find that both the rule for judging success and the number and arrangement of attempts can change attack rankings.

Across the same 150 request-model cases, the low-resource-language attack had a 96.0% success rate under keyword matching and 2.7% under SR-Q. Its rank shifted from first to last among eight attacks, showing that the evaluation rule changes what the comparison says.

A separate Vicuna follow-up changed two attacks’ response protocols and reversed their ordering. Together, these results show that reported rankings depend on how answers are judged and generated.

These are observed results from three small open-weight models and a 50-request subset. The full pipeline was not repeated, SR-Q was not independently validated in this setting, and evaluator scores do not measure realized harm; the findings do not establish general rankings or safety guarantees.

Analysis

Deep

Original paperConcerns in Benchmarking Jailbreaks: An Empirical Study of Attack Success RatesOpen original
Submitted to arXiv