A Brazilian Portuguese model adds a tested safety step
In plain words
The paper introduces Manacá-1B-Instruct-2, a language model made for Brazilian Portuguese, and tests how it handles harmful requests. Its safety training teaches it to refuse those requests while also including examples of helpful answers and general language tasks.
On 218 safety prompts, it refused more often than four other tested models given the same prompts. A manual check still found one harmful answer, so the result shows stronger refusal in this test, not perfect safety.
The research
When an open Brazilian Portuguese model is called safety-aligned, how can readers tell whether it refuses harmful requests in practice, rather than merely looking successful in a training metric or a small probe?
The paper takes up that question by reporting failed approaches and testing both model behavior and the evaluation process.
The authors add a dedicated safety-training stage to Manacá-1B-Instruct-2 and test its refusals against other Brazilian Portuguese models on shared prompts.
The authors compared Manacá-1B-Instruct-2 v4 with four external Brazilian Portuguese models using the same evaluation setup and 218 held-out safety items.
V4 refused 97.2% of items, and its 95% interval sat above the upper interval for each comparison model. This supports a higher measured refusal rate on the shared prompts.
This is a refusal-rate result on 218 prompts, not proof of complete safety: a manual audit found one harmful answer among 218 responses.
The four comparison models had different training histories and pretraining budgets, so the shared test does not show that safety training alone caused the difference.
Analysis
Deep
- Original paper authors
- Bruno Leonardo Santos Menezes, Carlos Leonardo Souza Cardoso, Fabio André Machado Porto
- Submitted to arXiv
- License
- CC BY 4.0