Simulation training improved psychiatric intake in controlled tests
Training language models in simulation for psychiatric intake · arXiv:2610.40476
In plain words
Psychiatric intake requires clinicians to ask about difficult topics while building enough trust for patients to share important details.
This paper presents a method that trains a language model in simulated psychiatric conversations: a patient simulator responds to the model’s questions, and a separate evaluator scores each encounter to guide training.
In text consultations where psychiatrists role-played patients, psychiatrists rated the trained model’s transcripts higher than its base model in 46 of 60 matched scenarios. These controlled tests do not show how the system performs with real patients.
The research
Psychiatric intake depends on both asking the right questions and building enough trust for patients to disclose difficult details. Could simulated conversations help a language model improve at both, and would any improvement carry over to encounters with people role-playing patients?
The method trains a language model through scored simulated interviews, then tests whether its intake skills carry over to psychiatrist-role-played consultations.
In blinded text consultations, psychiatrists role-played patients in 60 matched scenario pairs, and a separate panel rated transcripts. The trained model received the higher aggregate rating in 46 pairs, supporting an advantage in this controlled role-play setting.
On 127 expert-validated simulated intakes, an automated evaluator also gave the trained model a higher mean clinical rubric score than its base model.
But the measured gap grew when the evaluation switched to simulated patients and a different interface, so simulated scores alone may overstate transfer to human-patient encounters.
These were controlled, single-encounter English text tests using simulated or psychiatrist-role-played patients. They do not establish outcomes or safety with real patients or readiness for autonomous psychiatric assessment.
What the evidence shows
- The intake challenge
Psychiatric intake joins two demands: clinicians need to ask systematic questions that uncover important history, and they need to build trust so patients will share it. The authors asked whether language models could improve at both through simulated encounters.
They present a training method in which a model conducts psychiatric intake conversations in simulation and is evaluated afterward.
- Training through interaction
In the training method, a simulated patient changes its mood and willingness to disclose information in response to the model’s interaction. After each encounter, a separate language-model evaluator scores the conversation and clinical outputs against clinical and communication criteria, including safety penalties.
Those scores guide training updates. The design rewards both inquiry and communication rather than treating diagnosis or empathy as the only goal.
- Evidence beyond simulation
To assess performance beyond simulation, the authors compared the trained model with its base model in a blinded text study. Board-certified psychiatrists role-played patients in 60 matched scenario pairs, while a separate panel of psychiatrists scored the transcripts.
The trained model received the higher aggregate rating in 46 pairs. This is evidence of an advantage in psychiatrist-role-played scenarios, not a result from real clinical encounters.
- Gains with limits
Automated scoring across 127 expert-validated simulated intakes also favored the trained model, but the evaluation setting mattered: with the same automated evaluator, the measured score gap was larger with simulated patients than with psychiatrist-role-played patients, and the interface changed too.
Safety tests found less acute under-triage than for the base model, while many intakes still omitted at least one critical history topic. The evidence supports improvement in controlled tests, with important gaps remaining.
Deep
- Original paper authors
- Samuel Schmidgall, Tim Strother, Alex Bijamov, Min Woo Sun, Anil Palepu
- Submitted to arXiv
- License
- CC BY 4.0