Small adapters improve brain-response predictions across tested models

Pretrained models can extract-rich patterns from audiovisual input, but brain-encoding approaches often leave those representations fixed and train only a readout. Fully adapting a large model is difficult when neural recordings are scarce.

Can a smaller adjustment let brain data specialize the model’s representations for predicting responses?

BrainLoRA trains small adapters alongside a readout while keeping a pretrained model’s main weights fixed, allowing neural data to specialize its representations for brain-response prediction.

The question is whether scarce brain recordings can specialize a large pretrained model for predicting brain responses. Keeping all model weights fixed avoids that adaptation, while changing all of them is difficult with limited neural data.

BrainLoRA tests whether training a small adapter can make the model’s representations more useful for prediction.

Across nine multimodal models, each adapted version scored higher than its paired frozen-backbone condition on Friends season six. The score was mean Pearson correlation across four people, measuring how closely predicted fMRI responses tracked recordings.

For Qwen3-Omni, it rose from 0.2907 to 0.3263. This supports an adaptation benefit in the tested configurations.

The experiments trained on one benchmark with four people watching audiovisual narrative and tested transfer only within the narrative-film domain. The results do not establish performance for other kinds of stimuli or populations.