A tutor's higher fit score came with lower educator ratings

If a tutor is trained to improve a score for matching instruction to a learner, does that make its plans better in educators' eyes?

The study tests this question by comparing ratings of original and corrected plans, while also checking whether the score captures variation across a sequence of activities.

The study creates a score for instructional fit, then shows that training a proxy to maximize it can raise the score while producing repeated choices and lower educator ratings.

Can training a tutor to score better for instructional fit also improve the instruction educators see? The study introduces the PAI, which scores choices against a learner's described situation, and uses it to correct a proxy tutor.

On 30 intervention cases, the score rose, but educators rated the corrected plans lower than the original plans.

The corrected plans also repeated the same choice across five activities. Since the PAI scores choices independently against fixed case conditions, this pattern can raise its average without measuring variation across a sequence.

Ratings of unchanged control plans did not shift significantly, which weighs against general rating drift as the explanation.

The educator ratings were for 30 cases that were also used in corrective training, and the original and corrected plans came from different models.

The corrected descriptions also changed, and educators did not rate the un-fine-tuned proxy, so the study cannot isolate which changes caused the rating decline or show that the same pattern holds with real learners or other models.