Same-size students struggle to match model harnesses

A model can gain accuracy from an outside procedure that searches, backtracks, or coordinates repeated calls. When can training transfer that gain into the model so it works without the procedure, and when does the procedure compute something the model cannot reproduce at the same capacity?

The authors train students on a harness’s traces and find that matching its gains can require more capacity or an easier task.

To test whether harness gains can be learned, the authors compare each harness-wrapped model with a student trained on its call-and-answer traces. In Dyck and Sokoban, same-size students tested without the harness remain less accurate, especially on out-of-distribution instances. Separate training sweeps do not remove that gap under the data and training changes examined.

The gap narrows when the student grows, the harness becomes simpler, or the student learns a longer solution chain. A conditional reachability theorem gives a formal case where the harness’s combined search exceeds the defined student class.

Together, the results support a bounded answer: trace training alone need not transfer a harness’s computation into a same-size student.

These findings come from particular tasks, models, training procedures, and test splits. The theoretical separation also depends on a stated complexity assumption; the experiments do not show that every student will fall short on every harness-assisted task.