A language model grows exact answers by checking new code
Esta página aún no tiene una traducción verificada. Se muestra la versión en inglés. Leer la versión en inglés
En palabras sencillas
A computer can answer exact questions by running small programs, but it cannot answer questions that its available operations do not express. This paper introduces a system in which one frozen language model finds more answerable questions and proposes new program operations.
The system checks answers by comparing independently written programs and testing rules such as whether group totals add up to the whole. On synthetic merchant records, one round found 36 answerable question types using the same nine operations, up from 10. These checks cover tested records, not every possible input.
La investigación
When answers must be exact, how can a system learn to answer new kinds of questions without an answer key, reward, or judge that might accept a plausible but wrong capability?
The paper asks whether program operations and answers can instead be expanded through checks that rely on a trusted interpreter.
A two-loop system expands exact question answering by checking expressions and new operators against independent implementations and trusted relations.
The system's inner loop finds more questions answerable with existing operations; its outer loop proposes operations for gaps the current set cannot express.
The model and trusted interpreter check expressions through independent agreement and relations among operations, without an answer key or judge.
On synthetic merchant records, one inner-loop round grew the answerable set from 10 to 36 types while keeping nine operations fixed. A separate gold extractor matched the DSL on evaluated precise or compute types for non-truncated sequences, apart from count ties handled differently. These results support expanded, checked answers within the tested conditions.
The checks compare programs on finite validation records and fuzzed inputs; they do not prove correctness for every possible input. The system also cannot admit the most-frequent-value operation, and runs beyond three induction rounds were not tested.
Análisis
Detalles
- Autores del paper original
- Yun-Shiuan Chuang, Yirou Ge, Yixi Li, Yiming Bian, Zijie Pan
- Enviado a arXiv
- Licencia
- arXiv non-exclusive