A migration agent’s apparent learning fades after leakage checks
Does a Self-Learning Migration Agent Learn? An Audited Evaluation of Evidence-Graded Experience Memory for Oracle-to-PostgreSQL Migration · arXiv:2611.16143
In plain words
The paper tests a software method for moving database code from Oracle to PostgreSQL. It keeps the language model fixed, saves typed records of past migration attempts, and uses verifier results to decide which records count as evidence and which strategy to try.
On a documentation-based test, memory raised agreement with related PostgreSQL examples on the full test panel. But that advantage was near zero on units whose reference target was absent from training, so the full-panel gain did not clearly show learning that transfers to new tasks.
The research
A higher score on unfamiliar tasks can look like learning even when the system is reusing answers that crossed from training into the test set. This audit shows why the evidence behind a gain matters: here, the memory advantage did not persist once shared PostgreSQL references were excluded.
The paper audits an Oracle-to-PostgreSQL agent to test whether experience improves unseen tasks beyond gains from reused references.
The team compared a checklist system with and without persistent memory, measuring reference conformity: the weighted share of eligible checks against related documentation examples passed per unit.
The original panel showed a 0.015 gain, but only 0.002 on leak-free units, with an interval spanning zero. A reference-grouped rerun also found a small, uncertain memory contrast, consistent with shared references explaining the initial rise.
Studies 1–2 used related documentation examples, not executable Oracle translations, so their reference-conformity scores cannot show migration behavior.
The reference-grouped rerun used 301 training units, and no experience was verified, so its knowledge-admission and adaptive-strategy mechanisms were not exercised.
What the evidence shows
- What the agent is meant to learn
The paper’s Oracle-to-PostgreSQL migration agent is a method that stores typed experiences and can adjust its strategy while keeping the underlying language-model weights fixed.
The key question is whether those changes help with unseen tasks, or whether apparent improvement comes from retrieving similar examples, reusing reference targets, or repeated attempts on the same task.
- The initial gain measures agreement
The main measure, reference conformity, is the weighted share of eligible checks against related documentation examples that a unit passes. On the original test panel, verified persistent memory raised this score by 0.015 over a checklist system without memory.
That is a measured gain in agreement with examples, but the metric does not establish that the generated migration behaves like the Oracle program.
- Shared targets explain the concern
An audit found that 72% of held-out units had PostgreSQL reference targets that also appeared among training targets. When the researchers re-scored only units without such shared targets, the memory advantage over no memory was 0.002, with an interval that included zero.
The contrast weakens the interpretation that the original gain showed experience transferring to unseen tasks.
- Copying beats the LLM systems
A program that copied the PostgreSQL target paired with the most similar training source scored 0.222 in reference conformity on the original panel, above the best LLM system at 0.163.
Since both scores measure agreement with documentation examples, this control shows that the original measure and split could reward reference reuse without requiring the agent to learn a migration skill.
- The rerun finds no clear gain
A rerun grouped test units by their PostgreSQL reference target so those targets could not cross from training into the test set. Across independent target groups, memory’s RC gain was small and uncertain, and the adaptive strategy added no group-level gain over memory with a fixed strategy.
No experience was verified in this rerun, so the admission and adaptation mechanisms were not exercised.
- Behavior needs its own check
A separate execution-based study compared migrations with Oracle outputs and errors. It found cases where an empty script scored well because Oracle itself raised no error, and PostgreSQL acceptance sometimes disagreed with Oracle behavior.
These results show why agreement with documentation or successful acceptance must be distinguished from measured behavioral correctness.
Deep
- Original paper authors
- Oleg Grynets, Vasyl Lyashkevych, Valeriia Oliferuk, Vladyslav Babenko, Danylo Kovalchuk
- Submitted to arXiv
- License
- CC BY 4.0