Models can claim ignorance while recalling past financial facts
LLMs Don't Know What They Know: Benchmarking Temporal Self-Knowledge for Financial Backtesting · arXiv:2611.06808
In plain words
A model asked to predict an old stock price or economic number might remember the answer from past data. That can make a historical test look like forecasting when it is actually recall.
The paper introduces a test that asks models both for dated financial facts and whether they already know each answer. GPT-5 often answered facts correctly even when it said it did not know them, so its claims alone could not show where its factual knowledge ended.
The research
A backtest can make a model look predictive when it is repeating a past result from memory. For judging financial forecasts, this work changes what a cutoff or a model’s own claim can establish: neither reliably identifies what the model can already answer in these tests.
A benchmark pairs models’ answers to dated U.S. financial facts with claims about knowing them, then tests whether historical-date instructions expose recalled outcomes.
The benchmark compares GPT-5’s undated prediction prompts with the same prompts asking it to answer as of a historical date. Its share of answers claiming prior knowledge fell from 39% to 1% across the full sample, while accuracy on 1,340 controlled facts shifted from 76% to 75%.
The prompt changed the model’s claims much more than its answers. That supports caution about using date instructions to distinguish recalled outcomes from forecasts in historical financial tests.
The facts cover one economy and three families of monthly U.S. financial data, and the boundary estimates depend on chosen thresholds. The tested point-in-time models were small and showed no detectable boundary signal.
What the evidence shows
- Test recall behind forecasts
The paper introduces a benchmark that measures models’ answers to dated financial facts alongside claims about whether they know those answers. This comparison matters because a historical backtest can mistake recall of an outcome for forecasting ability.
The benchmark also tests whether a historical-date instruction changes those answers or merely changes what a model says about them.
- Control two easy shortcuts
The benchmark uses sampled monthly U. S. economic observations, stock prices, and Federal Reserve decisions, asking models for values and knowledge claims on the same facts. For factual accuracy, the researchers exclude numerical cases answerable by simply carrying forward the previous month’s value and balance the three kinds of Federal Reserve outcome. This makes the accuracy score less vulnerable to those two simple shortcuts.
- Claims miss some knowledge
GPT-5’s claims separated stronger from weaker answers: it was more often correct on facts it said it knew than on facts it disclaimed. Yet it still answered many disclaimed facts correctly, so a claim of ignorance did not establish that an answer was unknown.
- Claimed knowledge can persist
The temporal comparison shows that knowledge claims can continue after measured factual accuracy has fallen to chance. This gap suggests a model’s claims can extend beyond its measured ability to answer.
Because cutoff estimates depend on a threshold rule, the estimated boundary is not a fixed property of the model.
- Dates change claims, not answers
For GPT-5, adding a historical-date instruction changed knowledge claims far more than factual accuracy on the controlled facts. This suggests the instruction changed how the model described its answers more than what it could answer. In this tested setting, date instructions did not reliably separate recall from forecasting.
- Cutoff controls remain inconclusive
Small models trained on text ending at fixed dates did not provide a clear behavioral cutoff: their accuracy stayed near chance on both sides of those dates, with no detectable differences. This limits their value as controls for the benchmark’s boundary estimates.
Taken together, the results support checking demonstrated factual behavior rather than relying on a stated cutoff, a knowledge claim, or a date instruction alone.
Deep
- Original paper authors
- Lingfei Qian, Yan Wang, Xueqing Peng, Yupeng Cao, Yuechen Jiang
- Submitted to arXiv
- License
- arXiv non-exclusive