The same AI, two kinds of days
81 in 100days called right that it had read about
46 in 100days called right after its cutoff
50 in 100days a coin calls right
Tomorrow's paper
This AI beat the market for years. Then it ran out of newspapers.
An AI model reads the internet up to one day, its cutoff. Test it on years it already read about and it looks like a genius. Drag the day it stopped reading.
Why the genius always stops at the cutoff
The AI was not forecasting. It was remembering. A test on years inside its reading is a test of its memory, and its memory is very good.
Telling it "pretend it is 2010" does not help. The memory is still there. The only fair test uses days after the model stopped reading.
This is not a cartoon
In 2025 Lopez-Lira, Tang and Zhu tested GPT-4o, which stopped reading in October 2023. Asked for past S&P 500 levels, it was off by 0.6 percent on average and had the day's direction right 81 percent of the time. After its cutoff it was off by 17 percent and had the direction right 46 percent of the time.
Shown one day's Wall Street Journal front-page headlines, it named the exact date 47 percent of the time. For headlines after its cutoff, 8 percent. Told to ignore everything after 2010, it still called economic growth after 2010 right 98 percent of the time. On truly unseen years it managed 40 percent.
There is a second trap. Glasserman and Lin found that a model knowing a company's name can cloud its reading of the news. Hiding the name did better.
Before you trust an AI backtest, ask: did the test start after the model stopped reading?
Sources
- Lopez-Lira, Tang and Zhu, "The Memorization Problem: Can We Trust LLMs' Economic Forecasts?", arXiv 2504.14765 (2025). GPT-4o S&P 500 recall: 0.61 percent average error and 80.6 percent direction before its cutoff, 16.9 percent and 45.7 percent after. Headlines: 47.0 percent exact date before, 7.9 percent after. Fake 2010 cutoff: 97.6 percent before, 98.0 percent after, 40 percent on real unseen data.
- Glasserman and Lin, "Assessing Look-Ahead Bias in Stock Return Predictions Generated by GPT Sentiment Analysis", arXiv 2309.17322 (2023).
- Lopez-Lira and Tang, "Can ChatGPT Forecast Stock Price Movements?", arXiv 2304.07619. Tests only on headlines after the model's cutoff, for this reason.
- The market here is random, made fresh in your browser. The AI calls each day right with the chances GPT-4o showed: 80.6 percent before its cutoff, 45.7 percent after. It buys when it says up and sells short when it says down.