They had models guess, for example, who would win the NBA Championship and who President Trump would meet with.
“Very quickly we started to realize that models did really well,” Xu tells me. The models were sometimes able to do better than the average person without any special instructions or guidance, he says. In their testing, GPT-5 was the best at making predictions, followed by Grok 4, Claude Sonnet, Gemini, and Llama.
The team went on to develop Prophet Arena, a benchmark that measures a model’s ability to make predictions. It’s one of a number of benchmarks designed to test AI’s future-gazing capabilities; another, Prediction Arena, from researchers at Harvard University and a company called Arcada Labs, recently gave frontier models $10,000 to wager on prediction markets over 57 days. The models consistently lost money in their experiment, but the researchers saw evidence of them improving consistently over time.
Xu tells me that AI can most effectively make predictions using a custom harness, and notes that having agents with unique tasks—reading the news, collecting other data, and analyzing the whole system, for instance—seems to be the most effective approach.
Interestingly, AI still often lags behind very skilled experts known as superforecasters. Xu explains that, although humans cannot consume as much information, their understanding of the real world—that is, of the dynamics behind a news story—is deeper.
That said, there’s growing evidence that the human superforecaster’s edge is eroding. A contest called the Metaculus Cup recently showed that AI can sometimes beat even the best human predictors. And a benchmark called ForecastBench could provide further evidence that AI can match human skills when the results of an ongoing contest conclude soon. In fact, I think we can expect the big labs to turn their attention to future-gazing as a powerful new way to measure progress in machine intelligence.
All of these predictive experiments seem to suggest that, like math, coding, and law, industries predicated on trading may soon be upended by AI. Perhaps the best human traders will take on a new kind of role: supervising and orchestrating AI agents capable of divining the future far better than they can.
For the time being, my AI’s suggestions still don’t seem all that impressive. Its team hasn’t outscored the average over the past weeks. I certainly won’t be using any of its suggestions in my own selections.
But AI will no doubt get better, perhaps making it all too easy to dominate fantasy league pretty soon. I’m kind of hoping it won’t, though, as it would spoil the fun of trying to earn bragging rights by winning all by yourself.
This is an edition of Will Knight’s AI Lab newsletter. Read previous newsletters here.
