2 hours ago
7
An artificial intelligence system has beaten scores of forecasting enthusiasts, including several professionals, in a contest to predict events ranging from bust-ups between Donald Trump and Elon Musk to Kemi Badenoch being removed from the Conservative party leadership.
A British AI startup, co-founded by a former Google DeepMind researcher, has ranked in the top 10 of an international forecasting competition, which requires entrants to forecast the likelihood of 60 events over the summer.
ManticAI came eighth in the Metaculus Cup, run by a San Francisco-based forecasting company that tries to predict the future for investment funds and corporations.
AI’s performance still lags behind the best human forecasters, but it has left some believing AI could outstrip humans sooner than most expected.
“It’s certainly a weird feeling to be outdone by several bots at this point,” said Ben Shindel, one of the professional forecasters who found himself behind AI during the contest before finishing above Mantic. “We’ve really come a long way here compared with a year ago when the best bot was at something like rank 300.”
Questions in the Metaculus Cup included which party would win the most seats in the Samoan general election and how many acres in the US would be burned by fires from January to August. The contestants were scored on how well they predicted outcomes as of 1 September.
“What Mantic has done is impressive,” said Deger Turan, the chief executive of Metaculus.
Turan estimated that AI would be on a par or better than the best human forecasters by 2029, but said that in general “currently human forecasters are doing better than AI forecasters”.
On complex forecasts that rely on predicting interrelated events, AI systems can still struggle to carry out logic verification checks when translating the knowledge into a final prediction, he said.
Mantic breaks down a forecasting problem into different jobs and assigns them to a roster of machine-learning models including OpenAI, Google and DeepSeek, depending on their strengths.
Toby Shevlane, the co-founder of Mantic, said its performance was a milestone for the AI community using large language models for forecasting.
“Some say LLMs just regurgitate their training data, but you can’t predict the future like that,” he said. “It requires genuine reasoning. You could say our system’s predictions were more original than most human entrants, because people often cluster around the community average predictions. The AI system often strongly disagreed. So, AI forecasters could be an antidote to groupthink.”
Mantic’s system deploys a variety of AI agents to assess what is happening now, carry out historical research, game out scenarios and then predict what is likely to happen next. A strength of AI forecasting is its ability to work hard persistently, which is crucial to effective forecasting.
They can easily work on dozens of complex problems at once and revisit them daily to learn from changing information. Human forecasting also uses intuition, but Shindel is among the human forecasters who think this could emerge in AI.
“Intuition is very important, but I don’t think it’s innately human,” he said.
Leading human superforecasters still say they are best. Philip Tetlock, the co-author of the bestselling book Superforecasting, this year published research that found that expert humans were on average still outperforming the top-performing bots.
Turan said that on complex forecasts, which rely on predicting interrelated events, AI systems can still struggle to spot logical inconsistencies in their outputs and to carry out verification checks.
Warren Hatch, the chief executive of Good Judgment, a forecasting company co-founded by Tetlock, said: “We expect AI will excel in certain categories of questions, like monthly inflation rates. For categories with sparse data that require more judgment, humans retain the edge. The main point for us is that the answer isn’t human or AI, but instead human and AI to get the best forecast possible as quickly as possible.”
Or, as Lubos Saloky, a human forecaster who came third in the Metaculus Cup, put it: “I do not plan to retire. If you can’t beat them, merge with them.”
