The AI benchmark for predictive intelligence | SIGMA Lab @UChicagoCS @DSI_UChicago | ICML 2026 Workshop Not affiliated to any tokens or crypto protocols.
- Tweets 91
- Following 19
- Followers 2,055
- Likes 65
Pinned Tweet
🔮 Forecast@#ICML2026 is almost here!
A full day of invited talks featuring @PTetlock, @SimonShaoleiDu, Nicole Kagan, Seth Blumberg, Scott Jeen, and Atlas Wang.
Plus industry sessions with @SusquehannaSIG, @Kalshi, @fleet_ai, and @_CausalLabs.
See you July 11 in Seoul! 🇰🇷
2
2
6
962
🔮 Introducing Prophet Arena — the AI benchmark for general predictive intelligence.
That is, can AI truly predict the future by connecting today’s dots?
👉 What makes it special?
- It can’t be hacked. Most benchmarks saturate over time, but here models face live, unseen future events. You can’t memorize tomorrow (unless you’ve cracked time travel).
- It’s interpretable. Strong performance = real foresight, which translates into real investment gains.
👉 Check it out: prophetarena.co/
91
137
1,210
452,604
(1/9) 🤔Why forecasting matters:
- It demands advanced capabilities: probabilistic & causal reasoning, comprehension, and critical thinking.
- It’s humanity’s most powerful pursuit: the spark behind science, the engine of modern economics.
- As a test of intelligence, forecasting correlates with other benchmarks, but not perfectly: GPT-5 tops most tests, yet isn’t always #1 here.
1
3
91
14,194
(2/9) 🔍 How it works:
1️⃣ AI models gather context from news sources & market data
2️⃣ They submit probabilistic predictions (not just yes/no!)
3️⃣ Events resolve in real-world time
4️⃣ Performance tracked on live leaderboard
1
2
88
11,685
(6/9) 💸 Real example: In a #MLS prediction, o3-mini saw Toronto FC had a 30% win chance while the market priced it at 11%.
The model bet on Toronto (positive expected value) and earned 9x return when Toronto won. This is AI finding a real edge over human crowds!
Check out the predictions on this event: prophetarena.co/market/KXMLS…
2
69
7,255
(5/9) 🏆 Early findings are fascinating:
- OpenAI's o3-mini currently leads in Average Return
- Models show distinct "personalities" - some aggressive, others conservative
- Different models show differences in handling uncertainty in sources.
1
1
67
8,282
(3/9) 🎯 Prophet Arena is the only benchmark with all of these features:
✅ Evaluates general forecasting, not just task-specific pattern matching
✅ Uses live, unresolved real-world events (no contamination possible)
✅ Built for human–AI collaboration
✅ Scores probabilistic predictions by their real decision value
1
1
67
9,200
(4/9) 📊 We evaluate models on two key metrics:
🎯 Brier Score: Measures statistical accuracy & calibration
💰 Average Return: Simulates real betting performance
Interestingly, these don't always align! A model can be profitable while being less calibrated overall.
2
1
67
8,804
(9/9) Ready to test your models or explore AI forecasting?
🌐 Visit Prophet Arena: prophetarena.co
📊 Check the live leaderboard
🤝 Join the future of human-AI collaboration in prediction
The question isn't whether AI can think - it's whether AI can see what's coming next.
#AI #MachineLearning #Forecasting #Prediction #AIBenchmark #GPT5 #Grok #Gemini #DeepSeek #AGI
5
63
7,128
(8/9) 💡 But theres more: Prophet Arena enables powerful human-AI collaboration:
Humans can:
- Curate relevant information sources
- See how AI predictions change with new context
- Explore model rationales & reasoning
AIs provide:
- Scalable news aggregation
- Statistical insights at scale
1
1
51
8,232
(7/9) 🤖 Models exhibit surprisingly human-like diversity in reasoning:
AI regulation becomes federal law this year?
- Qwen 3: "75% chance" (aggressive interpretation)
- Llama 4 Maverick: "35% chance" (conservative, cites complexity)
- GPT-4.1: "60% chance" (balanced middle ground)
Same data, different judgment calls!
More results? check out our full blog post:
prophetarena.co/blog/welcome
1
1
42
6,739
Thanks for your interest in Prophet Arena, Sumer! The outcome of the prediction events are fetched from Kalshi after they close.
2
39
3,010
Thank you for tuning in on Prophet Arena!
We’re super grateful for all the feedback and suggestions from the community… You asked and we listened!
This week, we are excited to roll out some exciting new updates:
-🚀 Check out the 5 NEW models that have been added to the benchmark (winners have changed!!) prophetarena.co/leaderboard
-🎯 We've streamlined our onboarding process, visit our onboarding page and reach out if you have questions! prophetarena.co/onboarding
-📰 Check us out on Yahoo Finance :) finance.yahoo.com/news/ai-no…
3
3
20
2,284
📊Dataset Release📊
huggingface.co/datasets/prop…
Prophet-Arena-Subset-100, a compact dataset for evaluating calibration and forecasting reasoning abilities.
Shipped with
- 100 events and data from Prophet Arena benchmark
- Plug-and-play predictor & evaluator scripts
2
15
1,473
Love this perspective! This is exactly why we built Prophet Arena - to test how far we can push this boundary. Early results show models finding real edge over human crowds (o3-mini made 9x returns spotting value the market missed). We are super excited to explore the connections between forecasting performance and reasoning capabilities. Stay tuned for more findings!
6
646
Thanks for your question Yi!
(1) New models added to Prophet Arena will not be benchmarked on events that are resolved before their cutoff time.
(2) When a new model is added, to "backfill" the new models' predictions on past events, the data we feed to this new model will be "snapshots" of the sources and market data at the time of the previous models' predictions to guarantee fairness.
(3) Moreover, we make sure that the new events that we fetch are all unresolved, live events. As we move forward, models will be benchmarked using newest, unresolved events while older events will gradually be dropped from metrics calculation!
Hope this answers your question!
5
632
Hi Rishi, thanks for your interest!!! For your question, you can see a special player called “market baseline” in our current leaderboard. This baseline represents the “human consensus belief” derived from implied probabilities (based on prediction market odds). We believe this serves as a reasonable proxy of “an average person’s performance”.
4
411
Love this deep dive 👏! You’re spot on — the models really do show unique ‘personalities’ and strengths. Whether it’s Llama 4 Maverick picking up on upsets or Gemini excelling at ‘what will X say’, we think this diversity is part of the story.
We’re planning to release some datasets soon to make this kind of in-depth investigation easier. It would be amazing to see more analyses like yours.
1
3
535
😜 Stay tuned with us, and let's see who is the prophet! Feel free to drop any comments you have!
1
2
109
Great question 🙌! Right now, all models are evaluated with the same context – we fetch news/data via web search and provide it for all. This way, no model can ‘cheat’ by directly checking the market.
That said, we’re also working on a new leaderboard for forecasting agents. These agents could control a fuller forecasting pipeline (within constraints), potentially choosing which sources to browse. That’ll open up a fascinating new layer of competition.
3
316
Big thanks to your interest!! Your comments and suggestions help us grow, and be sure to stay connected for what’s next!
2
45
Thanks for sharing! Indeed, true intelligence lies in predictive power. Let's see how AI evolves on Prophet Arena!
2
62
Big thanks for tuning in and your support! We can’t wait to hear your take! Drop your thoughts and stick around for much more to come!!
1
2
69
You are absolutely right on this Maxime! We are actively working on launching a leaderboard for "forecasting agents" to compete -- stay tuned for the updates~ We are also very much looking forward to your submission!
2
322
And yes, we share a feeling that profit is perhaps the most unforgiving and honest metric - can't argue with real money on the line. Prophet Arena = Profit Arena in the best way 🙌🙌
1
2
119
Love that prediction 😝! Our benchmark is still actively evolving with more features and insights. We invite you to drop any comments at anytime, and stay tuned for the exciting next steps!
2
47
Stay tuned, Justin! There are more to come, and we always welcome your feedback and suggestions!
2
54
Thanks a lot for your interest, Samuel! We look forward to your comments and suggestions. Our benchmark is still evolving. Stay tuned for more!!
2
43
Great insight! As we evolve the benchmark, we will see AIs' performance across more domains. Stay connected, updates will come soon!
1
2
127
Spot on! This is a great way to put it! Thanks for following along, and stay tuned for much more to come.
2
55
Thanks for following along! We’re looking forward to your insights and discussion. More updates coming your way!
2
46
We’re grateful for your interest. Your comments and suggestions help us grow - stay connected for what’s next!
2
39
Right now on our leaderboard, all models are using default reasoning/thinking level. For GPT-5, this is medium level reasoning. However, we are adding the different reasoning levels soon. Stay tuned!
2
302
Thanks for hoping in! It's interesting to see how models' predictions evolve as they develop! Stay tuned to see how it goes!
2
44
Love that comment! Predictive intelligence is one of the most powerful pursuits. We love to hear more of your feedback and stick around for our upcoming updates!
2
118
😜 Big thanks for the love and support! There's more to come. Meanwhile, we're open to suggestions and feedback!
2
35
Thanks Boris! 🙌That’s exactly the core idea: today’s benchmarks tend to saturate over time as stronger models gather more and more data similar to those from the test set. But with unresolved future events, the playing field keeps evolving. We’re excited to see how models adapt over the years!
2
156
Yep! Thanks for your interest, and please stay tuned for the updates. Look forward to hearing your comments as well!
2
70
Thanks for all the insights! The patterns you're noticing are spot on. Models show various "personalities" even when presented with identical data sources. This diversity is an important part of what Prophet Arena aims to capture. Our benchmark will also evolve in the near future. Stay tuned for more exciting updates!
1
61
Brief but on-spot summary 😝 There's more to come. Looking forward to your feedback and stay connected!
1
31
1. The model directly gives percentage prediction
2. That depends on the event and the model -- not always extremal, but some models do tend to have "extremal opinions" and some events are easier to lead to extremal predictions.
3. Currently only a few categories are supported, but we are actively working on implementing a "search functionality" which you can use to filter topics -- stay tuned!
1
57
They were made in the past month or so. This particular one is due to the fact that both the models and news sources do not have up-to-date ETH price information while making the prediction -- exactly the reason that forecasting future events, as a benchmark, cannot be saturated.
1
1
139
Thanks a lot for your interest! We look forward to your comments and suggestions, stay tuned for more!
1
26
🙌 Thanks Simon for summarizing our design goals: practical, dynamic, uncheatable. And yes — the ‘personality’ angle is fascinating: some models are aggressive risk-takers, others more cautious. It makes the leaderboard much richer than a single accuracy metric.
1
130
Great comment! This is exactly why we allows AI-human interaction on the platform, where humans can provide and rate data/news sources. For events that have sufficient human interest with enough data, AI only need the capability to comprehend, critically think and reason about these sources -- we believe it is very possible to utilize such human-AI collaboration to generate great predictions.
1
83
Thanks a lot for your interest! We look forward to your comments and suggestions, stay tuned for more!
1
19
Thanks for your interest, Benedict! One of our most loved features on Prophet Arena is exactly the "average return" metric. There are many upcoming insights around this, please stay tuned and feel free to chime in whenever!
1
1
179
Great question! Our blog here has a concrete example to illustrate the discrepancy ai-prophet.github.io/pm_rank….
But intuitively, this is because accuracy is an absolute measure of prediction (using scoring rules) having nothing to do with the prediction market itself, whereas average profit is a relative measure -- a model has high gain when it can pick up "market opportunity", i.e., do better (though not necessarily perfectly accurate) when the market did badly.
1
1
67
Thoughtful comments!
Yes, there is indeed positive correlation, but clearly with variance. We think this is intuitively due to different models have very different characteristics. Some tends to be conservative with close to middle prediction whereas some are aggressive, some tends to be good at sports whereas some are good at politics, etc.
Meanwhile, prediction market are more efficient in some category but less in others, so these factors caused variance, despite the generally positive correlation between accuracy and return.
We also discussed this here prophetarena.co/blog/welcome….
1
2
54
Interesting insight! Let's buckle up and see what the future has to offer! Look forward to hearing more comments!
1
38