1/ Introducing Eternis
In Asimov's Foundation Series, a mathematician discovers you can predict the fate of civilizations by finding the emergent laws that govern them. For decades, that was pure science fiction.
New insights on using model internals for much more reliable forecasting will soon be live on Axion, our multi-agent system for reasoning through high-stakes forecasts and decision-making.
Exciting results from our partnership with @GoodfireAI!
Can LLMs predict the next World Cup champion?
Goodfire partnered with @EternisAI to improve how LLM forecasters use available evidence and manage uncertainty.
We found models were overconfident in their predictions – but probes significantly improved calibration. (1/6)
Language models as forecasters can be accurate while being badly calibrated and overconfident. Their chain-of-thought can also omit the evidence that actually changed a forecast.
In this new work, we find that internal activations give us a much more reliable signal! 1/6
Can LLMs predict the next World Cup champion?
Goodfire partnered with @EternisAI to improve how LLM forecasters use available evidence and manage uncertainty.
We found models were overconfident in their predictions – but probes significantly improved calibration. (1/6)
the greatest experiment in human history turns 250 years old 🇺🇸
deeply grateful to the opportunities this country has given me and for the people that have fought to maintain its freedom
1/ Forecasting models
We trained an 8B-parameter model that surpasses all published baselines on open-ended forecasting, including models 10–15x larger.
Post-training a small model to reason about uncertainty the way a good forecaster does turns out to give a significant portion of the gains!
7/ With these results, we’re now scaling up post-training to match the best reasoning models on general-purpose forecasting at a fraction of the cost.
We are also creating internal evaluations for what makes a "good forecaster". This will be critical in a world where models reinforce beliefs and what humans and agents alike coordinate over is heavily influenced by these
The goal: cheap, continually learning world models that maintain forecasts across every human-relevant question.
If this is exciting, reach out to us!
Full post: https://t.co/Z9SCcLPhJq
6/
- Will liquid biopsy detect 10+ cancers at 90% sensitivity by 2030?
- When does weakly general AI arrive?
- What happens to scientific output if global compute increases 10x?
If questions like these fascinate you and you want to build the systems that answer them, reach out:
[email protected].
Learn more about us:
https://t.co/WqRjT6n8hd
1/ Introducing Eternis
In Asimov's Foundation Series, a mathematician discovers you can predict the fate of civilizations by finding the emergent laws that govern them. For decades, that was pure science fiction.
5/ What Eternis is building
- Forecasting system built on cheap, continually learning and updating world models
- Axion, a decision product grounded in structured evidence: https://t.co/XRYMN7dpBY
- Lume, a prediction market designed for autonomous agents
- Research into multi-agent coordination and the emergent properties that arise at scale