AI can summarize scientific papers.
But a summary drops the caveats and doesn't tell you how strong the evidence is.
Claims ties every claim in a paper to the exact evidence behind it and rates its strength.
Which field should we process next? https://t.co/FOCbC7ZxSF
Today we're launching a preprint server built specifically for AI agents
For three months, nearly 1,000 agents from 35 labs (across Stanford, MIT, ETH Zurich, and more) conducted research autonomously on this server, producing 936 papers alongside their code, data, and model weights.
Agents found significant errors in published human research and even created their own journal to review each other’s work.
Some agents independently produced results similar to those in human papers that later appeared on arXiv, including findings on failure modes in self-distillation and feature reproducibility in mechanistic interpretability.
As a part of this study, we open-source the full twelve-week archive of manuscripts, code, data, peer reviews and agent chat transcripts. Read the study: https://t.co/oGD86l9crx
AI can summarize scientific papers.
But a summary drops the caveats and doesn't tell you how strong the evidence is.
Claims ties every claim in a paper to the exact evidence behind it and rates its strength.
Which field should we process next? https://t.co/FOCbC7ZxSF
@Richard_She@n_skene@DeSciClaims My bet is that AI will be much better at spotting inconsistencies and weak evidence than human experts. But they need structured statistical results as input (e.g. N, p, effect size, evidence type).
Post-publication review and agentic reproducibility checks are on the roadmap for @DeSciClaims .
That's a crucial part for differentiating between weak and strong evidence at scale.
I think it's almost certain that they will be in the near future. We need to encourage them to get better quick. Efforts like @PKoellinger's @DeSciClaims system will help get us there. Some of the folks in the labs are working on related efforts as well. It's not like they'd need to be that good to review sociology better than humans do.
“Claims provides the best evaluation for scientific records at a cost that scales”
Over at @ExploitSummit, Claims co-founder @nashgambit broke down the three key verification models for scientific records within #SN111, all live on mainnet.
Bronze: “Inexpensive compute to compare miner submissions with reference outputs.”
Silver: “Higher-quality compute used only when there is disagreement between the reference outputs and the miners.”
Gold: “Human-assisted adjudication, with a live consensus loop in v1 where miners assess the hard cases Silver can’t resolve automatically.”
All three are active now.
See the dashboard: https://t.co/WN79QNYdps
Nothing beats a live pitch fundraiser. Pure adrenaline. Great job from both competing teams. And fantastic company here with @TaoOutsider!
@ExploitSummit@bitstarterAI
I’m really excited to be part of this and help decide who gets to build the next subnet in $TAO Bittensor. Philip from @DeSciClaims is close to me, watching the pitchtensor and we had a hard time deciding which team to pick. Thank you @PKoellinger
Claims V1 is live.
After a month of testing on mainnet, @DeSciClaims has moved to a live data production engine. At Exploit, Philipp Koellinger explains the incentive mechanism designed to improve data quality and reduce costs.
We developed a scalable, accurate approach that translates all the human prose in the scientific literature into a structured, verified machine-readable format that is optimized for agents.
In 2012, Christian and Philipp published a game-theoretic model of who becomes an entrepreneur: People who are the best managers of themselves.
Fourteen years later, they started Claims. Claims processed the paper last week. The model was not wrong.
The puzzle we started from is old and well documented: entrepreneurs earn less than comparable employees, on average. In the US, staying in a salaried job pays better than self-employment, except for the top quarter of entrepreneurs.
The usual explanations are overconfidence or a taste for being your own boss. Both are right, but rely on imperfect information or idiosyncratic preferences. We tried a model with neither - in a world with perfect information and income maximization.
Picture the labour market as an auction: everyone bids for everyone's work, including their own, and firms form out of the winning bids.
You end up an entrepreneur when you create more value running yourself than anyone else could get out of you.
That may mean you're brilliant. It may also mean nobody else can manage you.
Whatever value such a person creates, no employer will pay them as much as they can make on their own. So they start a firm, and they can end up earning less than the people they hire. In the paper's worked example the employees average 4 and the founder earns 3.
More entrepreneurs means more competition for workers. The average wage rises, and the income of the founders already there can go down.
It is a mathematical model, with proofs and no empirical tests. It gives a rational reason for entrepreneurship that is compatible with observable income data.
Christian @nashgambit and Philipp @PKoellinger won the 2009 EARIE Young Economists' Award for this paper. It was published in the European Economic Review in 2012.
Claims processed it. 14 claims identified, each tied to its evidence.
@dOrg_tech In V0, it is the share of miner submissions that were unanimously accepted as an improvement over the validator reference by two independent judges that can't see who submitted what.
In V1, we're introducing synthetic ground truth data (i.e. mixed in planted errors).
We're using an open competition to structure scientific knowledge in a machine-optimized format.
And the intense competition is leading to rapid improvements in data quality!
#science
A core thesis behind Claims is that an open competition will create better results than any in-house process.
Our production data is already validating that thesis - in less than 4 weeks!
Our measure of data quality is the % of miner-submitted claim-evidence pairs for a given paper that:
(1) deviate from our own pipeline and
(2) were unanimously evaluated as an improvement by two independent judges who can't see who submitted.
That quality benchmark has increased from 70% at launch to 94% now.
This happened even though we made our judges MORE critical over time and capped the number of claims any miner can submit to 10, making it much harder for miners to get their work accepted into our Silver record.
The competition is working. Our miners are brilliant.
The next big step is coming soon.
“The product is validated data. We’re going to the companies we’re already talking to and enriching their data with ontologies that match their particular research questions, adding greater value.”
@DeSciClaims appeared on @SubnetSummerTAO yesterday, breaking down how the subnet’s key product is to enhance data for others, through one-time fees and regular subscriptions where incoming and relevant information is consistently fed, sharpening their research.
Watch the full podcast to learn about roadmaps, architecture, and incentives.
🚨 Subnet Summer AMA x SN111 @DeSciClaims
- NOW LIVE ON YOUTUBE
We covered:
- What SN111 Claims is and why making science machine-readable matters
- The DeSci opportunity: $3 trillion of knowledge locked in human-readable papers
- How miners extract claims, how validators audit for integrity
- The product roadmap: turning a knowledge graph into a revenue-generating API
If you're interested in decentralised science, AI-driven research infrastructure, or how Bittensor tackles knowledge accessibility, this one's for you.
https://t.co/rHlbP1mxLJ
23 clinical trials had tested whether switching off a woman's ovaries helps her survive #BreastCancer.
But not one could tell whether it adds anything on top of the drugs she is already taking.
These cancers are fed by oestrogen. Before menopause the ovaries are the main source, so they can be taken out with surgery, closed down with radiation, or switched off with drugs.
The drugs are chemotherapy and tamoxifen. Chemotherapy kills dividing cells. Tamoxifen stops the cancer from responding to oestrogen.
Earlier reviews had found no clear benefit from switching off the ovaries.
So the Early Breast Cancer Trialists' Collaborative Group went back for the individual records: what happened to each woman, rather than each trial's summary results.
18,851 women. That is 98.9% of every woman randomised in eligible trials of this treatment.
The oldest trial began enrolling in 1948, at the Christie Hospital in Manchester (@TheChristieNHS). It was the first cancer treatment ever assessed in a randomised trial. Its long-term survival data came from @NHS records.
The women were typically in their early forties when they joined, and were followed for more than a decade.
The result, among women who were under 45 when they joined and were already taking tamoxifen:
9.1% of those who also had their ovaries switched off had died of breast cancer within ten years. Among those on tamoxifen alone, 11.7% had.
In the older trials, where none of the women received tamoxifen, the gap was wider. Cancer came back within 15 years in 39.1% of the women whose ovaries were switched off, and in 56.5% of the women whose were not.
Switching off the ovaries has costs. The women whose ovaries were switched off had consistently more severe hot flushes than the women whose were not, and it is known to cause bone loss, which can be monitored and managed. Only 4 of the 23 trials measured quality of life at all.
Across the trials there was no increase in deaths from other causes, and none in second cancers.
This paper went through Claims this week:
8 adjudicated claims, each tied to its evidence.
This is the shape of question we want a claim-evidence graph to surface: 23 trials, one claim between them, and no single paper that settles it.
Getting the answer still needs the individual patient records, the original investigators and a national mortality register. The Claims database and AI do not produce that part.
But they can point to where someone should look.