AI-driven model discovery needs a test the builder doesn’t control.
This study found agents improving on reproduced published-best results across 20 scientific ML benchmarks.
The researchers separated model development, held-out testing and auditing to make those gains meaningful.
That’s closely aligned with what we’re building at Carbon: give miners room to experiment, then evaluate their models against the requirements of a specific engineering problem.
Great paper from Anthropic on AI agents doing ML-for-science research.
(bookmark it)
If you build AI-scientist systems, this one is worth your time.
They propose Agent4Sci, which gives frontier coding agents 20 scientific ML benchmarks across 11 domains and asks them to beat the published best model.
They beat it on all 20. Claude Opus improves on the published best by 10.4% on average, and no run was flagged for misconduct.
The gains come almost entirely from standard ML-engineering tricks. The trajectories show agents misjudging what is novel, leaving compute unused, and settling into small local sweeps.
Adding three lightweight skills for research ideation, experiment management, and scheduled checklists raises Claude Opus from 10.4% to 20.1% and Codex from 4.4% to 8.1%, with more varied and more domain-relevant changes.
The models stay the same, and the research scaffolding roughly doubles the gain.
https://t.co/E9oOxkBjiA
Chat with Paper: https://t.co/9B0ITtV9bW
Two live Revenue Searches this week. One on Thursday 1130 UK time and one on Friday 1500 UK time. Will be streamed here. Bring your questions and will see you then!
@nvidia cuDSS
NVIDIA’s cuDSS is now generally available, accelerating calculations behind engineering simulations. Better solvers strengthen the foundations for training and checking physics AI models.
https://t.co/2FrGf3QOVz
It's really exciting for us to see the acceleration of Physics AI.
Daily announcments show where engineering is heading: faster simulation, AI-assisted design and better ways to test physics models.
For Carbon, the opportunity is turning that progress into useful models that engineers can rely on.
🧵 Here’s what caught our attention in the last 24hrs ↓
@autodesk + @nvidia
Autodesk plans to bring NVIDIA PhysicsNeMo and Warp into NavPack, its AI-powered aerodynamic design tool.
The goal: give designers faster feedback while they’re still shaping the car.
https://t.co/rXjnQqkF9W
Dev Update Highlight: Carbon Landscape
Every model we test adds to Carbon’s map of what works.
We retain the recipe, training procedure, predictions and comparisons against the physics solver.
Over time, that builds knowledge about which methods work for which engineering problems, and where they fall short.
That knowledge helps guide future research and gives customers evidence for using the models we deliver.
Carbon | Development Update: October 12–19
Path to Mainnet
Last update I told you I won’t pay miners for anything until this system produces something a customer will pay for. That hasn’t changed.
We're getting close! We just had a model run a challenge end to end, submitted, rebuilt, scored, and evidence saved
Huge milestone.
This post is long, but hopefully it helps you all understand just how much thought we're putting into this. The design is meant to be robust and efficient at onboarding new challenges once it's built.
We are laser focused on creating a system that can target SOTA physics AI research at ANY valuable real world engineering problem we want...
Here's what changed this week is how we’re getting there.
1. The Model Rebuild
Last week Carbon miner submissions ran on a ladder of permission levels. We tested all of these permission levels for exploits which was great, but when we went to run the full system it added way too much complexity and fragility.
Now it's one path, any method. Using what we learned in testing. This is the hard part. Automated model building on command, but it'll be such a weapon.
Miners now submit a construction contract that takes in everything the validators need to rebuild the model from scratch. Then the validator uses that Challenge’s preset training data, freezes it, and tests it in a separate environment the miner’s code never touches.
No pretrained weights smuggled in. A second rebuild has to match the first. Every winning model is one we can reproduce, which is exactly what a customer needs to trust it.
This is a huge benefit of these physics models being small. Rebuilding them protects us, and gives us insight into what methods are working. I think this will be an excellent way to make sure we only pay for real valuable work.
2. The Score:
Score = Value
This is the biggest change, and it’s the one that matters most to customers.
ScoreLab is a new piece of Carbon. It’s an internal workbench where every model we build in testing is put head to head against the physics solver it’s meant to replace, on the same problem, at the same fidelity. It runs the exact same scoring code our validators run. What we measure in development is exactly what miners get paid on.
It's how we choose the optimal scoring design before launching. Unique for every challenge. Focused on value for the customer.
Here’s how a model gets scored now:
First, the gates. Does it obey the Challenge’s physics? Is it accurate enough to be used at all? Fail either and nothing else counts. Speed can’t buy you out of a physics failure.
Then two scores, and both have to be strong:
Accuracy across the whole problem: not just on average, but on the hard and unusual cases too.
Decision quality at the limits. Does the model put a design on the right side of the challenge's limit, and by the right margin?
Then we take its recommended design and run the real solver on it. Does it actually hold up? Calling a design safe when the solver says it fails is tracked separately, because that’s the mistake that costs a customer.
Then speed: measured honestly against the solver, on stated hardware.
Last update I told you our old score failed the customer test: models that were accurate on average were wrong at the limits.
This fixes that structurally. A model can’t score highly by being good at one thing. It has to be accurate everywhere, right where decisions get made, and fast. One strong number can’t hide a weak one.
Excellent across the board, or it doesn’t win.
The exact balance between those scores what ScoreLab is solving for, from evidence, not guessing.
Every model that goes through ScoreLab is kept: its recipe, its training procedure, every prediction, and exactly how it did against the solver. Over time that becomes a map of which methods of building a physics model actually work, for which kinds of problem. That’s the data our intelligence product is built on. Miners compete for rewards.
Carbon learns from every attempt. Internal and External.
3. The Product.
Why this is bigger than scoring. A score tells a miner who won. A customer needs much more than that. They need to know where the model can be trusted, and what to do when it can’t.
That’s what the Carbon Model Package is:
The model, the physics contract for the job it does, a map of where its answers hold, and a guarded design workflow.
When an answer is in doubt, it hands off to the full solver or to an engineer instead of guessing. The customer gets a decision with the evidence behind it, not just a prediction.
So we built ScoreLab to collect exactly what that package needs. Every model it tests is kept with all of it: every prediction, how far it was from the solver across the whole problem, which side of every design limit it landed on, which mistakes would have been dangerous and which merely cautious, and whether its recommended designs held up when the solver checked them.
The same measurements that decide who gets paid become the evidence that tells a customer where to trust the model.
Miners compete for rewards. The customer gets the proof, and an intelligent product designed for them.
4. Miners can bring the tools they actually use
This week opened the door a lot wider:
Miners can use the open-source libraries they already know. Carbon pins and provisions the dependencies on our side. The first library from outside our catalogue is in our test set now.
New architectures were proven. Miners can write code that builds a custom graph, a genuinely new model shape, and it goes through the same rebuild and judge as everything else. This week we tested ensembles of those graphs, and the graph operations that mesh-based models depend on.
Their own AI agents. Launchpad, the mining front-end, works with whatever agent a miner prefers, and we’re about to test it end to end with AI agents mining every launch challenge.
Errors that tell you what to do. If something isn’t supported, the miner gets the exact operation and what’s needed. Not a refusal, and not a silent swap for a simpler model.
Novel architectures, tuned to a specific engineering problem, rebuilt and verified by the network. That’s the part of this field Bittensor can lead. Unbelievably excited about this.
5. We closed a fairness gap
Our judge used to grade models on a batch of hidden questions at once, so a model could in principle look across the exam while answering it.
Nobody exploited it. We found it ourselves. Every hidden question is now graded on its own. A model’s score is about the model.
6. Rewards got simple
Each Challenge has a king. Beat the king and you take that Challenge’s full share for 24 hours, then it halves every day until someone beats you.
“Beat” means beat it for real. Inside measurement noise, the king keeps the crown, so copying the winner and tweaking it doesn’t pay.
A king who improves their own model raises the bar but doesn’t reset their clock. A king who fails a required physics check loses the crown.
7. Real miners are already shaping it
A test net miner already reported a crash with graph ensembles, that there was no visibility into rebuild time, and a constant budget that charged ensembles several times over.
The crash was fixed and tested the same day. Constants are now counted once, and published Challenge data comes in as a proper input. Rebuild timing goes into miner feedback next.
We're build alongside miners. 💪
8. Five Challenges planned
We’re building toward five launch Challenges:
1. Motor: predicts an electric motor’s torque across its rotation angle (60 points per design),
- Competing against a magnetic field simulation.
2. Cooling: a cold plate. Predicts peak temperature, the temperature profile along the plate, and pressure drop.
- Competing against a thermal-fluid simulation.
3. Battery: fast-charging and ageing, including the plating limit, where charging too hard damages the cell. It’s the one with a real miner on it today.
- Competing against PyBaMM, the open-source battery simulator
4. Ribbed Bracket: a structural bracket’s vibration, its natural frequencies and modes.
- Competing against a finite-element solver (CalculiX).
5. Burst Thermal: how a part heats up under short bursts of power, against a temperature limit near 95 °C.
- Competing against a heat-transfer solver (Elmer).
Where we stand on these.
The first motor model just made it end to end while I was typing this out. LFG
The next 4 will follow.
What’s between us and mainnet
DONE - One model scored end to end through the real path JUST HAPPENED.
NEXT - Proof that “any model” means any model. A stock architecture, custom training, a hand-built graph, an ensemble, an outside library, and a second rebuild that matches.
(LET THE MINERS EAT, but rebuild their model before grading it)
THEN - All five challenges on same loop
THEN - The miner experience, tested like a miner.
A fresh install, then AI agents mining every Challenge through Launchpad, from research to payment.
(S/O @engyai@chutes_ai for making this affordable)
LAST - Testnet running exactly like mainnet: same code, same rules, same rewards.
Why this way
Every shortcut we skip now is a bug a miner or customer doesn’t find later.
We're testing thoroughly on disposable machines (S/O @TargonCompute@lium_io), fixing on development branches, and only shipping what’s proven.
We’re close. I’ll launch when those boxes are checked, not before.
Bittensor can absolutely help design the machines the world runs on.
That’s why we’re building Carbon. Physics AI to help engineers design better motors, cooling systems and other hardware, faster and at lower cost.
The goal is measurable improvement in real hardware.
Will TAO’s development consider landing in manufacturing?
Right now the focus is still on general inference and compute, and manufacturing isn’t the main narrative. But factories are where AI is most likely to generate real paying demand — quality inspection, production scheduling, predictive maintenance. These use cases can actually be measured in dollars.
@const_reborn how do you see this path? Bittensor won’t centrally decide the direction, but specialized industrial subnets can absolutely emerge. The hard parts are data privacy and edge deployment. Whoever first delivers real paying factory use cases will prove the model works.
Manufacturing cares about results on the floor, not concepts. bittensor:native
👀 Sneak peek at something we're working on for our community: Carbon Council.
The idea is to give our most committed holders and contributors a direct line to the team and a place to ask questions, share ideas and give feedback.
We’ll host monthly online stakeholder meetings covering recent developments and the near-term roadmap, followed by open Q&A and discussion.
Eligibility:
→ First 30 days after launch: hold 1,000 alpha.
→ After that: hold 1,000 alpha for more than 30 days, or 2,000 alpha for at least 14 days.
We’ll also invite active miners and other valuable contributors directly.
Several miners are already testing on testnet and giving Ryan useful feedback. Thank you. That’s exactly the kind of contribution we want to recognise through the Council.
Once live, holders will be able to sign up with their email, connect a wallet to verify eligibility and claim their Council seat.