@RohanBhanotAI@MIT_CSAIL A few studies have started quantifying the time and costs of checking AI outputs, but systematic evidence on verification across scientific workflows remains fairly limited. Hopefully, future studies will shed more light on this.
@RohanBhanotAI@MIT_CSAIL Whether the next dollar should go toward another attempt or human verification is an empirical question, and potentially a very consequential one for how scientific research is organized.
@RohanBhanotAI@MIT_CSAIL That’s exactly the tradeoff we need to understand! As generating candidate solutions becomes cheaper, the question shifts from “Can AI produce an answer?” to “How much does it cost to establish that the answer is correct?”
This is a result I’ve dreamt about for many years: *unpaired translation between images and text*
I thought it might be only slightly possible, the kind of thing you have to really squint at. But Dominik proved this wrong. You don’t have to squint. Worth looking for yourself:
@cibelerusso Dê uma olhada em nossos papers recentes, Cibele; acredito que eles possam contribuir de certa maneira com o projeto de vocês :). Um exemplo: https://t.co/rbsN7maEZ2
New paper with Kazimier Smith and @ProfNeilT at @MITFutureTech and @mit_ide: "Birth, Life, and Death of AI Models." We track the lifecycle of open-weight AI models using a weekly panel of the ~108k most-downloaded models on @huggingface (>53B downloads) https://t.co/GGxsmws4cO
@JMateosGarcia@JMateosGarcia , thanks for sharing!!! We’ve just posted our preprint with more details in case folks want to dive a little deeper into our results :) https://t.co/IT6HkEG3Nr
"AI is becoming a universal scientific solver." We hear it about AlphaFold, neural weather models, foundation models, and research agents.
Is it true? Does AI beat every method it replaces, or only some, and at what compute cost? We test this on 2,507 head-to-head comparisons 🧵
We offer a principle, not a verdict. Report the task, the baseline and the compute budget in the same breath as the performance claim. Otherwise, the number is just a headline.
w/ Emma Fu, @ProfNeilT (@MITFutureTech, @MIT_CSAIL)
https://t.co/MmbzcuQUtP
#AIforScience#SciML#AI
"AI is becoming a universal scientific solver." We hear it about AlphaFold, neural weather models, foundation models, and research agents.
Is it true? Does AI beat every method it replaces, or only some, and at what compute cost? We test this on 2,507 head-to-head comparisons 🧵
Where numerical cost data exist (only 45.3% of comparisons report numerical cost 😭), AI uses ~10× the compute of traditional statistics and ~600× less than scientific computing, on average.