So stoked to be going to my first #NeurIPS!! It’s crazy that in 10+ years of robotics and AI, I've never been to the great Lollapalooza of Machine Learning. 🥳🎆
I’m presenting my Frankensteinian efforts to stitch together parts of different neural networks! 🧪
For years people in NeuroAI have been wondering what the relationship between CNNs and transformers is as brain models, but the ground has been kind of muddy. Now I think we are starting to see underlying situation more clearly. Transformers > CNNs -- in interesting ways!
@idewanck I do think I turned mine off the instant that changed. Don't recall but I just now tried one and it went to the browser.
All the apps want to do that so they can track you and it drives me up the wall.
@AlexKrentsel@rikiyatakehi@jayden_teoh_@lateinteraction@a1zhang Why is this an obvious next step??
Why is "understand the edits it already made" not the obvious next step? Or "try to map out what kinds of edits it can and can't do"? Or... any number of other things?
That weight-editing experiment that's gone viral? It's not actually editing the weights—it just wrote a completely separate controller. You can randomize the other weights or set them to 0, and it still works. (According to the claim below.)
We asked Claude Opus 5.5 and GPT-6 Astra to fix a small trained driving network so the car reaches the goal. Their working fixes didn't need what the network learned. https://t.co/v65k9lgyRG
@IntuitMachine Hey Carlos, where is this screenshot from?
B/c I'm curious about:
- How reliable are these stats?
- What does "Weighted CS student" mean?
- Some of the heaviest GPU users are *not* CS students. Is this accounted for?
- Etc.
I think the last month might have been the biggest month in (zeroth-order) optimization history.
In the 10 days since we released the Noise speedrun, the time to pretrain a transformer went from 1.76 hours to 1.6 seconds, a 4,000x (!!!) speedup. Dust itself was up to 10,000x more compute-efficient than the previous state-of-the-art.
Not only can we pretrain a transformer without backprop, it's much more efficient than anyone would've expected. Search fking works!
Now, scaling is the biggest open question. Some of the tricks that brought the time down to 1.6 seconds will simply not scale. To discard those, we'll be scaling the dataset every time it gets overfit, starting with 30M tokens instead of 10M.
Huge thanks to @jayden_teoh_, @spicey_lemonade, @RobertTLange, @EmanueleRodola, @ChunyuanDeng, @LuochengHuman, @Alaxarr, @pavleculafic, @IsaacYong13, Euraba and areu01or00 (GitHub), for the contributions. And @cs_serdar for gracefully maintaining the speedrun and @bishmdl76 for helping out.
Expect a lot more from us over the coming weeks.
My advisor just asked me, "Why is AI good at summarizing some documents, but so bad at scientific papers?" I posed that it's just *condensing* the text, when I actually need it to *translate* the ideas for me.
And lo and behold, a paper demonstrating the info density of AI text!
If you've noticed a distinct dialect emerging in frontier AI models and struggled to understand what they're talking about, you're not crazy!
We turned to psycholinguistics to investigate what makes the writing style so weird, and why this might be happening
🚨New preprint! 🧵
This textbook looks very interesting, and I skipped to the end and found a pretty compelling list of research directions. Some slightly hot takes here but a solid outlook, IMO.
The lectures of my new course "Principles of Deep Representation Learning" are now available at the Youtube site: https://t.co/9eCNxhxWLM with new lectures continuing to be added there. I will open-source all the slides later in the textbook website: https://t.co/ihPBCkI3x5
I have always taken paper reviewing very seriously. This time, all papers in my ICLR batch were completely AI-generated. Just reading the first paragraph of each introduction makes it blatantly clear. 100% in favor of aggressive desk rejections and blunt reviews for such cases!
As many have said before: why should we bother spending valuable human time reviewing papers that the authors might not even have read themselves end-to-end before submitting. I found it largely pointless to write long and detailed reviews as it is highly questionable whether the authors even care.
Also, it is wild to me that authors often claim in their AI disclosure statements that they used AI to "[...] improve the clarity of the manuscript." Sometimes I really wonder if they truly believe that's true. I would really like to know how overly formulaic, jargon-filled sentences without proper definitions or logical flow improve clarity.
In many ways, it is fascinating that language models have been around for a while now and still they struggle so much at clear writing and communicating ideas effectively.
@jeffreyseely Foundation models for neuroscience?
Maybe also the fact that we can now reliably train spiking nets on complex tasks via surrogate gradients and eligibility traces? (But I'm not really in this field so I don't actually know.)
I found this refreshingly honest and forthright statement about AI use in a recent paper! IMO this is exactly the right way to do it. https://t.co/0umetGS2cV
@OsSully@arindube@PeteButtigieg I think the chart shows the difference relative to states that did not raise MW. So it's showing that job growth was almost the same as it would have been absent the govt interference, if you assume the other states represent a reasonable counterfactual.
This is a consequence of the loss not gradient descent. E.g. by swapping PPO for a GFlowNet objective you can turn an LLM into an exorbitantly expensive random number generator.
We need tools that are more than a chat interface, and in the past year we've finally gotten that in the form of coding agents.
Who will have the necessary expertise in both AI and another domain, s.t. they can build an effective tool for that other domain??
An interesting aspect of the AI revolution is that its promise is becoming real mostly only in programming, so far. One factor not mentioned below is that AI developers get to "dogfood" their own product. It's the only instance of an AI product that truly delights the user.
Dusting off this tweet from April because this gap in shared understanding of LLM capability is *widening*. It's now less "two groups of people speaking past each other" and more a sharp funnel.
- Napkin math somewhere around 6B people (~75% of the population) have barely come in contact with LLMs at all.
- Around 1-2B (~20%) are casual and infrequent users of free-tier ChatGPT-like products. This group sees derpy chatbots and treats them a bit like a better Google search, a writing aid, or etc. Maybe an agent tries to book you a flight. I have non-tech friends who (reasonably, imo) say they have not much use for it at all. Even many professionals outside of math&code are in this tier. For example, execs and many other functions spend a lot of their time talking to other people, so while they understand what is happening intellectually, it is still second-hand and a bit abstract.
- Now we get to professional use of frontier-grade LLMs in math&code. Somewhere around 20M people (0.2%) see first-hand that large, complex projects that used to take them weeks/months can now be completed by agents with a prompt. Building apps, copying apps, translating apps, decompiling apps from binaries... This has all happened very quickly and recently - less than 1 year ago, I was writing code manually by hand, typing memorized computer code commands into a code editor character by character, occasionally pressing Tab to autocomplete a little chunk of code.
- And finally we get to the ~5,000 people (~0.00006%) with access to frontier-grade systems internally. The external world has seen the preview. It looks like swarms of thousands of agents collaborating over weeks on software mega projects: minting zero days, running cyber attacks and defenses at machine speeds, discovering new science, advancing the frontier of mathematics. Things that would have taken top professionals in the industry years of work. Meanwhile, human review and comprehension are starting to fall behind. For example, people are still involved in the "archeology" of the OpenAI-HF incident from many months ago. Mathematicians may be poring over the 722 manuscripts on frontier mathematics for a while.
The funnel is driven by a combination of factors:
1. The impact scales with ambition, problem size, and horizon. A question with a paragraph answer barely stresses the system. You need a reservoir of big, difficult problems that you really care about. This aspect drives the consumer / professional dimension of the funnel.
2. The jaggedness of the system (which I have written about a lot separately). Capability peaks in domains that are digital, verifiable and economically valuable. This is because LLM capability emerges from reinforcement learning on verifiable rewards on a curated environment mixture driven by revenue potential. This aspect primarily drives the area (e.g. math&code) dimension of the funnel.
3. Access. Free-tier, paid-tier, internal.
So this is the weirdness of the moment. The general public has mostly not interacted with these systems. When they have, it looks like a derpy chatbot. The majority of professionals still see only a modest uplift. And a small sliver of professionals are experiencing the vertigo of the curve going vertical. And it is all happening at the same time.
I worry that we won't have such an effective feedback loop in any other industry... Instead, developers just try to replace the jobs of those people, because they don't understand what is really *wanted* in those industries and what would actually add economic value.