Antonin SUMNER

was an architect, now Computer Vision Research Engineer

[Antonin SUMNER @BBrainkite]

@jeremyphoward @giffmana @kroscoo @dirkweissenborn Oh Yeah, I totally remember you mentioning it in a @fastdotai course and me applying this on a personal project.

[Grégory D'Angelo @gdangel0]

Vibe-coded this game this morning with 0 code edits.

Just hit ‘accept all’ in Cursor with Claude Sonnet 3.7 and @threejs 😎

[shadcn @shadcn]

For chatbot builders: Enter = new line. Cmd/Ctrl + Enter = submit. Please.

[Thomas Wolf @Thom_Wolf]

I shared a controversial take the other day at an event and I decided to write it down in a longer format: I’m afraid AI won't give us a "compressed 21st century".

The "compressed 21st century" comes from Dario's "Machine of Loving Grace" and if you haven’t read it, you probably should, it’s a noteworthy essay. In a nutshell the paper claims that, over a year or two, we’ll have a "country of Einsteins sitting in a data center”, and it will result in a compressed 21st century during which all the scientific discoveries of the 21st century will happen in the span of only 5-10 years.

I read this essay twice. The first time I was totally amazed: AI will change everything in science in 5 years, I thought! A few days later I came back to it and, re-reading it, I realized that much of it seemed like wishful thinking at best.

What we'll actually get, in my opinion, is “a country of yes-men on servers” (if we just continue on current trends). Let me explain the difference with a small part of my personal story.

I’ve always been a straight-A student. Coming from a small village, I joined the top French engineering school before getting accepted to MIT for PhD. School was always quite easy for me. I could just get where the professor was going, where the exam's creators were taking us and could predict the test questions beforehand.

That’s why, when I eventually became a researcher (more specifically a PhD student), I was completely shocked to discover that I was a pretty average, underwhelming, mediocre researcher. While many colleagues around me had interesting ideas, I was constantly hitting a wall. If something was not written in a book I could not invent it unless it was a rather useless variation of a known theory. More annoyingly, I found it very hard to challenge the status-quo, to question what I had learned. I was no Einstein, I was just very good at school. Or maybe even: I was no Einstein in part *because* I was good at school.

History is filled with geniuses struggling during their studies. Edison was called "addled" by his teacher. Barbara McClintock got criticized for "weird thinking" before winning a Nobel Prize. Einstein failed his first attempt at the ETH Zurich entrance exam. And the list goes on.

The main mistake people usually make is thinking Newton or Einstein were just scaled-up good students, that a genius comes to life when you linearly extrapolate a top-10% student.

This perspective misses the most crucial aspect of science: the skill to ask the right questions and to challenge even what one has learned. A real science breakthrough is Copernicus proposing, against all the knowledge of his days -in ML terms we would say “despite all his training dataset”-, that the earth may orbit the sun rather than the other way around.

To create an Einstein in a data center, we don't just need a system that knows all the answers, but rather one that can ask questions nobody else has thought of or dared to ask. One that writes 'What if everyone is wrong about this?' when all textbooks, experts, and common knowledge suggest otherwise.

Just consider the crazy paradigm shift of special relativity and the guts it took to formulate a first axiom like “let’s assume the speed of light is constant in all frames of reference” defying the common sense of these days (and even of today…)

Or take CRISPR, generally considered to be an adaptive bacterial immune system since the 80s until, 25 years after its discovery, Jennifer Doudna and Emmanuelle Charpentier proposed to use it for something much broader and general: gene editing, leading to a Nobel prize. This type of realization –"we've known XX does YY for years, but what if we've been wrong about it all along? Or what if we could apply it to the entirely different concept of ZZ instead?” is an example of out-side-of-knowledge thinking –or paradigm shift– which is essentially making the progress of science.

Such paradigm shifts happen rarely, maybe 1-2 times a year and are usually awarded Nobel prizes once everybody has taken stock of the impact. However rare they are, I agree with Dario in saying that they take the lion’s share in defining scientific progress over a given century while the rest is mostly noise.

Now let’s consider what we’re currently using to benchmark recent AI model intelligence improvement. Some of the most recent AI tests are for instance the grandiosely named "Humanity's Last Exam" or "Frontier Math". They consist of very difficult questions –usually written by PhDs– but with clear, closed-end, answers.

These are exactly the kinds of exams where I excelled in my field. These benchmarks test if AI models can find the right answers to a set of questions we already know the answer to.

However, real scientific breakthroughs will come not from answering known questions, but from asking challenging new questions and questioning common conceptions and previous ideas.

Remember Douglas Adams' Hitchhiker's Guide? The answer is apparently 42, but nobody knows the right question. That's research in a nutshell.

In my opinion this is one of the reasons LLMs, while they already have all of humanity's knowledge in memory, haven't generated any new knowledge by connecting previously unrelated facts. They're mostly doing "manifold filling" at the moment - filling in the interpolation gaps between what humans already know, somehow treating knowledge as an intangible fabric of reality.

We're currently building very obedient students, not revolutionaries. This is perfect for today’s main goal in the field of creating great assistants and overly compliant helpers. But until we find a way to incentivize them to question their knowledge and propose ideas that potentially go against past training data, they won't give us scientific revolutions yet.

If we want scientific breakthroughs, we should probably explore how we’re currently measuring the performance of AI models and move to a measure of knowledge and reasoning able to test if scientific AI models can for instance:
- Challenge their own training data knowledge
- Take bold counterfactual approaches
- Make general proposals based on tiny hints
- Ask non-obvious questions that lead to new research paths

We don't need an A+ student who can answer every question with general knowledge. We need a B student who sees and questions what everyone else missed.

---

PS: You might be wondering what such a benchmark could look like. Evaluating it could involve testing a model on some recent discovery it should not know yet (a modern equivalent of special relativity) and explore how the model might start asking the right questions on a topic it has no exposure to the answers or conceptual framework of. This is challenging because most models are trained on virtually all human knowledge available today but it seems essential if we want to benchmark these behaviors. Overall this is really an open question and I’ll be happy to hear your insightful thoughts.

[Sayash Kapoor @sayashk]

At a recent event, an AGI company employee said 'You should read our public documents *very* carefully. The next 18-24 months are *especially* critical.'

When I said I don't think insiders have any greater insight than outsiders, people looked at me as if it were blasphemy.

[Antonin SUMNER @BBrainkite]

💯

[Hamel Husain @HamelHusain]

My thesis is LLMs significantly boost your motivation to work on something. Getting a shitty prototype fast is immensely motivating

Code and time are often not the bottleneck, motivation is the bottleneck.

[Firecrawl @firecrawl]

Introducing our official LLM.txt Generator API 📃

Concatenate any website into a single text file that can be fed into any LLM.

With our new alpha endpoint you can quickly generate llms.txt and llms-full.txt files for any website.

[Antonin SUMNER @BBrainkite]

@karpathy I think there is something to be said that we aren’t able to just put all those tabs in a bookmark folder and re-open them later when we actually need them. We are just certain that if we close them we’re going to loose them forever.

[Antonin SUMNER @BBrainkite]

@karpathy Did you end up finding which S2T solution was using Simon Willson? 😅

[Antonin SUMNER @BBrainkite]

@_philschmid Interesting thought about how to tune the reward function to encourage the use of non-numeric tokens. Another approach would be to mix or alternate GRPO steps with simple next-token prediction steps to maintain some « verbosity » of the model throughout the training.

[tomaarsen @tomaarsen]

Alibaba just published their latest GTE models based on ModernBERT-base! An embedding model *and* a reranker, each outperforming much larger models.

Both are Apache 2 licensed, and the former is nr.1 on MTEB for <300M param out of all models that don't need setup.

Details in 🧵

[Antonin SUMNER @BBrainkite]

@karpathy Generalized First person character syndrome going to get on a new level

[Antonin SUMNER @BBrainkite]

@francoisfleuret « Ne pas vendre la peau de l’ours avant de l’avoir tué » 😂

[Antonin SUMNER @BBrainkite]

@jxmnop Banger

[Antonin SUMNER @BBrainkite]

@jeremyphoward Seems fine to me

[Andrej Karpathy @karpathy]

To help explain the weirdness of LLM Tokenization I thought it could be amusing to translate every token to a unique emoji. This is a lot closer to truth - each token is basically its own little hieroglyph and the LLM has to learn (from scratch) what it all means based on training data statistics.

So have some empathy the next time you ask an LLM how many letters 'r' there are in the word 'strawberry', because your question looks like this:
👩🏿‍❤️‍💋‍👨🏻🧔🏼🤾🏻‍♀️🙍‍♀️🧑‍🦼‍➡️🧑🏾‍🦼‍➡️🤙🏻✌🏿🈴🧙🏽‍♀️📏🙍‍♀️🧑‍🦽🧎‍♀🍏💂

Play with it here :)
https://t.co/pFQGZIAW1k

[Andrej Karpathy @karpathy]

Jagged Intelligence

The word I came up with to describe the (strange, unintuitive) fact that state of the art LLMs can both perform extremely impressive tasks (e.g. solve complex math problems) while simultaneously struggle with some very dumb problems.

E.g. example from two days ago - which number is bigger, 9.11 or 9.9? Wrong.
https://t.co/dUrR6wm8GC

or failing to play tic-tac-toe: making non-sensical decisions:
https://t.co/XarwfUBtod

or another common example, failing to count, e.g. the number of times the letter "r" occurs in the word "barrier", ChatGPT-4o claims it's 2:
https://t.co/xpffK2r0pv

The same is true in other modalities. State of the art LLMs can reasonably identify thousands of species of dogs or flowers, but e.g. can't tell if two circles overlap:
https://t.co/HCXxBxosAu

Jagged Intelligence. Some things work extremely well (by human standards) while some things fail catastrophically (again by human standards), and it's not always obvious which is which, though you can develop a bit of intuition over time. Different from humans, where a lot of knowledge and problem solving capabilities are all highly correlated and improve linearly all together, from birth to adulthood.

Personally I think these are not fundamental issues. They demand more work across the stack, including not just scaling. The big one I think is the present lack of "cognitive self-knowledge", which requires more sophisticated approaches in model post-training instead of the naive "imitate human labelers and make it big" solutions that have mostly gotten us this far. For an example of what I'm talking about, see Llama 3.1 paper section on mitigating hallucinations:
https://t.co/pjuxoIOJCY

For now, this is something to be aware of, especially in production settings. Use LLMs for the tasks they are good at but be on a lookout for jagged edges, and keep a human in the loop.

[Antonin SUMNER @BBrainkite]

« take the summer to disconnect a bit » she said

[Ahmad Al-Dahle @Ahmad_Al_Dahle]

With today’s launch of our Llama 3.1 collection of models we’re making history with the largest and most capable open source AI model ever released. 128K context length, multilingual support, and new safety tools. Download 405B and our improved 8B & 70B here. https://t.co/F8cI1bUL8h

[Antonin SUMNER @BBrainkite]

@ducha_aiki The steamdeck has little to no issues running windows games on linux. I’m sure we’ll very quickly get back on our feets for those

[Antonin SUMNER @BBrainkite]

Seriously, can we agree as a civilization to put #Windows away and all go to #Linux . Chatbots will help the late adopters to catch up and we’ll all live happier.