Grégoire Delétang

Intelligence is good statistics

[Yun-Ta Tsai @yunta_tsai]

Many people think any given ML project is 99% training.

In reality, it’s 50% evaluation, 40% data cleaning, 8% integration, and 2% training.

The first two set the noise floor for learning. No ML magic matters; the model cannot lower the noise floor, as that’s the optimal bound of Shannon encoding of your data.

Thus, not a single day goes by without me thinking about ontology. Even the old labels have to be constantly reviewed.

[Grégoire Delétang @gregdeletang]

Claude code is actually an insane tool, I can only say congrats to the team

[Dimitris Papailiopoulos @DimitrisPapail]

512 parameters: a new top scorer for 10-digit addition with transformers!

Who can beat it?

[Grégoire Delétang @gregdeletang]

@DimitrisPapail Cool stuff! I think the challenge would be even more interesting with a scaling law num parameters versus number of digits (at iso ~100% accuracy)

[Google DeepMind @GoogleDeepMind]

Genie 3 🤝 @Waymo

The Waymo World Model generates photorealistic, interactive environments to train autonomous vehicles.

This helps the cars navigate rare, unpredictable events before encountering them in reality. 🧵

[Grégoire Delétang @gregdeletang]

I find that LLMs went much further than I initially thought 3-4 years ago. My impression is that pretraining enables many non trivial things to work, that would never otherwise (test-time "thinking", rlhf, fine-tuning on non trivial targets, interaction with tools etc). Am I the only one to be positively surprised?

[Grégoire Delétang @gregdeletang]

@jxmnop Language modelling is compression. Lossy (weights) and lossless (sequence).

[Grégoire Delétang @gregdeletang]

@zhaisf Back to 1995... How many people still discover NNs for the first time?

[Grégoire Delétang @gregdeletang]

If you pass a sequence of constant tokens to a transformer with rotary encodings, it will return a constant output distribution, because the attention values are constant. Idk if someone spotted this "bug" before?

[Grégoire Delétang @gregdeletang]

We slowly realize that humans do not, in fact, "generalize"

[Grégoire Delétang @gregdeletang]

Interesting effect, I think I saw you mentioned it in one of your tweet indeed. Yes my point is that ultimately even with cot, the model must be pretrained on a sufficiently large context length L, that contains at most L*log2(vocab_size) bits. That's enough to solve most problems, but that's still a hard limit.

[Grégoire Delétang @gregdeletang]

@DimitrisPapail Your work is still interesting for sure, but what I mean is that it doesn't solve the fundamental problem of the transformer architecture, which is that you must train it with a large enough context window.

[Grégoire Delétang @gregdeletang]

@DimitrisPapail I mean you could train with a longer context window yes, with even just some fine-tuning. But testing with a longer context window experimentally doesn't work, it's what people have been struggling with for a long time (i.e. length generalization).

[Grégoire Delétang @gregdeletang]

@_akhaliq @mhutter42 haven't checked it but mmh it smells a lot like Solomonoff's reference UTM

[Grégoire Delétang @gregdeletang]

@nouhadziri @DimitrisPapail It also makes me think about universality: is it really the right concept? Couldn't we get, say, a proof of the RH in a few years, with a model that can't sum and add two numbers?

[Grégoire Delétang @gregdeletang]

@nouhadziri @DimitrisPapail that comes back regularly. But when will it finally be solved?

[Grégoire Delétang @gregdeletang]

GPU hours is the new bitcoin

[François Fleuret @francoisfleuret]

Here you know you should pull up the logit of the class you proposed when you get +1 and push it down with -1, but that's it. You have to "poke around" until you get a +1.

So in the classification setup you get ~log_2(N) bits of information, and in the RL case 1 bit.

4/4