Fadli Damara

ML PhD student @ml_tuberlin & @bifoldberlin. Diffusion post-training. Understanding the core of intelligence. Views aren’t my employer’s.

[Fadli Damara @fadlidamara]

@jdeschena @ValentinDeBort1 👀

[Andrew Gordon Wilson @andrewgwils]

As much as AI is impressively progressing, current systems are appallingly poor at conducting end-to-end scientific research. 1/4

[Ulugbek S. Kamilov @prof_kamilov]

Writing papers teaches you that every elegant idea comes with an inelegant appendix.

[Jean Kossaifi @JeanKossaifi]

PSA for anyone preparing a NeurIPS rebuttal (or any OpenReview response).

Draft in Google Docs, go to Tools > Preferences > Enable Markdown.
When you’re done you can simply select the text, right-click, and “Copy as Markdown”.

No more fixing formatting by hand. You’re welcome!

[Andrew Conner @connerdelights]

Terence Tao posted his ChatGPT session trying to understand the Jacobian conjecture counterexample. It's so lovely reading a slice of how his mind works, the connections he's making, etc.

https://t.co/wu6CcAk3V6

[Aryeh Kontorovich @aryehazan]

LLMs should be viewed as an efficient search method through a certain kind of structured space

[codila @0xCodila]

Kimi CEO Zhilin Yang:

"Claude didn't win on reasoning - they bet everything on agents

but the layer everyone skips - a great agent needs a great base model, that's all we do at Kimi 3 "

in 90-min workshop he explains why the smartest agent still fails - if you can't configure it correctly

his one big idea: most people are still solving the old one

"the real goal? we want K2 to help build K3 - without agent skills, that's impossible"

watch & bookmark - then learn the article on best agent system ↓

[Yi Ma @YiMaTweets]

Just came back from ICML. Gave a keynote at the Foundation of Deep Generative Models Workshop, in which I stated that Intelligence should be a scientific subject, arguably much more significant than Physics. It is high time we study it with the same scientific methodology and mathematical rigor as modern physics, instead of always at the level of being empirical, meta physical, meta mathematical, philosophical or speculative... This remains as the biggest opportunity ever for young scientists. To my knowledge, this is not the focus of any of the frontier "AI" companies.

[David Tse @dntse]

A colleague reached out recently. His field - information theory - is trying to figure out its future.

But the more interesting question underneath that one: what is the future of research itself?

If AI can run experiments, check proofs, and synthesize literature, what is the researcher's job?

I don't have the answer. But I'm teaching a large undergraduate course partly to find out.

What we've been teaching engineering students for 30 years........ is it still the right thing to teach?

[Chubby♨️ @kimmonismus]

Google DeepMind published a 60-page paper mapping the road from AGI to superintelligence, written by Hutter, Legg, and Genewein. No hype, just a sober analysis

The paper uses three levels. AGI = roughly average human performance across most cognitive tasks. ASI = a system that beats large, well-coordinated groups of human experts across virtually everything (their bar: tens of thousands of experts working ten years on one problem). Universal AI / AIXI = the theoretical ceiling, uncomputable, only approachable from below.

Then they explore the question of how this could be achieved:

Scaling compute, models, and data, the continuation of the trend that drove the breakthrough so far. It is the only path with historical data available for extrapolation. The core question: Does quantity transform into quality? Even if individual models plateau, the sheer act of running millions of faster AGI instances could trigger the leap. (A quick aside: that is a fascinating philosophical idea. It always reminds me of Hegel’s dialectic, the notion that quantity transforms into quality. We ought to start drawing on philosophical theories to make sense of the future.)

Algorithmic paradigm shifts: a genuine break from the transformer pretraining paradigm. New architectures, new learning methods. However, hard to predict by definition.

Recursive self-improvement: AI accelerates AI research, which produces better AI, which accelerates research further.

Multi-agent coordination: superintelligence emerges from large collectives of AGI agents working together, like automated corporations or AI economies. Collective intelligence potentially far exceeding any individual model.

The authors naturally point to what I repeatedly describe as the biggest bottleneck: energy. I recently linked to a few graphs showing, on the one hand, the extent to which energy is already becoming a problem and, on the other, how China dominates the expansion of both nuclear and solar energy in the global race. But the authors also address a profound shift in the world of work in a post-AGI era. I would say this is a reality we must face.

So, it is not just about scaling, but also about whether the underlying conditions - such as energy and hardware - can be effectively established.

Six things that could slow or stop all of this:

The data wall. Quality training data runs out, possibly before the end of this decade.

Resource demand grows too fast. Energy, chips, rare earths, investment. The physical infrastructure can't scale arbitrarily.

The neural paradigm hits a ceiling. Pretrained transformers plus fine-tuning may not be enough to reach AGI, let alone go beyond it.

Research gets harder. Keeping Moore's law going already needs 18x more researchers than in the 1970s. Ideas are genuinely harder to find as fields mature.

The abstraction barrier. Models trained on human concepts may never invent new ones from scratch. Saturating GPQA or SWE-bench shows mastery of what humans already worked out, not the ability to go beyond it. Train only on pre-Newtonian physics and you won't reason your way to relativity.

Deliberate slowdown. Regulation, accidents, public backlash. Real, but likely countered by the competitive pressure between companies and nations.

I think it’s great that Google is addressing questions such as which paths they believe lead to AGI, what the road to ASI might look like, what challenges will arise, and much more. Overall, however, it sounds to me like all of this could actually succeed, making it, in that sense, a call to discuss and reflect on the consequences.

[Yi Ma @YiMaTweets]

For intelligence, compression is not the goal. It is a means to an end. The true goal is to gain information that helps reduce uncertainty, which is entirely measurable. For any particular data of interest, to obtain a most informative representation of its distribution (also called memory or knowledge), an intelligent system tries to learn the most effective and efficient compressing operations (say layers of a network). This is what I have been saying: We learn to compress, and we compress to learn! This is precisely the main theme of our new open book.

[Tiberiu Mușat @Tiberiu_Musat_]

Why does deep learning generalize? What does weight decay really do? Can algorithmic information theory address these questions?

In my latest preprint, I give a proof that the minimum neural weight norm matches the minimum program length (aka Kolmogorov Complexity), up to a logarithmic factor. In other words, the neural network with the smallest possible weight norm (that fits the data) must encode the shortest program (that fits the data).

The result only holds for fixed-precision neural nets: infinite precision nets can store infinite information with finite (small) weights.

https://t.co/eMZIGQDf2f

[𝚐𝔪𝟾𝚡𝚡𝟾 @gm8xx8]

Shannon Scaling Law tries to model where scaling stops being monotonic.

The setup treats LLM training like a noisy channel:
- model size = bandwidth
- tokens = signal
- data/model/perturbation effects = noise

The target is the failure mode classical power laws miss: loss improves with scale, then degrades under overtraining, quantization, SFT perturbations, or injected Gaussian noise.

The law fits U-shaped loss basins across Pythia and OLMo2 under:
- Gaussian noise
- GPTQ quantization at 4/3/2 bit
- SFT on GSM8K, SiQA, and StarCoder-Python

On extrapolation, fitting only ≤6.9B Pythia models and ≤180B tokens predicts the unseen 12B model up to 307B tokens at pooled R² = 0.847. Monotonic OpenAI/Chinchilla-style laws fail in the same setting.

The claim is not that scaling stops working. It is that scaling depends on signal-to-noise. More parameters or more tokens help only while the signal grows faster than the noise.

Paper: https://t.co/qKVcIfjn9g

[Andrew Gordon Wilson @andrewgwils]

Just like the no free lunch theorems, the data processing inequality (DPI) is irrelevant to the practice of machine learning, and that’s okay. It doesn’t mean we can��t develop info theory to be descriptive of practice.

[Ravid Shwartz Ziv @ziv_ravid]

My two cents on why information theory doesn't quite work in the real world: as someone who's been arguing with people about IT and its connection to ML since 2016(!), I mostly agree with Alex.
(At least) two problems show up when you try to use information theory in ML. First, the distance between theory and reality. Nothing is clean, everything is noisy, full of engineering tricks, and you have to estimate everything. We run experiments with information estimators whose quality we don't really know.
But the bigger problem (and I hope Shannon will forgive me): information theory isn't really about learning. In the classical setting you're in an idealized world where everything is given, with no optimization and no learning. There have been attempts to change this (Stefano Ermon's usable information, for one), and many works took inspiration from IT, but actually applying its tools has had limited success. So, next time that people ask you "Does Information theory 'explain AI you can say, "No, and it doesn't supposed..."

[Alex Dimakis @AlexGDimakis]

https://t.co/qqVULIZAWD

[Peyman Milanfar @docmilanfar]

TLS is an elegant extension of OLS when both dependent and indep variables are noisy. TLS looks just like ridge regression (aka regularized OLS) except it's “de-regularized”

TLS solution is less numerically stable than OLS since both dependent & independent variables are noisy

[Peyman Milanfar @docmilanfar]

being wrong is the best way to get cited a lot

[Peyman Milanfar @docmilanfar]

“Our keynote speaker needs no introduction”

the introduction:

[Manuel Faysse @ManuelFaysse]

Getting (first author) papers in at ICML, NeurIPS and ICLR in a single calendar year is called a "Grand Slam" for AI PhD students apparently. I won't be the one attempting that, I like sleeping too much I think.