Duc Nguyen Huu

[Duc Nguyen Huu @ducnh279]

Nothing new under the sun!

[Jürgen Schmidhuber @SchmidhuberAI]

A recent NYT article by @CadeMetz [1] claims that neural network distillation was first developed in 2015 by a team of Google researchers including G. Hinton [2]. Not true! I published the technique in 1991 [3]. [2] failed to cite [3]. See the overviews [4][6].

The NYT article [1] states: "... distillation is an effort to copy the behaviour of the system, as opposed to copying the text verbatim." That's right! Distillation enables non-trivial forms of plagiarism [5].

Annotated References (easy to find on the web):

[1] Cade Metz, New York Times (6 July 2026). "American AI companies Say Chinese Copycats are Quickly Catching Up"

[2] O. Vinyals, J. A. Dean, G. E. Hinton (2015). Distilling the Knowledge in a Neural Network. arXiv:1503.02531. [2] did not cite the 1991 neural network distillation procedure [3].

[3] J. Schmidhuber (JS). Learning complex, extended sequences using the principle of history compression. Neural Computation, 4(2):234-242, 1992. Based on TR FKI-148-91, TUM, 1991.
See Section 4 on the "conscious" chunker and a "subconscious" automatiser which introduced a general principle for transferring the knowledge from one neural net (NN) to another. Suppose a teacher NN has learned to predict (conditional expectations of) data, given other data. Its knowledge can be compressed into a student NN, by training the student NN to imitate the behavior of the teacher NN (while also re-training the student NN on previously learned skills such that it does not forget them). In 1991, I called this "collapsing" or "compressing" one NN into another. Today, this is widely used, and also referred to as "distilling" or "cloning" the behavior of a teacher NN into that of a student NN. It even works when the NNs are recurrent and operate on different time scales.

[4] JS. Who invented knowledge distillation with artificial neural networks? Technical Note IDSIA-12-25, IDSIA, Nov 2025. https://t.co/w0WhIVGXQx

[5] JS. How 3 Turing awardees republished key methods and ideas whose creators they failed to credit. Technical Report IDSIA-23-23, Swiss AI Lab IDSIA, 2023 (updated 2025).

[6] @hardmaru & JS (2026). Munich 1991: the Roots of the Current AI Boom. With a preface by David Ha.

[Duc Nguyen Huu @ducnh279]

@cosminnegruseri @JFPuget my intuition is that this is because Transformers contain many linear layers with 2D weight matrices. The Muon optimizer is specifically designed to optimize 2D weight matrices, so it's naturally a good fit for Transformer architectures.

[Duc Nguyen Huu @ducnh279]

@bogoconic1 Reasoning language models can do it too!

[Duc Nguyen Huu @ducnh279]

@rasbt Without your interest in LLMs and your perseverance, I don't think we'd be seeing this tweet. Congrats on the new book!

[Duc Nguyen Huu @ducnh279]

@rasbt Wonderful article, as always!

I'd love to learn more about the agent literature. Could you recommend a few good papers to start with, or perhaps share some of your favorite ones?

[Duc Nguyen Huu @ducnh279]

@prior_labs I wish TabPFN would beat supercomputer's prediction!

https://t.co/YuLfTjiOQ9

[Duc Nguyen Huu @ducnh279]

@GaelVaroquaux @andrewgwils RealMLP is also an inspiring story in my eyes!

[Duc Nguyen Huu @ducnh279]

@NotDrAbeFroman @kimmonismus An average person can get a sub-10 solve if they put in enough time and practice. So I dont believe in his talent but hard work in the field

[Duc Nguyen Huu @ducnh279]

@siddsax This is probably made by an Indian guy!

[Duc Nguyen Huu @ducnh279]

@rasbt Seb is always my GOAT!

[Duc Nguyen Huu @ducnh279]

@DHolzmueller Do you have any plans for the prior as well?

I read the paper and know you were mainly responisble for developing the prior! You might be the best one for ... 😂

[Duc Nguyen Huu @ducnh279]

@a_erdem4 Nice! reading ur write-up really made me interested in participating in future application-oriented hackathons.

[Duc Nguyen Huu @ducnh279]

@kagglingdieter Agreed! For TFMs/TSFMs, the prior development should be guided by maximizing dataset diversity while encoding useful inductive biases and preserving computational efficiency!

[Duc Nguyen Huu @ducnh279]

@DHolzmueller Hi, I tried TabICLv2 in a community tabular competition.
Just simply fit the model and predict, your TFM generalizes quite well on both the public and private lb! I’m not concerned about decontamination in this case, looking forward to the upcoming improvements to TabICL 👍

[Duc Nguyen Huu @ducnh279]

@cis_female Woww this is a great lesson for training NNs! I am having problem with train a neural net on a small and noisy dataset, I'd think sampling a batch strategically might give model more meaningful gradients and better optimization 😂😅

[Duc Nguyen Huu @ducnh279]

@Kinosuke_sophi Congrats @Kinosuke_sophi on the solo gold🥇🪙 Hard work finally pays off!

Look forward to your write-up!

[Duc Nguyen Huu @ducnh279]

@ethylene_66 Nice! I'm quite sure he will be very willing to give you detailed answers, even though he is not really an expert in NLP/LLMs field.

[Duc Nguyen Huu @ducnh279]

@KURISEW @Kinosuke_sophi Sometimes, public discussion can be misleading haha! Let's give it a try :))