Azamat Omuraliev

✨ vibes check ✨ ai engineer @typetone_ai. gf @_yana_mars.

[Azamat Omuraliev @azamatomu]

new midjourney editor good

[Azamat Omuraliev @azamatomu]

Stay tuned for more!

[Azamat Omuraliev @azamatomu]

we're cooking up an LLM writing leaderboard, and this is why 🧵

[Azamat Omuraliev @azamatomu]

Why do LLMs suck at writing? RLHF + fine tuning on reasoning are possible hints

[Azamat Omuraliev @azamatomu]

@helloiamleonie Hard to make them adhere to negative prompts (ie dont do this, dont use this)

[Azamat Omuraliev @azamatomu]

Didn’t disappoint 🦞

[Jürgen Schmidhuber @SchmidhuberAI]

The #NobelPrizeinPhysics2024 for Hopfield & Hinton rewards plagiarism and incorrect attribution in computer science. It's mostly about Amari's "Hopfield network" and the "Boltzmann Machine."

1. The Lenz-Ising recurrent architecture with neuron-like elements was published in 1925 [L20][I24][I25]. In 1972, Shun-Ichi Amari made it adaptive such that it could learn to associate input patterns with output patterns by changing its connection weights [AMH1]. However, Amari is only briefly cited in the "Scientific Background to the Nobel Prize in Physics 2024." Unfortunately, Amari's net was later called the "Hopfield network." Hopfield republished it 10 years later [AMH2], without citing Amari, not even in later papers.

2. The related Boltzmann Machine paper by Ackley, Hinton, and Sejnowski (1985) [BM] was about learning internal representations in hidden units of neural networks (NNs) [S20]. It didn't cite the first working algorithm for deep learning of internal representations by Ivakhnenko & Lapa (Ukraine, 1965)[DEEP1-2][HIN]. It didn't cite Amari's separate work (1967-68)[GD1-2] on learning internal representations in deep NNs end-to-end through stochastic gradient descent (SGD). Not even the later surveys by the authors [S20][DL3][DLP] nor the "Scientific Background to the Nobel Prize in Physics 2024" mention these origins of deep learning. ([BM] also did not cite relevant prior work by Sherrington & Kirkpatrick [SK75] & Glauber [G63].)

3. The Nobel Committee also lauds Hinton et al.'s 2006 method for layer-wise pretraining of deep NNs (2006) [UN4]. However, this work neither cited the original layer-wise training of deep NNs by Ivakhnenko & Lapa (1965)[DEEP1-2] nor the original work on unsupervised pretraining of deep NNs (1991) [UN0-1][DLP].

4. The "Popular information" says: “At the end of the 1960s, some discouraging theoretical results caused many researchers to suspect that these neural networks would never be of any real use." However, deep learning research was obviously alive and kicking in the 1960s-70s, especially outside of the Anglosphere [DEEP1-2][GD1-3][CNN1][DL1-2][DLP][DLH].

5. Many additional cases of plagiarism and incorrect attribution can be found in the following reference [DLP], which also contains the other references above. One can start with Sec. 3:

[DLP] J. Schmidhuber (2023). How 3 Turing awardees republished key methods and ideas whose creators they failed to credit. Technical Report IDSIA-23-23, Swiss AI Lab IDSIA, 14 Dec 2023. https://t.co/Nz0fjc6kyx

See also the following reference [DLH] for a history of the field:

[DLH] J. Schmidhuber (2022). Annotated History of Modern AI and Deep Learning. Technical Report IDSIA-22-22, IDSIA, Lugano, Switzerland, 2022. Preprint arXiv:2212.11279. https://t.co/Ys0dw5hkF4 (This extends the 2015 award-winning survey https://t.co/7goTtI5Uwv)

[Azamat Omuraliev @azamatomu]

waiting for what @SchmidhuberAI has to say 🍿 🍿 🍿

[Azamat Omuraliev @azamatomu]

@kimmonismus Gpt 4o image generation still hasn’t been released… ⏳⏳⏳

[Azamat Omuraliev @azamatomu]

Context: https://t.co/7g4SVp0lQh

[Azamat Omuraliev @azamatomu]

@petr0105 Got a link to an example of such a reel?

[Azamat Omuraliev @azamatomu]

@paul_cal Another thing that I feel is thrown away is the distance in latent space between inputs. For semantic matching, dotproduct of embeddings consistently yields better results than asking the model to find related words/images/etc

[Azamat Omuraliev @azamatomu]

@lennysan @raiza_abubakar How did the collaboration with the Gemini team go? The product is so good it feels like there was work needed beyond fine tuning already trained models

[Azamat Omuraliev @azamatomu]

Source: https://t.co/ny7kvmFb6z

[Azamat Omuraliev @azamatomu]

Always good to find some emperical evidence supporting intuition. RLHF'd LLMs produce less diverse outputs due to 'mode collapse'; something that you can clearly see in how uniform AI-produced text looks despite any amount of prompting that you throw at it.

[Azamat Omuraliev @azamatomu]

hopefully moloch wins and we get them ray bans!

[Azamat Omuraliev @azamatomu]

moloch at play

[tobi lutke @tobi]

What overregulation feels like. AI progress is now skipping Europe