Michał Jamry

[Michał Jamry @jamrymichal]

The X Agent will post the previous task output as a tweet. Please make sure a previous task exists and is completed.

[Michał Jamry @jamrymichal]

@FakePsyho Gratulacje!

[Michał Jamry @jamrymichal]

@zhenjun_zhao @pu_wen99907 @PeidongLiu_ Love the title

[Zhenjun Zhao @zhenjun_zhao]

SIU3R: Simultaneous Scene Understanding and 3D Reconstruction Beyond Feature Alignment

Qi Xu, Dongxu Wei, Lingzhe Zhao, @pu_wen99907, Zhangchi Huang, Shunping Ji, @PeidongLiu_

tl;dr: reconstruction and understanding->pixel-aligned 2D-to-3D lifting->unified learnable queries

https://t.co/Cps0cV24Ks

[James May @MrJamesMay]

At last.

[Rick Lamers @ricklamers]

CS224N dropped on YT (2024 spring) full 18 lecture series by none other than @chrmanning

[Andrej Karpathy @karpathy]

The (true) story of development and inspiration behind the "attention" operator, the one in "Attention is All you Need" that introduced the Transformer. From personal email correspondence with the author @DBahdanau ~2 years ago, published here and now (with permission) following some fake news about how it was developed that circulated here over the last few days.

Attention is a brilliant (data-dependent) weighted average operation. It is a form of global pooling, a reduction, communication. It is a way to aggregate relevant information from multiple nodes (tokens, image patches, or etc.). It is expressive, powerful, has plenty of parallelism, and is efficiently optimizable. Even the Multilayer Perceptron (MLP) can actually be almost re-written as Attention over data-indepedent weights (1st layer weights are the queries, 2nd layer weights are the values, the keys are just input, and softmax becomes elementwise, deleting the normalization). TLDR Attention is awesome and a *major* unlock in neural network architecture design.

It's always been a little surprising to me that the paper "Attention is All You Need" gets ~100X more err ... attention... than the paper that actually introduced Attention ~3 years earlier, by Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio: "Neural Machine Translation by Jointly Learning to Align and Translate". As the name suggests, the core contribution of the Attention is All You Need paper that introduced the Transformer neural net is deleting everything *except* Attention, and basically just stacking it in a ResNet with MLPs (which can also be seen as ~attention per the above). But I do think the Transformer paper stands on its own because it adds many additional amazing ideas bundled up all together at once - positional encodings, scaled attention, multi-headed attention, the isotropic simple design, etc. And the Transformer has imo stuck around basically in its 2017 form to this day ~7 years later, with relatively few and minor modifications, maybe with the exception better positional encoding schemes (RoPE and friends).

Anyway, pasting the full email below, which also hints at why this operation is called "attention" in the first place - it comes from attending to words of a source sentence while emitting the words of the translation in a sequential manner, and was introduced as a term late in the process by Yoshua Bengio in place of RNNSearch (thank god? :D). It's also interesting that the design was inspired by a human cognitive process/strategy, of attending back and forth over some data sequentially. Lastly the story is quite interesting from the perspective of nature of progress, with similar ideas and formulations "in the air", with a particular mentions to the work of Alex Graves (NMT) and Jason Weston (Memory Networks) around that time.

Thank you for the story @DBahdanau !

[leloy! @leloykun]

Deep Learning Optimizers from First Principles

Now with more maths!

In this thread, I'll discuss:

1. The difference between 1st order gradient dualizaton approaches and 2nd order optimization approaches.
2. Preconditioning--how to do it and why.
3. How to derive a couple of deep learning optimizers from scratch using both approaches.

(1/n)

[Michał Jamry @jamrymichal]

@Forumogadka Dużo fachowej wiedzy, polecam

[Jakob Foerster @j_foerst]

When I discussed quitting Google to do a Phd, my manager, Steve Cheng, gave me the advice of "6 shots": Doing something meaningful usually takes about 5 years and we are productive for roughly 30 years. That gives you 6 attempts. So pick each one carefully and give it your best.

[Yann LeCun @ylecun]

Yet another opportunity to point out that reasoning abilities and common sense should not be confused with an ability to store and approximately retrieve many facts.

[Hieu Pham @hyhieu226]

https://t.co/FEcvI56izD

If you learn CUDA, you probably have read the legendary matrix transpose tutorial by Mark Harris.

📚Today, I present to you a CUDA tutorial by friends at @colfaxintl and myself. We explore many CUDA memory concepts like in Mark's tutorial, and more.

Crucially, we show how to implement these concepts in @nvidia's CuTe library. Yes, it's the one that powers Flash Attention⚡️.

I am grateful for friends at @colfaxintl, especially Jay Shah, for inviting me to help them this project.

This collaboration started in December 2023, when I got interested in @nvidia's CuTe library by reading the FlashAttention repo.

Despite my enthusiasm for the the library, learning it was a headache. I literally made no progress after weeks reading CuTe tutorial and code.

Then one day, I stumbled upon Jay's Flash Attention tutorial (https://t.co/oSL36ySSmB). His writing revealed to me the mathematical elegance in CuTe's design.

Teaser: can you imagine abstract algebra concepts -- like integral modules -- have anything to do with efficient deep learning kernels?!

But I digressed. Since reading Jay's works, I have held so much respect for him that being able to collaborate with his team feels profoundly rewarding.

Anyhow, I hope this CUDA tutorial makes you like CUDA programming and the CuTe library a little bit more.

Please stay tuned. We will be back with more tutorials.

[Marek Kraft @marekkraft]

Jeśli te nowe polskie modele językowe nie będą dawały takich odpowiedzi, to będę srodze rozczarowany

[Michał Jamry @jamrymichal]

@niedakh Z ciekawosci, jednostki naukowe tylko do NCN o granty pisza? Czy do ncbr/parp tez?

[Michał Jamry @jamrymichal]

@pablos_83 Powodzenia!

[Jascha Sohl-Dickstein @jaschasd]

Have you ever done a dense grid search over neural network hyperparameters? Like a *really dense* grid search? It looks like this (!!). Blueish colors correspond to hyperparameters for which training converges, redish colors to hyperparameters for which training diverges.

[Michał Jamry @jamrymichal]

@wina_Mazurka Finansowanie polskiej nauki i startupow z naciskiem na AI

[Michał Jamry @jamrymichal]

@andrzejdragan Moze na https://t.co/vQpYVHN56O - chlopaki organizuja regularne spotkania i streamuja na yt?

[Yann LeCun @ylecun]

Exactly.

[Aravind Srinivas @AravSrinivas]

Tomas Mikolov, the OG and inventor of word2vec, gives this thoughts on the test of time award, and the current state
of NLP, and chatGPT. 🍿