Dry

[Massimo @Rainmaker1973]

Using an hologram fan to visualize 360° ancient relics.
https://t.co/TauQmdS2pm

[Paul Xue @pxue]

Remember Devin?

Apparently demo's fake.
Paul is sad. 😥

[Andrej Karpathy @karpathy]

A few new CUDA hacker friends joined the effort and now llm.c is only 2X slower than PyTorch (fp32, forward pass) compared to 4 days ago, when it was at 4.2X slower 📈

The biggest improvements were:
- turn on TF32 (NVIDIA TensorFLoat-32) instead of FP32 for matmuls. This is a new mathmode in GPUs starting with Ampere+. This is a very nice, ~free optimization that sacrifices a little bit of precision for a large increase in performance, by running the matmuls on tensor cores, while chopping off the mantissa to only 10 bits (the least significant 19 bits of the float get lost). So the inputs, outputs and internal accumulates remain in fp32, but the multiplies are lower precision. Equivalent to PyTorch `torch.set_float32_matmul_precision('high')`
- call cuBLASLt API instead of cuBLAS for the sGEMM (fp32 matrix multiply), as this allows you to also fuse the bias into the matmul and deletes the need for a separate add_bias kernel, which caused a silly round trip to global memory for one addition.
- a more efficient attention kernel that uses 1) cooperative_groups reductions that look much cleaner and I only just learned about (they are not covered by the CUDA PMP book...), 2) the online softmax algorithm used in flash attention, 3) fused attention scaling factor multiply, 4) "built in" autoregressive mask bounds.

(big thanks to ademeure, ngc92, lancerts on GitHub for writing / helping with these kernels!)

Finally, ChatGPT created this amazing chart to illustrate our progress. 4 days ago we were 4.6X slower, today we are 2X slower. So we are going to beat PyTorch imminently 😂

Now (personally) going to focus on the backward pass, so we have the full training loop in CUDA.

[Yohei @yoheinakajima]

Seems like a busy game of hungry hungry hippo happened this weekend

[Tech Burrito @TechBurritoUno]

Human evolution process

[🌴Le Média des Antilles🌴 @antilles97x]

Le 8 Mai 1902, la Ville de Saint-Pierre, ancienne capitale de la Martinique, disparaissait suite à une éruption Volcanique 🌋🙏🏾

[Outremers360 @outremers360]

#Aérien : @CorsairFr s’est associé au chef guadeloupéen Jimmy Bibrac pour ses repas en business au départ des #Antilles

🍽️🛫Une façon de mettre en avant "la #gastronomie créole" et le "terroir antillais"

➡️https://t.co/g84RIy06Hq

[Watcher.Guru @WatcherGuru]

JUST IN: #Bitcoin whitepaper is hidden on every Apple MacBook computer running recent versions of macOS software.

[Elon Musk @elonmusk]

Old joke about agnostic technologists building artificial super intelligence to find out if there’s a God.

They finally finish & ask the question.

AI replies: “There is now, mfs!!”

[Prof. Feynman @ProfFeynman]

There are two rules in life:

1) Never give out all the information.

[Ouragans.com @ouragans]

L'exercice international de préparation aux tsunamis #CaribeWave2023 se déroule aujourd'hui dans toute la Caraïbe à partir de 10h, avec comme scénario, l'effondrement d'un pan de la Montagne Pelée causant une vague destructrice.

Mais savez-vous quoi faire en cas de tsunami ?

[Mediavenir @Mediavenir]

🇫🇷 FLASH - "Si les Français étaient vraiment en colère, je n'aurais pas été réélu il y a un an", affirme Emmanuel #Macron. (France 2) #Macron13h #reformedesretraites

[Tech Burrito @TechBurritoUno]

Are you?

[Elon Musk @elonmusk]

“I used to be in crypto, but now I got interested in AI"

[POOJA!!! @PoojaMedia]

There was no VAR check for that penalty call. Wow

[(fan) Trey @UTDTrey]

Never seen them give one man so many penalties in one competition, this shit is rigged

[Dry @DryLdn]

https://t.co/eUVfYQAZxF
https://t.co/8m0GaJwk2F
https://t.co/ORTbjdKT0Y
https://t.co/G41Lmq9lRr
https://t.co/M4j83i07dr
https://t.co/pvwFO5lfgQ
https://t.co/ZoEqtAXaRC
https://t.co/jMc1eOAGl2
https://t.co/cCIgMDaxs4
https://t.co/HLy0elTApM

[Dry @DryLdn]

Top nodes runners 24h
https://t.co/skViVtbyGq
https://t.co/kdZe46o2xh
https://t.co/lJBvpwETA7 🚀
https://t.co/4XcWmeeFP2
https://t.co/4uQW4ZF6OE
https://t.co/vqTYgSxqmI
https://t.co/0laTz5ZwAH
https://t.co/NwTyM7MbjC
https://t.co/hDdSPHYNpQ��

[Dry @DryLdn]

👇
https://t.co/jPzlSqEBxI
https://t.co/phq5lzccv8
https://t.co/pYhmQ72Sdm
https://t.co/OLNsSMjPJ8
https://t.co/U7KllMV6Do
https://t.co/bePR33GG8u
https://t.co/cmnbtmEqDk
https://t.co/Byl4Yb5wUa
https://t.co/I7bzmKhAPE
https://t.co/LLMQd2zaAv

[Dry @DryLdn]

Top nodes runners 48h
https://t.co/zWyv7UbA1S 🚀
https://t.co/2LcWGBCeFa
https://t.co/C9rflr3cPT
https://t.co/yLiizFNxLa
https://t.co/r8WwONHKzu
https://t.co/f7VtXgJ6SR
https://t.co/dxlxsWsX6q
https://t.co/zF0LC6eAj0
https://t.co/rHdfKR0mkS