Hicham

[Kylian Mbappé @KMbappe]

Madame Celeste Amarilla,
Vous êtes une femme méprisable et indigne de sa fonction.
Vous ne représentez pas le Paraguay, ce pays qui a transpiré la passion et l’honneur tout au long de la compétition. Par votre inconscience et votre racisme décomplexé, le monde entier a déjà oublié le parcours et l’effort historique que vos joueurs ont réalisés durant cette coupe du monde pour laisser place à une dame incompétente donnant la pire image possible de son pays.
Je ne laisserai jamais aux gens comme elle, la liberté de laisser propager leur haine et leur racisme à travers le monde.

[François Chollet @fchollet]

For anyone getting started with deep learning in 2025 and looking to do large scale training: use JAX + Keras 3. Unless you like suffering.

[🍉 Abubakar Abid @abidlabs]

Get started in just a few lines of Python: https://t.co/33NP1AAc8J

[Edward Z. Yang @ezyang]

People who run lots of small training jobs for your day job, what is one thing about experiment management / hygiene that you wish you knew when you started out?

[Hippolyte Gisserot-Boukhlef @gisship]

🚨 New paper drop: Should We Still Pretrain Encoders with Masked Language Modeling?
We revisit a foundational question in NLP: Is masked language modeling (MLM) still the best way to pretrain encoder models for text representations?
📄 https://t.co/W1p5mjTTf2
(1/8)

[Manuel Faysse @ManuelFaysse]

🚨Should We Still Pretrain Encoders with Masked Language Modeling? We have recently seen massively trained causal decoders take the lead in embedding benchmarks, surpassing encoders w/ bidirectional attention. We revisit whether Bert-style encoders are a thing of the past? (1/N)

[Jacob Austin @jacobaustin132]

Making LLMs run efficiently can feel scary, but scaling isn’t magic, it’s math! We wanted to demystify the “systems view” of LLMs and wrote a little textbook called “How To Scale Your Model” which we’re releasing today. 1/n

[Nate Finch -- natefinch.bsky.social @NateTheFinch]

I'm moving to blue sky folks. You should, too. A ton of people have moved recently. Use Sky Follower Bridge to refollow people on bsky that you follow on here. Put your bsky account name in your details here and the tool will let others refollow you over there.

[Nous Research @NousResearch]

@chargoddard @dottxtai @haizelabs @apyh__ @osanseviero @huggingface Then @bloc97_ presented on the difficulties of distributed training and gave new insights on the work on DISTRO, @NousResearch's work on massive bandwidth reduction for training over the internet:

https://t.co/62SsK7JZwI

[Soumith Chintala @soumithchintala]

There's three parts.

1. Fitting as large of a network and as large of a batch-size as possible onto the 10k/100k/1m H100s -- parallelizing and using memory-saving tricks.
2. Communicating state between these GPUs as quickly as possible
3. Recovering from failures (hardware, software, etc.) as quickly as possible

1. Fitting as large of a network and as large of a batch-size as possible onto the 10k H100s.

Parallelizing:
1. parallelize over batches
2. parallelize over layers (i.e. split a layer across GPUs)
3. parallelize across layers (i.e. 1 to N are on GPU1, N+1th layer to N+10th layer are on GPU2)
Keep parallelizing until you are able to use all GPUs well, with maximum utilization.

Checkpointing / Compute vs memorize:
* You need to save certain terms from forward to compute the backprop (save_for_backward). However, if the network is sufficiently large, it is more profitable to free these terms in order to fit a larger batch-size, and recompute them again when you need them to compute the backprop.
* Tricks like FSDP discard parts of weights that are held in one GPU (to save memory), and ask for the shards of weights from other GPUs right before they need them.

2. Communicating state between these GPUs as quickly as possible

Communication overlap:
When you need to communicate among GPUs, try to start communication as soon as you can:
* Exampel: when Nth layer is done with backward, while N-1th layer is computing backward, all GPUs with an Nth layer can all-reduce their gradients)

Discover and leverage the underlying networking topology:

Communicating large amounts of state (gradients, optimizer state) across multiple nodes is complicated. with Sync SGD, you have to communicate this state in a burst, as quickly as you can.

we might have multiple layers of switches, and have RDMA (ability to copy GPU memory directly to NIC, bypassing CPU ram entirely), and have frontend and backend NICs (frontend connects to storage like NFS, backend connects GPUs to other GPUs in cluster).

So, it's important to leverage all this info when running communication collectives like all-reduce or scatter/gather. All-reduce for example can be done algorithmically in log(n) if you tree-reduce; and the constant factors that change based on the type of fiber connecting one node to another in the tree of networking fiber is important to reduce overall time and latency.

Libraries like NCCL do sophisticated discovery of the underlying networking topology and leverage them when we run all-reduce and other collectives.

3. Recovering from failures (hardware, software, etc.) as quickly as possible

At 10k GPU scale, things fail all the time -- GPUs, NICs, cables, etc. Some of these failures are easy to detect quickly, some of them you can only detect because one node isn't replying back in time (say a NCCL all-reduce is stuck). We build various tools to monitor and detect fleet health, and remove failed nodes from the fleet as quickly as possible. This is quite hard.

Separately, at this large of a scale you can have silent data corruptions from memory bits flipping randomly (due to basic physics and amplifying the probability at this scale), and you suddenly have loss-explosions for no reason other than this random phenomenon. These happen at small-scale too, but very very infrequently so you barely notice. This is very hard to detect before-hand in software. Some hardware has hardware circuitry that does built-in checksums after it computes things -- this way if bit-flips occur the hardware can throw an interrupt. H100s and previous NVIDIA GPUs don't have this feature.

To counter all these failures, you would want to save your model state as frequently and as quickly as you can; and when a failure occurs, you want to recover and continue as quickly as you can. Usually, we save model state really quickly to CPU memory in a separate thread and in the background we save from CPU memory to disk or remote storage.
We also save model state in shards (this is torch.distributed's checkpointing feature), i.e. not every GPU needs to save all of the model weights; each GPU only needs to save a portion of weights -- and they can recover the other part of weights from other GPU shard checkpoints.

[Nous Research @NousResearch]

What if you could use all the computing power in the world to train a shared, open source AI model?

Preliminary report: https://t.co/4Rhd0ddp9z

Nous Research is proud to release a preliminary report on DisTrO (Distributed Training Over-the-Internet) a family of architecture-agnostic and network-agnostic distributed optimizers that reduces the inter-GPU communication requirements by 1000x to 10,000x without relying on amortized analysis, and matches AdamW+All-Reduce in convergence rates. This enables low-latency training of large neural networks on slow internet bandwidths with heterogeneous networking hardware.

DisTrO can increase the resilience and robustness of training LLMs by minimizing dependency on a single entity for computation. DisTrO is one step towards a more secure and equitable environment for all participants involved in building LLMs.

Without relying on a single company to manage and control the training process, researchers and institutions can have more freedom to collaborate and experiment with new techniques, algorithms, and models. This increased competition fosters innovation, drives progress, and ultimately benefits society as a whole.

This research is thanks to the hard work of @bloc97_ @theemozilla @apyh__ @UmerHAdil.

We invite researchers interested in exploring this area to join us in our quest.

[Charlie Marsh @charliermarsh]

Today, we're shipping a series of features that move uv beyond a pip alternative, and into an end-to-end solution for managing Python projects, command-line tools, single-file scripts, and even Python itself.

A single, unified tool. Like Cargo, for Python.

It's very fast.

[Daniel Galvez @memorypaladin]

A little fact people miss about nvidia-smi's "GPU-Util" percentage is the exact meaning of utilization it is measuring.

It simply refers to the portion of time that the GPU had a kernel active.

All 100% means is that at least one thread in one kernel is executing.

[Adam Paszke @apaszke]

Many of you are excited about H100 attention, so it’s a good time to show you Mosaic GPU: a Python DSL for H100s.

The attention example matches FA3 performance, while being only ~200 lines of Python: https://t.co/12ecz3LftV

It's easy to install too! Latest JAX packages have it.

[Dmytro Mishkin 🇺🇦 @ducha_aiki]

PaliGemma: A versatile 3B VLM for transfer

@giffmana et 34 al.

tl;dr: citation API for PaliGemma and interesting ablations :)
https://t.co/pOrg2dWM7B

[Jon Barron @jon_barron]

The legendary Ross Girshick just posted his CVPR workshop slides about the 1.5 decades he spent ~solving object detection as it relates to the ongoing LLM singularity. Excellent read, highly recommended. https://t.co/i3qS4iheU5

[Benjamin F Spector @bfspector]

(1/7) Happy mother’s day! We think what the mothers of America really want is a Flash Attention implementation that’s just 100 lines of code and 30% faster, and we’re happy to provide.

We're excited to introduce ThunderKittens (TK), a simple DSL embedded within CUDA that makes it easy to express key technical ideas for building AI kernels. TK lets us write clean, easy-to-understand code that maximizes GPU utilization -- on all kinds of kernels!
Code: https://t.co/zlWwbMymiw
Writeups: (short) https://t.co/7XMSQDmWEx, (long) https://t.co/T3WSc4cKop.

Joint with @AaryanSinghal4, @simran_s_arora, @HazyResearch and team!

[Clémentine Fourrier 🍊 is off till Dec 2026 (🪂) @clefourrier]

And it's out! :D

A good read if you want to think about doing robust evaluation, going in depths into the nits of it.

https://t.co/y7k6Q9X4A6

[clem 🤗 @ClementDelangue]

GPU-Poor no more: super excited to officially release ZeroGPU in beta today. Congrats @victormustar & team for the release!

In the past few months, the open-source AI community has been thriving. Not only Meta but also Apple, NVIDIA, Bytedance, Snowflake, Databricks, Microsoft, Google, and more have released open models and datasets on Hugging Face, which now hosts over 1M models on the Hub which have been downloaded over a billion times. More than that, many are starting to be better than proprietary APIs.

This movement has been supported not only by big tech but also by a thriving open-source AI community that includes academic labs, startups, and independent hobbyists. For example, more than 35,000 variation models of Llama have been shared on Hugging Face since Meta’s first version a year ago—including more than 7,000 based on Llama-3—ranging from quantized and merged models to specialized models in biology and Mandarin, to name a few. More than 4 million AI builders are now using Hugging Face.

However, the open-source community doesn’t have the same resources available to train and demo these models that big tech have at their disposal, which is why ChatGPT remains the most used AI application today.

@huggingface is fighting this by launching ZeroGPU, a shared infrastructure for indie and academic AI builders to run AI demos on Spaces, giving them the freedom to pursue their work without the financial burden of compute costs. Spaces have been the most popular way to build AI demos, with over 300,000 AI demos created so far on CPU or paid GPU (and a thousand more every day). To foster the continued development of the AI ecosystem, Hugging Face is committing $10M of free GPUs with the launch today of ZeroGPU.

Technically speaking, ZeroGPU leverages Hugging Face's experience in hosting and serving more than 100 Petabytes monthly from the Hugging Face Hub. ZeroGPU allows Spaces to run on multiple GPUs by making Spaces efficiently hold and release GPUs as needed (as opposed to a classical GPU Space that holds exactly one GPU at any time). This architecture is also more energy-efficient since GPUs are shared rather than duplicated. ZeroGPU uses @nvidia A100 GPU devices under the hood.

You can learn more about ZeroGPU here: https://t.co/1mxUxXmElv

More than 1,300 ZeroGPU spaces have been built since we started giving early access to AI builders on May 1, 2024: https://t.co/XvJ2MkcK7R

You can explore some examples from @victormustar: https://t.co/b8SUcRelJf

You can find the article from @kyliebytes: https://t.co/87uN1vnMu8

🤗🤗🤗

[Chenyan Xiong @XiongChenyan]

My last work at Microsoft Research is finally released: https://t.co/OWhvzqFJEH 10 MILLION REAL Bing search queries with 60 MILLON+ REAL user clicks on 10 BILLION ClueWeb22 documents. Have fun scaling up!