Loïck BOURDOIS

FAT5 (Flash Attention T5) boy
@huggingface Fellow 🤗

[Loïck BOURDOIS @BdsLoick]

New blog post on @huggingface!
An introdution to Trimming ✂️ a little-known but highly effective model reduction method. We achieved up to 87.24% size reduction while preserving performance 🧵

[Loïck BOURDOIS @BdsLoick]

@IBM @Microsoft @GoogleDeepMind @baai @qwen @orionweller @vllm @Google @Meta @huggingface @Nils_Reimers @jaseweston Big thanks to my HF Fellows bros for multilingual evaluation @tomaarsen, Bram Vanroy, @christopher, @w00jun_ @mrm8488, @prithivMLmods and to @AI_AlphaEdge for the time dedicated to this project 🙏

Links 👇
Blogpost: https://t.co/hBvVKpk7So
Models: https://t.co/FsycUcxyAK

[Loïck BOURDOIS @BdsLoick]

New blog post on @huggingface!
An introdution to Trimming ✂️ a little-known but highly effective model reduction method. We achieved up to 87.24% size reduction while preserving performance 🧵

[Loïck BOURDOIS @BdsLoick]

@IBM @Microsoft @GoogleDeepMind @baai @qwen @orionweller @vllm @Google @Meta @huggingface @Nils_Reimers @jaseweston From these 16 families, we generated more than 5,500 monolingual models in 124 different languages.

[Niels Rogge @NielsRogge]

Introducing a revival of PapersWithCode!

As @ilyasut said, we're back to the "age of research".

Hence, it's important to share research and build on each other's work.

> find SOTA per domain, not just LLMs
> leaderboards
> methods
> all parsed at scale using AI agents.

[tomaarsen @tomaarsen]

🌐 I've just released Sentence Transformers v5.4: we're going fully multimodal for embeddings & reranking!
Also featuring a modular CrossEncoder, and automatic Flash Attention 2 input flattening.

Highlights in 🧵

[みぃ🍵 @mithernet]

著者です！
Attentionの「相対比較しかできない」という制約を外した、新しい機構を提案しました

①まずわかりやすい利点

✅学習時より圧倒的に長い文でも性能維持＆正確な情報取得
✅収束が非常に高速（LR=1でも学習可能）
✅モデルサイズ4割削減
✅推論速度3倍超

(続く)

https://t.co/75rZpnqieu

[Loïck BOURDOIS @BdsLoick]

CuTeDSL is really nice
For those wishing to get into writing kernels in this language, https://t.co/7fjj26yj0F can be useful
Boris ALBAR reimplemented Flash Attention, RoPE, RMSnorm, etc.
Everything compatible with HF Transformers (tests on llama3, GLM4.7, Qwen3), TRL, PEFT/LoRA

[maharshi @maharshii]

CuTeDSL is my new favourite thing: I wrote a kernel for RMS norm after learning about layouts, tiling, copying tensors, reductions and so on, especially for inference and it is about 2.13x faster than a triton fused kernel for the given shape.

[Basile Terver @BasileTerv987]

𝗜𝗻𝘁𝗿𝗼𝗱𝘂𝗰𝗶𝗻𝗴 𝗘𝗕-𝗝𝗘𝗣𝗔 ⚡

An open-source library making JEPAs accessible, trainable on a single GPU in hours! 🚀

🔗 Paper: https://t.co/7YDSt0AiiA
💻 Code: https://t.co/KWkhcoDidU

[Loïck BOURDOIS @BdsLoick]

@MaziyarPanahi @OpenMed_AI @huggingface huggingface_hub api is all you need to do it programmably 👀

[Loïck BOURDOIS @BdsLoick]

@gui_penedo I suppose all good things must come to an end. Thank you very much for the high-quality multilingual datasets

[Loïck BOURDOIS @BdsLoick]

@lhoestq @huggingface @mervenoyann @abhi1thakur If you rename the `datasets` library to `nlp` as in early 2020, I'll make sure it passes 700k before the end of the year 👀