Eric Schreiber

Faster models, bigger questions

[Eric Schreiber @schreiberic]

NVIDIA’s CuTe layouts are gaining traction. I wanted to see why everyone loves them. The basics were easy, but intermediate resources beyond matrix algebra were scarce. So I wrote a blog post sharing my journey, building up to a GEMM kernel that can beat cuBLAS 🧵

[Eric Schreiber @schreiberic]

@tugot17 @UniofOxford So cool! congrats 🙌🏻

[Eric Schreiber @schreiberic]

@TomaszSternal Huge congrats 🙌🏻

[Eric Schreiber @schreiberic]

At ICLR. Let’s connect and chat: hardware, CUDA, architecture, pre/post-training, and whatever’s got you excited.

[Eric Schreiber @schreiberic]

@tugot17 🫡 Great as always.
Why do you think there are not a lot more models that perform continued pre-training to sparsify their attention?

[Eric Schreiber @schreiberic]

Nice one! I can tell you put a lot of effort into this post. I’ve started reading it and will need some time to go through it all :)

I did it the opposite way, started with CuTe (bit of self-promotion: have als a blogpost) and now looking into the mxfp8 and fp4 stuff.

[Eric Schreiber @schreiberic]

@yacinelearning @jonashubotter Reasoning trace length increasing is usually a good proxy for a healthy GRPO run. This method, to me, produces a strong instruct model that performs well without verbose reasoning (which is fantastic). However, for ood cases, preserving backtracking likely still matters.

[Eric Schreiber @schreiberic]

@yacinelearning @jonashubotter Don’t get me wrong, I love the paper. However, I see some weaknesses in the method. Generalization may be challenging because reasoning traces are heavily reduced, pushing the model to jump straight to the correct answer (https://t.co/TcXOG7bruq).

[Eric Schreiber @schreiberic]

@SzymonOzog_ you realise just how much you‘re standing on the shoulders of giants

[Eric Schreiber @schreiberic]

@willccbb Thought this recent work from @jonashuebotter was pretty cool:
For SFT: https://t.co/fHiGjDCsTl
For RL: https://t.co/RxNo5TjxGP

[Eric Schreiber @schreiberic]

@karpathy @maxbittker For me it helped to create a human-opus-interaction.txt file for outputs and interaction, telling the model an absurd and unrealistic goal (kernel duration, target loss …) and not to come back to me until it has achieved it. Prolonged the loop significantly

[Eric Schreiber @schreiberic]

@maharshii When I started trying it out last summer it was awful. Since this year, given some initial ideas, it‘s been pretty neat. Also running in a loop to improve an implementation works quite well too

[Eric Schreiber @schreiberic]

@tri_dao Once I got the hang of CuTe I'm loving it as well. The compile time is amazing!
But the entry barrier feels huge. Feels like you need to know CUDA and have a PhD in math before you can even begin

[Eric Schreiber @schreiberic]

Feedback very welcome.
Blog: https://t.co/lnIdhrpujO

Code & Profiles: https://t.co/Nsd9hOHOIP

[Eric Schreiber @schreiberic]

Last week we explored NVIDIA's CuTe layouts. Today, we put that theory into practice. Part 2 is out now! Most CuTe examples skip straight to highly optimized code without explaining the reasoning. Join me as we build a MM kernel with CuTe, that can beat cuBLAS in certain cases 🧵

[Eric Schreiber @schreiberic]

The final kernel reaches 116% of cuBLAS on A100s for 2048×2048 BF16 matrices. Though to be fair, the specific lead at 2048 is more likely due to them underperforming rather than to my ingenuity

[Eric Schreiber @schreiberic]

@elliotarledge check out https://t.co/aq1Lfzc6Fp