Iaroslav Elistratov

ML engineer/ kernel research engineer | prev @ MIT startup

B200 FA4 in 60 diagrams (2026)
triton compiler passes (2025)
minimal DL library in C and CUDA (2024)

[Iaroslav Elistratov @iaro_e]

Build B200 attention kernel from scratch in CUDA and a little PTX, starting from a naive baseline and reaching 94.4% of FlashAttention-4.

I spent the last couple of months building visual, beginner-friendly progression around 60 diagrams.

We first develop an intuitive understanding of how the naive baseline works, then add one major optimization at a time, pairing each step with detailed diagrams, concise explanations, and code.

For the capstone project, we plug the final kernel into a video-generation model.

https://t.co/YFT7n2K6LX

[Iaroslav Elistratov @iaro_e]

@MillionInt, let's build some cars together?

[Iaroslav Elistratov @iaro_e]

Build B200 attention kernel from scratch in CUDA and a little PTX, starting from a naive baseline and reaching 94.4% of FlashAttention-4.

I spent the last couple of months building visual, beginner-friendly progression around 60 diagrams.

We first develop an intuitive understanding of how the naive baseline works, then add one major optimization at a time, pairing each step with detailed diagrams, concise explanations, and code.

For the capstone project, we plug the final kernel into a video-generation model.

https://t.co/YFT7n2K6LX

[Iaroslav Elistratov @iaro_e]

@Leik0w0 no need, thank you 🙂

[Iaroslav Elistratov @iaro_e]

Open to kernel research / ML systems roles (training and inference optimization, performance engineering). Fully remote. DM me, happy to chat

[Iaroslav Elistratov @iaro_e]

@xanamini Thank you so much!

[Iaroslav Elistratov @iaro_e]

@dejavucoder thank you for reposting! https://t.co/rgmpAxORDZ

[Iaroslav Elistratov @iaro_e]

Lovely

[SzymonOzog @SzymonOzog_]

You can just take your tea gear outside and enjoy it while reading "B200 Attention Kernel from Scratch to Near-SOTA in 60 Diagrams"

Very well written progressing on the attention kernel on Blackwell, skimmed a bit through the ending because of time and Ill def revisit

[Iaroslav Elistratov @iaro_e]

@SzymonOzog_ I’m really glad you liked it (and I love the tea setup 🙂). For anyone interested: https://t.co/UG9VR70b3Y

[Iaroslav Elistratov @iaro_e]

@SzymonOzog_ @SzymonOzog_ Thank you for sharing. I saw your videos on youtube some time ago. Great work!

[Iaroslav Elistratov @iaro_e]

@gaunernst @SzymonOzog_ Thank you both! Appreciate it!

[Iaroslav Elistratov @iaro_e]

the framing on the @SemiAnalysis pod that no one at the labs understands kernels anymore is nonsense.

one reasonably good kernel researcher is what, like 1M / year?

and how much does OAI spend on running their kernels? Like $30B/year?

does it make sense to assign somebody to work *deeply* on kernels? hmmm....

[Iaroslav Elistratov @iaro_e]

brothers and sisters, I moderated my language

[Iaroslav Elistratov @iaro_e]

@firesidealpha let's do a quick math:

one reasonably good kernel researcher is what, like 1-2M / year?

and how much does OAI spend on compute serving their kernels? Like $50B/year?

does it make sense to assign somebody to work *deeply* on kernels? hmm....

[Dinara @Dinara27984392]

I’ve been learning CUDA for a few months now, mostly through handwritten notes. Finishing another notebook felt like a little milestone, so I wanted to share a few highlights from the journey.

[Iaroslav Elistratov @iaro_e]

@Dinara27984392 👏👏👏

[Iaroslav Elistratov @iaro_e]

@MillionInt Apply this daily if your tensor cores are starving:
https://t.co/UG9VR70b3Y

[Iaroslav Elistratov @iaro_e]

here me out boys, it's not "200x faster" as you used to seeing on twitter nowadays, but i'm getting there

[Iaroslav Elistratov @iaro_e]

been working on veeery elegant speculative optimization to make FlashAttention4 up to 6% faster in some cases.
Will test more, if results hold will write a post