Sergei Lebedev

Adding ∇ for a living.

[Adam Paszke @apaszke]

Want to improve GPU compute/comms overlap? We just published a new short tutorial for you!

A few small changes to the Pallas:MGPU matmul kernel is all it takes to turn it into an all-gather collective matmul that overlaps NVLINK comms with local compute: https://t.co/HY4C3MwMb7

[Sergei Lebedev @superbobry]

@WCelhen @apaszke Curious what you are looking to do with SparseCore. Can you share more?

[Adam Paszke @apaszke]

Curious how to write SOTA performance Blackwell matmul kernels using MGPU? We just published a short step-by-step tutorial: https://t.co/XRVX34juEz

At each step, we show exactly what (small) changes are necessary to refine the kernel and the final kernel is just under 150 lines.

[Sergei Lebedev @superbobry]

@AliesTaha Thanks for sharing, Ali! Can you link to the kernel and benchmarking code, please?

[Sergei Lebedev @superbobry]

@gallabytes @sharadvikram @finbarrtimbers @IlyasHairline @_xjdr OOC what kind of documentation are you missing most? Is it GPU-specific guides, API docs, "the sharp edges", something else?

[Dan F-M @exoplaneteer]

I've finally landed my first proper JAX feature since joining the team: a supported "foreign function interface", which makes it easier to call into external libraries from within JAX code. Check it out: https://t.co/8ilDuW1IYI

[Sergei Lebedev @superbobry]

@youlovegriszka получается, это уже традиция

[Adam Paszke @apaszke]

Many of you are excited about H100 attention, so it’s a good time to show you Mosaic GPU: a Python DSL for H100s.

The attention example matches FA3 performance, while being only ~200 lines of Python: https://t.co/12ecz3LftV

It's easy to install too! Latest JAX packages have it.

[Ivan Lobov @ilobov]

Our team is looking for a strong research engineer, hardware background is *not* obligatory. Please, share and recommend someone!
https://t.co/AGrF7rwRzZ

[Sergei Lebedev @superbobry]

@DrLangstrand Whenever there is a LaTeX alignment/spacing problem, the answer is usually minipage and its friends hspace and vspace :)

[Sergei Lebedev @superbobry]

@vincenthz @ChShersh Curious to hear what crufts you came across?

[Sergei Lebedev @superbobry]

@mitsuhiko @robsmallshire That's sadly "by design". PEP-484 decided that int should be a subtype of float, even though it isn't at runtime...

[Sergei Lebedev @superbobry]

@mitsuhiko I think a lot of the ergonomics were sacrificed to the idea of annotations being used for things other than types. So, most typing features are designed to abuse existing syntax to the benefit of nobody.

[Kevin Patrick Murphy @sirbayes]

The trilogy is complete! My "Advanced Topics" book is officially released today. Buy it on Amazon, or get it for free at https://t.co/yWK8zMbMRA.

[Tom Carpenter, PhD (@tcarpenter.bsky.social) @tcarpenter216]

Friend reminder that it’s all regression

https://t.co/OLqR5Qadqw

[ëugene kharitonov 🏴‍☠️ @n0mad_0]

Are you a PhD student interested in the interface between generative AI, LLMs and audio? Our team at Google behind AudioLM, MusicLM and AudioPaLM is looking for a talented student researcher!

See details and apply https://t.co/qFY95633P6 and send your cv to [email protected].

[Sergei Lebedev @superbobry]

@yminsky @ezyang You can use it in CI as well, though, right? You don't have to use mypy there.

[Sergei Lebedev @superbobry]

@yminsky The Python type system has no spec, so I can guarantee there will be plenty of things they disagree about.

Pyright is also usually faster to adopt new type system features, so it's possible to get type errors simply because mypy doesn't fully support some feature (yet).

[Sergei Lebedev @superbobry]

@yminsky Is there a way to learn more about what Jane Street is doing in this space without applying? :)

[KC Sivaramakrishnan @kc_srk]

OCaml wins the 2023 SIGPLAN programming languages software award! 🎉https://t.co/GpxK3bRutM