rdyro

JAX @ Google

[Jacob Austin @jacobaustin132]

Today we're putting out an update to the JAX TPU book, this time on GPUs. How do GPUs work, especially compared to TPUs? How are they networked? And how does this affect LLM training? 1/n

[rdyro @rdyro128523]

Code here: https://t.co/O1XPYeTIHa

[rdyro @rdyro128523]

Llama 4 inference in pure JAX! Expert/tensor parallelism with int8 quantization. Contributions welcome!

[rdyro @rdyro128523]

@rosinality For prefill we can do token dropping with some conservatism factor (a static shape value). For decode it can be faster to process all tokens and avoid conditionals from dynamic shapes, but still investigating.

[rdyro @rdyro128523]

Deepseek R1 inference in pure JAX! Currently on TPU, with GPU and distilled models in-progress. Features MLA-style attention, expert/tensor parallelism & int8 quantization. Contributions welcome!

[Roy Frostig @froystig]

A nice and concise R1 inference jax:tpu port by @rdyro128523. Good for both reading and running. Watch the repo for more.

[rdyro @rdyro128523]

Link to repo: https://t.co/7HWnICjSk2

[Cristian Garcia @cgarciae88]

The JAX team is hosting a dinner / networking event during NVIDIA's GTC in March. Join us for an evening of food, drinks, and discussion of all things JAX.

@SingularMattrix and other JAX team member will be attending. Please register early as capacity is limited.

[Jacob Austin @jacobaustin132]

Making LLMs run efficiently can feel scary, but scaling isn’t magic, it’s math! We wanted to demystify the “systems view” of LLMs and wrote a little textbook called “How To Scale Your Model” which we’re releasing today. 1/n

[Sharad Vikram @sharadvikram]

We now have a guide to writing distributed communication on TPU using Pallas, written by @JustinFu769512! https://t.co/9TcqNizGV4

Overlapping comms + compute is a crucial performance optimization for large scale ML. Write your own custom overlapped kernels in Python!