Chen Lu

PhD student in math @MIT

[Chen Lu @_Chen_Lu_]

Composer 2 is pretty good!

[Cursor @cursor_ai]

We're releasing a technical report describing how Composer 2 was trained.

[Chen Lu @_Chen_Lu_]

@kevinyang 🔥 congrats!

[Magic @magicailabs]

LTM-2-Mini is our first model with a 100 million token context window. That’s 10 million lines of code, or 750 novels.

Full blog: https://t.co/oFz4A9ynVZ

Evals, efficiency, and more ↓

[Chen Lu @_Chen_Lu_]

@yapdianang Hey Dian Ang! Don't think there were too many surprises because this was a learning project. I did learn that convolutions are hard, and you really should worry about memory transfers

[Chen Lu @_Chen_Lu_]

I wrote a UNet diffusion model in pure CUDA: https://t.co/JQaLDywKtS

This project was inspired by @karpathy 's llm.c (https://t.co/aybQH3NAo8). I also learnt a lot about CUDA kernels from @Si_Boehm 's Matmul blog (https://t.co/PKphlZRHz6).

(1/3)

[Chen Lu @_Chen_Lu_]

@MaheshaGodekere Read Simon's blog, it's terrific

[Chen Lu @_Chen_Lu_]

@RisingSayak @karpathy @Si_Boehm images are 64x64, using fp32, not sure if that's what you're asking?

[Chen Lu @_Chen_Lu_]

@iman2_718 @karpathy @Si_Boehm Thanks for the suggestion!! It looks promising for reducing the memory reloads for the convolutions, let me check it out

[Chen Lu @_Chen_Lu_]

@naklecha @karpathy @Si_Boehm Thanks @naklecha ! Excited to see what's next from aaaaaaaaaa!

[Chen Lu @_Chen_Lu_]

@TiggerSharkML @karpathy @Si_Boehm Ah that should be doable already with the kernels in Andrej's llm.c, since DiT uses the same components

You'd probably want to rewrite some kernels to get best performance, otherwise you will be doing a bunch of slow tensor permutes. Definitely would be cool to see though

[Chen Lu @_Chen_Lu_]

@ChrisChoy208 @karpathy @Si_Boehm Ah thanks for spotting the cudaMalloc in the training loop Chris! I thought I had moved all of them outside the loop😅

Almost all of the memory is otherwise allocated before the training loop, so hopefully this is not affecting the times too much.

[Chen Lu @_Chen_Lu_]

@eliebakouch @karpathy @Si_Boehm Thanks elie!

[Chen Lu @_Chen_Lu_]

@karpathy Thanks Andrej! Big fan of your work 😊

[Chen Lu @_Chen_Lu_]

The main targets for optimization are the forward and backward passes of convolutions, which are currently written in a matmul-like fashion.
(3/3)

[Chen Lu @_Chen_Lu_]

Most of the effort so far was spent getting the whole model to work. There is still a lot of room for optimization. Current per iteration timings on a single RTX 4090:
- this repo: 143ms
- PyTorch: 66ms
- PyTorch with torch.compile: 59ms
(2/3)