Ologan

all-in NVFP4 in CuTe DSL

[Ologan @OloganAI]

Don't waste your money on nerfed @nvidia SM120:

- tiny smem, no tcgen05/tmem
- 16-bit/8-bit GEMM capped 50% (ht/ @gaunernst)
- max cluster (1, 1, 1)

I gave up: local dev on $350 5060 8GB + jobs on @modal ($)/@vast ($$).

Real blocker is no ncu on cloud SM100, anyway.

[Lazarz @Laz4rz]

Owning a 5080 and buying a 5090 is probably the single most shared experience

[Ologan @OloganAI]

@Leik0w0 @nvidia @gaunernst @modal @vast

[Ologan @OloganAI]

Don't waste your money on nerfed @nvidia SM120:

- tiny smem, no tcgen05/tmem
- 16-bit/8-bit GEMM capped 50% (ht/ @gaunernst)
- max cluster (1, 1, 1)

I gave up: local dev on $350 5060 8GB + jobs on @modal ($)/@vast ($$).

Real blocker is no ncu on cloud SM100, anyway.

[Lazarz @Laz4rz]

Owning a 5080 and buying a 5090 is probably the single most shared experience

[Tom Yeh @ProfTomYeh]

Self-Attention vs Cross-Attention interactive diagram. Open https://t.co/RgpT6ger01

[Ologan @OloganAI]

DSpark

How LLMs Get Faster Without Changing Their Outputs | Speculative Decoding
By @jbhuang0604

https://t.co/mMijgCLAfR

[Ologan @OloganAI]

AI addict

[Dabs🩸 @DabsMalone]

What do you call someone who orchestrates agents to code?

A programmer?

Agentic engineer?

Autistic?

[brandon @burcs]

i wanted to build something this weekend that needed GPUs but wasn't sure where the cheapest place to host them was...

so of course instead of building the actual idea, i built https://t.co/pN1xRxUCxV

[Tanishq Mathew Abraham, Ph.D. @iScienceLuvr]

Sliding-window beats linear attention

"In this work, we show that Sliding Window Attention (SWA) with sinks performs as well or better than post-trained Linear Attention models. We observe this across multiple LLMs on various downstream tasks. For long-context reasoning tasks (Needle-in-a-Haystack and BABILong), SWA achieves massively higher performance (2 to 10 times higher than linear attention). SWA requires no post-training, is extremely fast, and requires low memory; therefore, making it an extremely cheap and reliable solution.

To reduce inference memory cost, we strongly recommend switching to SWA instead of post-training linear models. Linear attention models may have shown some promise, but they likely require to be trained from scratch or extensive post-training in order to even match SWA."

paper link: https://t.co/rlYgchGVlQ

[Kimbo @kimbochen]

@nrehiew_ lol Gluon is basically Triton GPU IR + CUTLASS-like programming semantics

[Ologan @OloganAI]

@Leik0w0 I only see other people say it's coming any minute. One claims it's for Enterprise users.

I would love to use it in Modal, but I understand it must be tricky due to security.

[Ologan @OloganAI]

And here I can't find a free B200 to profile my backward kernels before the deadline. smh

[Bryce, the CUDA Colonel @blelbach]

"I'm going to write the world's fastest CUDA kernels!"

Typical real world CUDA profile:

[Ologan @OloganAI]

@Leik0w0 Sure, Modal is great, but AFAIK you can't profile code with Modal. No ncu.

[Bryce, the CUDA Colonel @blelbach]

"I'm going to write the world's fastest CUDA kernels!"

Typical real world CUDA profile:

[Ologan @OloganAI]

Wake up in the middle of the night and can't resist to check how the coding agent is carrying the training run only to find one of the models is barely training. An obvious misconfig. You point it out and go back to sleep.

Worst part is this now validates your agentic addiction.

[Ologan @OloganAI]

@junrushao @Simon_Vt The one who “carries the weight”, maybe?

[Look babe, humans are so adorable when they post instead of just asking us]

[Ologan @OloganAI]

Hear me out:

nGPT speedrun-like competition but for reducing the amount of device memory required *for training* ~80% speed (moving data to host memory/NVMe).

Ideas from FreeToken, LMCache, Engram/Lngram, etc.

[Ologan @OloganAI]

A layer cake of abstractions.

A typical mistake is trying to do both high and low level in the same language.

Ideally leaning on existing DSLs.

[Ologan @OloganAI]

We need to make a language that is more convenient for LLMs to generate *and for us to verify it*. That's a radically different approach, but doable.

I'm planning to give it a shot as next project but others should try, too. Be humble but also bold.

[ADSP: The Podcast @adspthepodcast]

📢 ADSP Episode 301 is out! 📢 In this episode, @blelbach and @code_report chat about Conor's existential crisis, AI fluency, technical communities, auto autoresearch and more! https://t.co/ItyKAm37vS

[Ologan @OloganAI]

Agent took “you can just do things” to heart

[HikiPTG @HikiPTG]

okay i will admit, this impresses me more than the mathematical proofs

[Ologan @OloganAI]

@Leik0w0 @henrylhtsang I only have beginner knowledge of CuTe DSL Task Scheduling. It’s very recent!

[Ologan @OloganAI]

I'd rather lean on CuTe DSL's Task Scheduling and helpers for things like TMA async reductions. Much easier to reason and debug.

[henry tsang @henrylhtsang]

I am getting more CUDA pilled than JIT DSL pilled now

[Ologan @OloganAI]

@Leik0w0 Yes, if you are already using cluster in the code I bet it is likely worth it.

I'm thinking sync across 8/16 SMs is smaller than whole device.

[Ologan @OloganAI]

amax reduction: warp shuffle for max, warp thread 0 writes to warp_fmax[warp_id] (shared), CTA reduce and write to cta_fmax[cta_id] (global), then a small reduction kernel cta_fmax -> tensor_fmax.

Codex Sol max.

The news of our death has been greatly exaggerated, kernel bros.