Will Bui

vLLM contributor.

doing things at

[Will Bui @will_ea]

27x faster Attention Residuals!!! 🚀

We implemented Block AttnRes as a pip-installable package.

!pip install flash-attn-res

No annoying kernel nonsense.
No compile/autograd plumbing.
Call it like a regular PyTorch op.

It just works.

Methodology:
🔹 fused triton kernels
🔹 batched attention over residual blocks
🔹 online-softmax merge
🔹 flash attention-style split-KV reduction

Thanks @LLMenjoyer and @cartesia for the support and guidance✌️

[Kimi.ai @Kimi_Moonshot]

Introducing 𝑨𝒕𝒕𝒆𝒏𝒕𝒊𝒐𝒏 𝑹𝒆𝒔𝒊𝒅𝒖𝒂𝒍𝒔: Rethinking depth-wise aggregation.

Residual connections have long relied on fixed, uniform accumulation. Inspired by the duality of time and depth, we introduce Attention Residuals, replacing standard depth-wise recurrence with learned, input-dependent attention over preceding layers.

🔹 Enables networks to selectively retrieve past representations, naturally mitigating dilution and hidden-state growth.
🔹 Introduces Block AttnRes, partitioning layers into compressed blocks to make cross-layer attention practical at scale.
🔹 Serves as an efficient drop-in replacement, demonstrating a 1.25x compute advantage with negligible (<2%) inference latency overhead.
🔹 Validated on the Kimi Linear architecture (48B total, 3B activated parameters), delivering consistent downstream performance gains.

🔗Full report:
https://t.co/u3EHICG05h

[llm_enjoyer @LLMenjoyer]

video of my training runs

[Yacine Mahdid @yacinelearning]

if you aren’t sleeping at least 9h+ per day you are going to get obliterated by better sleepers

like the quality of their thought will just leapfrog you out of existence

you’ll be out there yawning away in meetings with your brain clogged up with amyloid-beta

them? sharp as an arrow with not a tau in sight

[llm_enjoyer @LLMenjoyer]

bro the NEURAL CIRCUITS bro!! the SPARSE AUTOENCODERS...!

[Will Bui @will_ea]

@michellechen the computer history museum nearby is also very lit

[Will Bui @will_ea]

@Onanaroghene @TheVixhal Learning something deeply requires actively doing something with the information you learned

[Deep-ML @real_deep_ml]

We just launched a new project that teaches you how to build Flash Attention with CUDA, step by step.

By the end, you’ll have a working Flash Attention kernel built from the ground up.

The project covers:
-CUDA primitives warm-up
-Matrix operations
-Naive attention baseline
-Online softmax math
-Tiled attention building blocks
-Fused Flash Attention kernel
-Causal Flash Attention

It will be open to everyone for the first 2 weeks, then it will become part of our premium projects.

[llm_enjoyer @LLMenjoyer]

u js trained on test bro, it's not that deep

[Will Bui @will_ea]

@CoreAutoAI not deep enough learning

[Will Bui @will_ea]

@dogacel0 @ChinmayKak new epoch

[Will Bui @will_ea]

@Tim_Dettmers The better baseline would have been to use KVTC, which this blog built upon. https://t.co/NeWLQ3lBpn

[Will Bui @will_ea]

@krishgarg @GoogleDeepMind This needs a KVTC baseline https://t.co/NeWLQ3lBpn

[Will Bui @will_ea]

@gurishsharma @briar2682 so, no.

[Will Bui @will_ea]

@MainzOnX glad to hear! it's interesting to hear people talk about using agents and hardly writing code since my experience is that i still have to be heavily involved in editing the code (though i also have really high code quality standards as an oss contributor)

[Will Bui @will_ea]

@LLMenjoyer i better see attention residuals in sonic 4 arch

[Will Bui @will_ea]

@chongz I have always believed in SSMs

[Will Bui @will_ea]

@rayandabbagh @Meta It's not like she joined and they realized she's an underperformer either, because otherwise, they wouldn't have extended her a return offer otherwise!

[Will Bui @will_ea]

@josip_ @rayandabbagh @Meta okay how about this? same role same location same yoe

[Will Bui @will_ea]

@LLMenjoyer @tri_dao gpt helped me crank out my kernels (w/ a lot of handholding from me ofc) https://t.co/s006fUVmRK

[Will Bui @will_ea]

27x faster Attention Residuals!!! 🚀

We implemented Block AttnRes as a pip-installable package.

!pip install flash-attn-res

No annoying kernel nonsense.
No compile/autograd plumbing.
Call it like a regular PyTorch op.

It just works.

Methodology:
🔹 fused triton kernels
🔹 batched attention over residual blocks
🔹 online-softmax merge
🔹 flash attention-style split-KV reduction

Thanks @LLMenjoyer and @cartesia for the support and guidance✌️