Ronak Singhal

Current @ MatX, ex-Intel Senior Fellow. CMU alumnus. Pittsburgh in the blood.

[Ronak Singhal @rsinghal1]

One of the examples of why it’s great having our own ML team at @MatXComputing

[Alexander Tian @Alex2262T]

Hardware yearns for block sparse attention, yet it seems largely absent from open weight LLMs. DeepSeek developed NSA, and people speculated DeepSeek v4 would integrate it, yet it was never utilized.

We have a hypothesis as to why.

We found that replacing dense attention with NSA significantly degraded its ability on synthetic retrieval tasks, even when finetuned on them. On our 32k context benchmark, it scored 0.300 compared to dense attention’s 0.904.

We found the reason, and how to fix most of it. Introducing COBS🌽(Cumulant Order Block Sparse Attention), with @AdiGhai18 @sanjitneelam @ZVasania @tensorpro:

• Raises NSA’s 0.300 → 0.820, closing ~86% of the gap to dense
• 15.15x less KV-cache read traffic than dense (just 1.21x the NSA baseline)
• Lower position-wise NLL than dense in our comparison

The key insight: block selection is the keystone to block sparse attention, and existing methods are mathematically stuck in storing a first-order approximation of a cumulant generating function. COBS caches a compressed second cumulant and escapes this ceiling.

Details in the paper.

[Ronak Singhal @rsinghal1]

@lauriewired See section 2.1.1 https://t.co/BLeWvMGMWs

[Ronak Singhal @rsinghal1]

@suekhim Bored soon to be 8th and 6th graders who love challenges and puzzles.

[Dwarkesh Patel @dwarkesh_sp]

New blackboard lecture w @reinerpope

How do chips actually work – starting with basic logic gates, and working up to why GPUs, TPUs, FPGAs, and the human brain each look the way they do.

0:00:00 – Building a multiply-accumulate from logic gates
0:16:20 – Muxes and the cost of data movement
0:25:59 – How systolic arrays work
0:39:00 – Clock cycles and pipeline registers
0:51:40 – FPGAs vs ASICs
1:03:14 – Cache vs scratchpad
1:07:16 – Why CPU cores are much bigger than GPU cores
1:11:49 – Brains vs chips
1:15:22 – A GPU is just a bunch of tiny TPUs

Look up Dwarkesh Podcast on YouTube/Spotify/etc to watch. Enjoy!

[Ronak Singhal @rsinghal1]

Fun discussion

[Austin Lyons @austinsemis]

It's so much fun talking with builders like Reiner. Pure dopamine for me, and I walk away smarter than I started. Interconnects for MoE, BlueSpec, Little's law. Lots of nuggets in this one.

[Ronak Singhal @rsinghal1]

@TheValueist What does "perf-per-sandbox" mean?

[Ronak Singhal @rsinghal1]

@lauriewired @p_budziszewski There are significant differences between P4's trace cache and what Intel/AMD (and even ARM !) are doing for uop caches. The big downside though on P4 was if you missed the trace cache, the decode path was very narrow. That is not the case on modern implementations.

[Ronak Singhal @rsinghal1]

@tbpn @sarahookr Finally someone with the reasonable/right take on the whole idea of data centers in space.

[Ronak Singhal @rsinghal1]

@djtodoro Good place to start -- https://t.co/p7lk7gl7wl

[Ronak Singhal @rsinghal1]

@brk0v Would be good to include in your results what system you did your benchmarking on

[Ronak Singhal @rsinghal1]

@halvarflake Looks like some good CPU data is about to be published by Meta -- https://t.co/J9dt66GfUc

[Ronak Singhal @rsinghal1]

@IanCutress Ha! I was just at my parents' house for the holidays and they were imploring me to get rid of (or at least just take home) my notes/books that are taking up space in their house. But I can't....let....go...

[Ronak Singhal @rsinghal1]

@ToddFuhrman You can't fumble forward on a 2-point try. The issue here is that it was a backward pass which is treated distinctly in the rulebook from a fumble.

[Ronak Singhal @rsinghal1]

@lemire I am sure you know but it is important to always note that R8a a vCPU == a core (SMT disabled). For R8i, a vCPU == a thread (SMT enabled). This shouldn't matter if your code is testing a single thread; it obviously does if you are testing N threads.

[Ronak Singhal @rsinghal1]

@i_bogosavljevic See: https://t.co/Rn8JzBQdYD

[Intel Business @IntelBusiness]

Tune in live from #HPEDiscover! Tomorrow, Intel’s @rsinghal1 and HPE’s John Carter will discuss how the latest #HPE servers with #IntelXeon 6 CPUs are transforming what’s possible for today’s enterprises. Hear firsthand how enterprises can lower costs with greater security and improved efficiency across the broadest set of workloads. Set your reminders! #Intel #HPE #Innovation #AI

[Lisa Spelman @LisaSpelman]

Great example of how to use Xeon for an off the shelf chatbot by ⁦@earlruby⁩ AI without GPUs: Accessing Sapphire Rapids AMX instructions on vSphere | VMware https://t.co/AQZtmDsaGR

[Ronak Singhal @rsinghal1]

@FelixCLC_ @IntelDevTools Yes. Deprecated.

[Ronak Singhal @rsinghal1]

@Dejan_Kovacevic Yes!

[Ronak Singhal @rsinghal1]

@Dejan_Kovacevic @PointParkU Seattle meetup?