An.twi

Afria | Alverium | ChekBuk l GA Engineer | Blockchain & Laravel Artisan | 21st century alchemist

[An.twi @SupremeAntwi]

Make the time you spend on earth expand the horizons of human capabilities.

[vx-underground @vxunderground]

> Peter Stokes
> Scattered Spider guy
> Arrested
> Microsoft helps FBI
> Read court documents
> Page 12
> Microsoft tracks Stokes from GDID
> Microsoft Global Device Identifier (GDID)
> Stokes used Windows
> Page 34
> GDID assigned to each OS install
> GDID unique to each device
> GDID only change if OS wiped
> Stokes GDID 6755467234350028
> GDID reported internet activity to Microsoft
> GDID showed Stokes using Ngrok
> GDID reported Stokes IP address
> GDID showed Stokes web activity
> GDID showed timestamps of web activity
> GDID mapped with video game activity
> GDID showed games played
> GDID undocumented
> GDID only mentioned in one MSDN document
> Azure UCDOStatus
> Azure Monitor Logging

[Big Boss @0xBADB01E]

First you have to understand that modern LLM inference already disaggregates weights as models outgrew single chips years ago. You shard either by layer (pipeline parallelism) or by slicing every layer (tensor parallelism), and the two do very different things.

As an example, let’s look at Llama 3.3. It has 70B of weights and at FP8 that’s 70 GB of memory which is enough to fit on a single H100. Now that H100 has 3.35 TB/s of HBM, so the fastest it can ever decode for one user is 70/3.35 ≈ 21 ms/token or ~48 tok/s while using under 1% of its FLOPs. Now if we pipeline it across 8 chips: each chip holds ~8.75 GB, which means it only needs 1/8th the bandwidth and 1/8th the FLOPs to sustain the same aggregate throughput. Now crucially the token/sec a user gets is limited by the amount of data that crosses the link. In current LLMs all that is a small amount of activations for LLama 3.3 it’s ~8 KB per token….

Yes, you read that right it’s 8 KILOBYTES we are sending over a <900 GB/s link. That’s only 9 ns of serialization time but the overhead of 224G PAM4 SerDes adds ~100 ns per link traversal with RS-FEC which is 11x longer than the payload itself. And then you have the NVSwitch adding ~300 ns per hop and you need to pay twice. That’s ~600 ns of just hardware latency wrapped around 9 ns of data making a 98% tax before software even shows up. Then NCCL’s collective stack turns 600 ns into 10-20+ us… all to move 8 kilobytes lol. For comparison 8 KB serializes over 10 Gigabit Ethernet NRZ, in just 6.6 us. Pipeline parallelism however doesn’t make a single user faster as the token still needs to visit every layer in the sequence, so per-user speed is still weights / per-chip bandwidth.

To get more speed per user token you need to use tensor parallelism and have all the chips work on the same layer simultaneously. TP costs you 2 all reduce OPs per layer, 160 per token on llama 3, that’s still kilobytes of traffic but with NVLink overhead it’s a massive tax and why pipeline parallelism on most models still gives more interactivity per user. However, this gives you a huge latency lever to pull that scales tokens per second with interconnect speed instead of memory BW.

The clever amongst you might have also realized that sharding doesn’t just cut memory bandwidth per chip it also cuts FLOPs per chip and is why we have such bad MFU on decode. So once you’ve sized the link for the memory, you need to size the compute for it too. This is called “balancing the pipeline”, and currently no shipping chip does it because they were all designed as standalone monsters. Remember Tokens/sec = ~aggregate memory BW / bytes touched per token. At batch 64 in FP4 you need ~250 FLOPs per byte, and Blackwell ships 1,250. Provisioned 5x more than the narrow pipe of HBM. Nobody saturates shit cause they are all building around HBM.

So now it all comes full circle. Parallelism reduces memory bw pressure and thus FLOPs but increases interconnect latency pressure. Despite having HBM and GigaSERDES we aren’t actually doing more work lol. But if you really wanted to balance the pipeline you need to match the memory bandwidth, the flops, and most importantly the interconnect.

So what does that look like ? Well if you build around LPDDR’s lower bandwidth, lower your interconnect latency, you actually can beat Nvidia on decode with a fraction of the silicon.

[An.twi @SupremeAntwi]

@masterofnone @KrackedDevs @NousResearch One day when I fly to Malaysia, I hope to attend!

[An.twi @SupremeAntwi]

@MrBeast Zero

[An.twi @SupremeAntwi]

@MrBeast This is Ai

[An.twi @SupremeAntwi]

@cgpov Love it

[Harrison Kinsley @Sentdex]

Such an iconic scene in Breaking IPO

[Soontiago @soontiago]

🔊 the sound is from raking a plastic comb, recorded w voice memos in my closet. comb → codex. created options that randomised the pitch and tested it using a dial kit-like set up. My final choice was closer sounding to a disposable camera's dial (still using a comb), where it up pitches the faster you turn the dial

[An.twi @SupremeAntwi]

@awkwardgoogle I miss it so much… what a time to be alive.

[An.twi @SupremeAntwi]

@Rainmaker1973 Scientific evidence of the machine elves??

[Owen Carey @owenthcarey]

Practicing LeetCode at 32 years old.

[An.twi @SupremeAntwi]

Hmmm

[zeb @zebassembly]

this just blew my fucking mind

[Praveen Kumar @praveenisomer]

Dither + ASCII + Motion =

[An.twi @SupremeAntwi]

@jahirsheikh8 @masterofnone This is why it’s so fun

[Jahir Sheikh @jahirsheikh8]

Me explaining my idea to Claude Fable 5 at 3 AM be like:

[An.twi @SupremeAntwi]

@Claude1o Wow, impressive. It’s a very interesting theory and I’d like to see where you take it. I’d like to learn more about this and I appreciate you providing some resources for self-reading. Great work!

[GitHub @github]

We heard you. And we agree.

In light of recent developments in physical media, GitHub is proud to announce that you can now obtain your public repo on CD-ROM.

Keep it. Lend it to friends. Pass it on to your children.

Your code is physically yours, forever. Until you lose it, let's be real.

Order yours today.
https://t.co/z041pdMH7h

[An.twi @SupremeAntwi]

@IceSolst Wait you might be cooking something here… just gave me an idea

[An.twi @SupremeAntwi]

@RespectfulMemes What is faded in the background of this…?