Adi

Currently Human. Working on Microsoft Ads

[Adi @rabbitpersona]

Cerebras is backed into a corner with the memory bandwidth bottleneck.

They were in the forefront of inference with high onchip memory, but with the advent of long-context, they need to account for the KV cache, which can become much larger. No wonder they use Trainium for the prefill and WSE for the decode.

Single digit TB/s memory bandwidth to WSE.

Now Nvidia has shown that with Rubin doing prefill, and Groq-LPX doing decode, they can pipe more with NVLink.

Hundreds of TB/s memory bandwidth to Groq-LPX

Cerebras made the wafer chip in an interesting way, since no foundry can support a mask the size of a wafer chip, so with clever collaboration with TSMC, they take a bunch of identical masks and copy-paste it on the entire wafer, without any deadzone in the middle, and voila.

Now if they want to increase memory bandwidth they need to stamp the same mask multiple times, adding more PHY to the edges means adding them to the center as well, which is not ideal. Switch masks during a wafer run is also not super possible with current machinery.

I eventually like the Taalas approach that burns the weights into silicon, they have a hilarious (almost spam like) website that presents their almost ridiculous speed, https://t.co/dAQSQ1XItM, but with their acquisition by AMD, I'm skeptical if we will see something good come out.

Cerebras proposes some 3D wafer stuff to solve it, let's see how far they can push this. Maybe the Nvidia way of using more proven hardware is the way, but again they also propose some 3D stuff, so I don't know what's going on.

[Adi @rabbitpersona]

Implemented a version of @everestpipkin default filename TV, https://t.co/2cACCt9UGC

Truly a great way to glimpse into the life of others

[Adi @rabbitpersona]

There is soooo much information density, that I'm unable to find the patience to read "Too Much to Know" a book literally about how people in the 16th century faced the same amount of overwhelm with regards to information.

[Adi @rabbitpersona]

@tessafyi I might dissolve into complete paranoia if I start to secure everything, might as well avoid being a target by making myself a victim not worth the time.

[Adi @rabbitpersona]

@annasofialesiv https://t.co/QXNitFiwlq