Matt Potma

[Matt Potma @MattPotma]

@robj3d3 @AnthropicAI What are they doing?

[Matt Potma @MattPotma]

@francoisfleuret They likely won't serve big models from NVMe, so it's unclear to me if the number of layers and sparsity are true limitations or just not worth exploring.

[Matt Potma @MattPotma]

@francoisfleuret You got me curious, and I'm similarly ignorant, so I had claude make this: https://t.co/D5gvDgjl3X

<=1B active is reasonably fast, but I don't know if 1000:1 sparsity works to get to 1T params. Deepseek/Kimi are ~3% sparse

[Matt Potma @MattPotma]

@francoisfleuret @redchethan Fast but maybe not blazing.

Assume FP8, so 100 MB off SSD. 5 GB/s would be 20 ms per token, or 50 tok/s.

But the 5 GB/s is even optimistic, because our queue depth is low and we can't buffer streaming experts. More like 30/s. Impressive for $2k hardware, but not blazingly fast

[Matt Potma @MattPotma]

@hankgreen I'm disappointed that snap, crackle, and pop were rice krispie noises and not high order derivatives of velocity

[Matt Potma @MattPotma]

@bryan_johnson Will you make a video on the effects of the change in your diet?

[Matt Potma @MattPotma]

@jarredsumner @LukasHozda What model did most of the work on the port? Did you use mythos, or all opus/sonnet?

[Matt Potma @MattPotma]

@ThePrimeagen It's something for sure.
I'm sure it was a lot of work and many $$ of tokens, but the result sure seems significant.

[Matt Potma @MattPotma]

@_katetolo Can you share that image? I want to set your boyfriend as my lock screen too

[Matt Potma @MattPotma]

@kapilansh_twt I just did this with age and sops. Encrypted env files are checked in. Add a new user's public key and they can decrypt.

[Matt Potma @MattPotma]

@MrCatid @luigifcruz These are much closer to rack servers than gaming PCs, except nobody makes a case that fits 4 non-blower gpus with room to breathe

[Matt Potma @MattPotma]

@jun_song check out Deepseek's conditional memory paper from a few months ago. Something like that might have the effect you're referring to.
Take a larger memory bank sitting in system RAM or even on disk and load up a comparatively small n gram embedding onto GPU at each inference step

[Matt Potma @MattPotma]

@ThePrimeagen I want to know what AI water usage is in units of ChatGPT messages per kilogram of beef

[Matt Potma @MattPotma]

@The_Plain_Bagel Will you make a video about it so I know what to think of the plan?

[Matt Potma @MattPotma]

@hetmehtaa Sticky notes

[Matt Potma @MattPotma]

@Mashcr0ft Sorry mate, the algorithm put this in my feed.

[Matt Potma @MattPotma]

@KyeGomezB @realsigridjin @yuekun_yao @hayden_prairie Do you think Opus 4.7 could be a T=1 loop of the same model as mythos?

The number of regressions of 4.7 makes me think they've tried something novel for it, rather than iterative improvement on 4.6

[Matt Potma @MattPotma]

@davepl1968 The problem is that the big matrices get chopped up and run in non deterministic ordering, dependent on the GPU scheduler. And floating point arithmetic is not commutative.
Also temperature in the model is random and not a fixed seed

[Matt Potma @MattPotma]

@NoahKingJr Goat farm in Chile

[Matt Potma @MattPotma]

@hiddentruthcent @bryan_johnson Larry Ellison will still be alive, they could just become roommates