Bernard Huang @bernardh789 - Twitter Profile

Twelve machines. 2.4 terabytes of unified memory. One coordinated inference problem. Seven DGX Sparks: 896 GB, CUDA, brutal at compute. Five Mac Studios: 1.5 TB, Metal, brutal at bandwidth and unbeatable per watt. Plus three Mac minis, a 5080, some V100s. At 4-bit, that Studio memory holds a 1 to 2 trillion parameter model with room left over for a 900,000-token context. The weights fit. That was never the problem. The problem is prefill. Before a model with that context says one word, it has to read everything you gave it. On Apple Silicon that is roughly 400 tokens a second. A 900K-token load is over half an hour of silence. Decode is fine, 25 to 30 tokens a second all day, quiet, 300 watts. It’s the first word that costs you. The Sparks prefill four to five times faster and can’t hold the model. The Studios hold the model and can’t prefill. Everyone with mixed silicon owns both halves of the answer and no way to join them. So we joined them. NVIDIA prefills, Apple decodes, one request. Two engines that share no cache format, no framework, no vendor. Instead of transferring a cache neither can read, the prefill box computes the decoder’s finished cache using the decoder’s own weights and writes it into the decoder’s prefix store. About 10 KB per token crosses the wire, over plain 10 gigabit Ethernet through the two switches in the first picture. No RDMA, no Thunderbolt. Measured today on DeepSeek-V4-Flash, 284B, 241,000-token cold load: Mac Studio alone, 12 minutes to the first word. Two Sparks feeding it, 3 minutes. Same prompt again, 19 seconds. Decode identical. Answers identical. That ratio is what makes the goal real: prefill 900K on the Sparks for a trillion-parameter model living on five Studios. Tonight we took the prefill window from 262K to 524K. Not finished, and every number gets posted either way. Why this is a paradigm shift for us: our agent is persistent and has 54MB of memory files..and then we drop a transcript or a codebase on top of it mid-conversation. The wait was the product’s real cost. It isn’t anymore. All credit to everyone who contributed to these concepts before us. We distill knowledge from all the greats and give credit to all. Standing on the shoulders of GitHub wizards unapologetically without fear of failure or judgement. Local ai must win! https://t.co/X94wAZsND8 #localai #heterogeneousinference #dgxspark #applesilicon

volatilemarkts's tweet photo. Twelve machines. 2.4 terabytes of unified memory. One coordinated inference problem.

Seven DGX Sparks: 896 GB, CUDA, brutal at compute.
Five Mac Studios: 1.5 TB, Metal, brutal at bandwidth and unbeatable per watt.
Plus three Mac minis, a 5080, some V100s.

At 4-bit, that Studio memory holds a 1 to 2 trillion parameter model with room left over for a 900,000-token context. The weights fit. That was never the problem.

The problem is prefill. Before a model with that context says one word, it has to read everything you gave it. On Apple Silicon that is roughly 400 tokens a second. A 900K-token load is over half an hour of silence. Decode is fine, 25 to 30 tokens a second all day, quiet, 300 watts. It’s the first word that costs you.

The Sparks prefill four to five times faster and can’t hold the model. The Studios hold the model and can’t prefill. Everyone with mixed silicon owns both halves of the answer and no way to join them.

So we joined them. NVIDIA prefills, Apple decodes, one request. Two engines that share no cache format, no framework, no vendor. Instead of transferring a cache neither can read, the prefill box computes the decoder’s finished cache using the decoder’s own weights and writes it into the decoder’s prefix store. About 10 KB per token crosses the wire, over plain 10 gigabit Ethernet through the two switches in the first picture. No RDMA, no Thunderbolt.

Measured today on DeepSeek-V4-Flash, 284B, 241,000-token cold load:
Mac Studio alone, 12 minutes to the first word.
Two Sparks feeding it, 3 minutes.
Same prompt again, 19 seconds.
Decode identical. Answers identical.

That ratio is what makes the goal real: prefill 900K on the Sparks for a trillion-parameter model living on five Studios. Tonight we took the prefill window from 262K to 524K. Not finished, and every number gets posted either way.

Why this is a paradigm shift for us: our agent is persistent and has 54MB of memory files..and then we drop a transcript or a codebase on top of it mid-conversation. The wait was the product’s real cost. It isn’t anymore.

All credit to everyone who contributed to these concepts before us. We distill knowledge from all the greats and give credit to all. Standing on the shoulders of GitHub wizards unapologetically without fear of failure or judgement. Local ai must win!

https://t.co/X94wAZsND8

#localai #heterogeneousinference #dgxspark #applesilicon

110

1K

96

957

183K

Bernard Huang

@BernardH789

24 days ago

@dhh @AIatMeta OMG

0

8

Bernard Huang

@BernardH789

24 days ago

GPT的训练方向上好像有偏差了。对于潜在错误、可能发生的问题过于注意，过于小心翼翼。只要深度使用GPT都应该能发现，比如询问一个功能如何实现，答案最后总会带一个“避免...”，内容是某个极小概率可能会发生的异常。Codex最近经常过多写一些超过必要的防御性代码跟这个也有关系。 #Codex #ChatGPT @thsottiaux

0

93

Bernard Huang

@BernardH789

Last Seen Users on Sotwe

Trends for you

Most Popular Users