Lazy Analyst turned Vibe Coder. Trying not to waste an M5 Ultra 512GB. Running local models, testing MLX, and spending time automating things I can do manually
Hello, against good advice on X, I went ahead and bought a M5 Ultra 512GB to start on my local llm journey.
I'mma gonna benchmark it and find an optimal config for M5 512GB. Something that is smart and fast enough. Then maybe use it to make something cool. Requests welcome!
@ho_bevan@thinkymachines@jundotkim@Apple On 50 short greedy questions, drafts on vs drafts off gave different text on 57 of 100 replies. I didn't grade quality and didn't check text on the 4K runs, so I can't say better or worse yet.
Just tested Inkling 2 Mini on oMLX. Wow, if you enable DSpark drafting, it increases the decode speed from 54.4 to 65.9 tok/s (+21%) on 4K mixed prompts.
https://t.co/Vv0BP36DFl
Model by @thinkymachines , oMLX by @jundotkim , M5 Ultra 512GB by @Apple
TensorFold 1.3.3 is out 🚀
124 PR's merged and 34 Issues resolved in 24h
GLM-5.3-Flash went from 151.51 to 172.39 tok/s in median drafted decode after a 1,040,384-token prompt on four GB10s at TP=4. That’s 13.8% faster on CUDA with the same output tokens in that test. Thanks to @MiaAI_lab for the kernel behind that gain.
- On Metal, GLM-5.3-Flash gets DFlash2 drafting. On a Mac without tensor units, GLM cold prompts from 2K to 64K took about 5–6% less time in the release tests.
- MOSS-Image generates pictures on CUDA through /v1/images/generations. GLM-OCR runs on HIP, and Idol gets its image encoder on Metal.
- DeepSeek-V4.1 gets structured output in the dsv41-cuda source build. Long CUDA prompts now prefill in chunks alongside running replies by default.
- LFM2-MoE, LFM3, MiniMax-M3 and Qwen3.5-2B on Metal can fill new prompts between other streams’ decode rounds.
- The server adds per-request deadlines, a prompt-cache flush endpoint, and deferred and namespaced tools.
Massive thanks to everyone who sent patches, tested on their own hardware, and filed repros. @AIFlow_ML@bikepresentball@Bizuayeu@devopsfortunato@unigilby@javierbits@jayleaton@teleogen@niklaslenz_ai@plotarmordev@woolsey@petruspennanen@andersjw_@shantanugoel@dennis_rye, and on GitHub AbrahamCete, BadAd84, Ch0mpaa, christopherowen, CodingEmil, eXsoR65, Flyinpiz-dev, garvrao80, hobbesd, jschmied, kotevcode, MaverickKB, millaguie, nuk3s, SergiioB, supernovae, tacosta and ultradaoto. I really appreciate the time you put into this.
@plotarmordev@jayleaton@petruspennanen@MiaAI_lab@volatilemarkts
TensorFold 1.3.3 is out 🚀
124 PR's merged and 34 Issues resolved in 24h
GLM-5.3-Flash went from 151.51 to 172.39 tok/s in median drafted decode after a 1,040,384-token prompt on four GB10s at TP=4. That’s 13.8% faster on CUDA with the same output tokens in that test. Thanks to @MiaAI_lab for the kernel behind that gain.
- On Metal, GLM-5.3-Flash gets DFlash2 drafting. On a Mac without tensor units, GLM cold prompts from 2K to 64K took about 5–6% less time in the release tests.
- MOSS-Image generates pictures on CUDA through /v1/images/generations. GLM-OCR runs on HIP, and Idol gets its image encoder on Metal.
- DeepSeek-V4.1 gets structured output in the dsv41-cuda source build. Long CUDA prompts now prefill in chunks alongside running replies by default.
- LFM2-MoE, LFM3, MiniMax-M3 and Qwen3.5-2B on Metal can fill new prompts between other streams’ decode rounds.
- The server adds per-request deadlines, a prompt-cache flush endpoint, and deferred and namespaced tools.
Massive thanks to everyone who sent patches, tested on their own hardware, and filed repros. @AIFlow_ML@bikepresentball@Bizuayeu@devopsfortunato@unigilby@javierbits@jayleaton@teleogen@niklaslenz_ai@plotarmordev@woolsey@petruspennanen@andersjw_@shantanugoel@dennis_rye, and on GitHub AbrahamCete, BadAd84, Ch0mpaa, christopherowen, CodingEmil, eXsoR65, Flyinpiz-dev, garvrao80, hobbesd, jschmied, kotevcode, MaverickKB, millaguie, nuk3s, SergiioB, supernovae, tacosta and ultradaoto. I really appreciate the time you put into this.
@plotarmordev@jayleaton@petruspennanen@MiaAI_lab@volatilemarkts
How I measured. One user, 512-token replies, median of 5 runs to 128K and of 3 at 512K and 1M.
Smart means the Artificial Analysis index, 28 or more. Fast means 50 tok/s or more. The index scores models with thinking on. I timed DeepSeek and Qwen with thinking off.
Engines by @ashxhart (TensorFold), @jundotkim (oMLX) and ddalcu (mlx-serve). Models by @Alibaba_Qwen, DeepSeek, Z AI and @thinkymachines.
Questions and feedback welcomed.
I benchmarked 10 open-weight models on my Mac Studio M5 Ultra 512 GB, with prompts from 4K to 1M tokens.
The TLDR: I think - if you are a 512GB Owner, and you care about the fastest that's smart enough, you can go with Qwen3.8 Flash Next 4-bit
If you care about having the smartest that's also fast enough, you can go with DeepSeek V4.1 Flash.
Qwen3.8 Flash Next 4-bit writes 124.1 tokens per second at a 128K prompt, 129.2 at 4K and 93.2 at 1M, on TensorFold 1.2.8 with MTP.
DeepSeek V4.1 Flash 1017 Lossless writes 55.0 to 58.3 at every prompt from 4K to 1M, on mlx-serve 26.11.2.
Memory. Flash Next 4-bit peaks at 170.3 GiB even at 1M. DeepSeek peaks at 328.4 GiB at 1M, more than a 256 GB Mac has.
GLM-5.3 and Inkling 2 Small already reached 423.1 and 428.4 GiB at 128K. Peak is the whole Mac's wired memory.