Inkling-2-Small on four DGX Sparks, exact TP4: v0.3
v0.3 of the recipe that serves Thinking Machines' Inkling-2-Small across four GB10 hosts. The checkpoint's weights and quantization are unchanged, and text replies are the same as v0.2's.
- Images, on by default: PNG, JPEG and WebP at their own size (up to 4,096 pixels a side and 10,816 image tokens a request), using the checkpoint's own vision tower on rank 0. A 1920 x 1080 image starts answering in 0.973 s, a 3840 x 2160 one in 4.294 s.
- Image correctness: pixels equal Transformers' processor's, bit for bit. Against the fp32 PyTorch reference, forced along the server's replies on 14 image prompts, 696 of 723 served positions agree. Every departure is where the reference's top two tokens are under 1 logit apart (largest gap 0.59).
- Prompts fill faster: cold time to first token is 1.00 s at 2k (v0.2: 1.18), 17.7 s at 32k (v0.2: 20.7) and 91 s at 126,000 tokens (v0.2: 102).
- DSpark at the full 1,048,576-token context: 19.43 tok/s by default on a 1,038,466-token book. v0.2 managed 12.5 plain, on a synthetic 1,048,000-token state.
- Kept prompts on NVMe (opt-in): a 128,581-token prompt answers in 0.394 s after a server restart, against 95.87 s cold.
- Memory reserve 6 GiB (was 8): that keeps 8 idle 128k prompts at the default context (v0.2: 7).
Thanks to @0xSero. Several ideas this recipe builds on come from their inkling2-exl3 (https://t.co/ZegwndKst2):
- /health reporting KV capacity and admission;
- the benchmarking process behind https://t.co/BPMLFtxev7: one command, a fixed prompt set, a bandwidth ledger, needle sessions;
- FP8 weight-only projections for the DSpark drafter. Ours is MXFP8, with a power-of-two scale per 32 values.
No code was copied, but these ideas shaped how this recipe is measured and served. Thank you.
Release, tarball and sha256: https://t.co/zspGUoZTk5
TensorFold 1.3.3 is out 🚀
124 PR's merged and 34 Issues resolved in 24h
GLM-5.3-Flash went from 151.51 to 172.39 tok/s in median drafted decode after a 1,040,384-token prompt on four GB10s at TP=4. That’s 13.8% faster on CUDA with the same output tokens in that test. Thanks to @MiaAI_lab for the kernel behind that gain.
- On Metal, GLM-5.3-Flash gets DFlash2 drafting. On a Mac without tensor units, GLM cold prompts from 2K to 64K took about 5–6% less time in the release tests.
- MOSS-Image generates pictures on CUDA through /v1/images/generations. GLM-OCR runs on HIP, and Idol gets its image encoder on Metal.
- DeepSeek-V4.1 gets structured output in the dsv41-cuda source build. Long CUDA prompts now prefill in chunks alongside running replies by default.
- LFM2-MoE, LFM3, MiniMax-M3 and Qwen3.5-2B on Metal can fill new prompts between other streams’ decode rounds.
- The server adds per-request deadlines, a prompt-cache flush endpoint, and deferred and namespaced tools.
Massive thanks to everyone who sent patches, tested on their own hardware, and filed repros. @AIFlow_ML@bikepresentball@Bizuayeu@devopsfortunato@unigilby@javierbits@jayleaton@teleogen@niklaslenz_ai@plotarmordev@woolsey@petruspennanen@andersjw_@shantanugoel@dennis_rye, and on GitHub AbrahamCete, BadAd84, Ch0mpaa, christopherowen, CodingEmil, eXsoR65, Flyinpiz-dev, garvrao80, hobbesd, jschmied, kotevcode, MaverickKB, millaguie, nuk3s, SergiioB, supernovae, tacosta and ultradaoto. I really appreciate the time you put into this.
@plotarmordev@jayleaton@petruspennanen@MiaAI_lab@volatilemarkts
TensorFold 1.3.3 is out 🚀
124 PR's merged and 34 Issues resolved in 24h
GLM-5.3-Flash went from 151.51 to 172.39 tok/s in median drafted decode after a 1,040,384-token prompt on four GB10s at TP=4. That’s 13.8% faster on CUDA with the same output tokens in that test. Thanks to @MiaAI_lab for the kernel behind that gain.
- On Metal, GLM-5.3-Flash gets DFlash2 drafting. On a Mac without tensor units, GLM cold prompts from 2K to 64K took about 5–6% less time in the release tests.
- MOSS-Image generates pictures on CUDA through /v1/images/generations. GLM-OCR runs on HIP, and Idol gets its image encoder on Metal.
- DeepSeek-V4.1 gets structured output in the dsv41-cuda source build. Long CUDA prompts now prefill in chunks alongside running replies by default.
- LFM2-MoE, LFM3, MiniMax-M3 and Qwen3.5-2B on Metal can fill new prompts between other streams’ decode rounds.
- The server adds per-request deadlines, a prompt-cache flush endpoint, and deferred and namespaced tools.
Massive thanks to everyone who sent patches, tested on their own hardware, and filed repros. @AIFlow_ML@bikepresentball@Bizuayeu@devopsfortunato@unigilby@javierbits@jayleaton@teleogen@niklaslenz_ai@plotarmordev@woolsey@petruspennanen@andersjw_@shantanugoel@dennis_rye, and on GitHub AbrahamCete, BadAd84, Ch0mpaa, christopherowen, CodingEmil, eXsoR65, Flyinpiz-dev, garvrao80, hobbesd, jschmied, kotevcode, MaverickKB, millaguie, nuk3s, SergiioB, supernovae, tacosta and ultradaoto. I really appreciate the time you put into this.
@plotarmordev@jayleaton@petruspennanen@MiaAI_lab@volatilemarkts
Inkling-2-Small v0.2 is out: exact TP4 on 4× DGX Spark, now faster.
Decode speed vs v0.1:
→ 520k-token book: 25.2 tok/s (+50%)
→ Prose: 53.9 tok/s (+35%)
→ 129k-token context: 67.8 tok/s (+32%)
→ Code: 79.7 tok/s (+31%)
→ 1 stream, plain: 33.9 tok/s (+9%)
Several streams, total:
→ 4 streams: 118.5 tok/s (+30%)
→ 4 streams, 128-token replies: 118.7 tok/s (+45%)
→ 8 streams, plain: 86.7 tok/s (+10%)
Time to first token:
→ 32k prompt: 30.6 s → 20.7 s (−32%)
→ 128k prompt: 221 s → 102 s (−54%)
→ 512k prompt: 46 min → 14 min (−70%)
What's new:
• fp32 attention, 4–5× closer to float64 than v0.1's
• Decode in fewer kernel launches
• Leaner DSpark drafts, with FP8 drafter projections
Replies stay exact to the checkpoint: same weights, same quantization, and no dependence on timing, drafting, caching or other streams.
One command installs it on four Sparks. Release, tarball and sha256:
https://t.co/OJ1ZAqdlDs
My first local AI recipe is out!
Inkling-2-Small on 4× DGX Spark, using Thinking Machines’ original checkpoint weights, with no additional weight quantization. Built on TensorFold with DSpark speculative decoding.
My measurements:
• Single stream with DSpark: 70 tok/s on tool calls, 61 on code, 56 on thinking (31 plain)
• 4 concurrent streams: ~101 tok/s combined at the default settings
• First token at 2K: 1.3 s cold, ~0.2 s cached
• Context up to 1M tokens
https://t.co/OJ1ZAqdlDs
Thanks to @thinkymachines for the weights, and @ashxhart + the @tensorfold_dev contributors for the engine.
Would love independent runs and feedback!
@MiaAI_lab Ok, so bandwidth between your cards and mine should be about the same, not sure how much inference would improve at pcie5 16x, or at what context size at least. Thanks for sharing!
@MiaAI_lab Appreciate the 300w! Just got a pair of max-Q’s I’m installing in an old 256gb ddr4 / pcie-4 rig I’m going to see what I can get out of it. Looking forward to all of your dual 6000 recipes, hope to get GLM running tonight to compare to your results.