Four DGX Sparks cabled in a ring, no switch, now run GLM-5.3-Flash at up to 86 tokens per second.
4 DGX Sparks, GLM-5.3-Flash XSFP4 by Bizuayeu, TensorFold 1.2.9: 86.1 tok/s on his decode test. His number: 74.14 on 1.2.7
I ran Bizuayeu's settings with the official TensorFold 1.2.9 package. His decode test came out faster than the number in his README, prompt speed is within 3 % of his.
Sampled it is a different picture. On the same four Sparks Mia's EXL3 recipe was faster in the standard test on Nov 11.
Decode, his test, temperature 0: 86.1 tok/s (his: 74.14)
Prompt speed, 39K tokens: 1,811 tok/s (his: 1,868)
Edit reply: 236.8 tok/s (his: 239.7)
199,652 token prompt: 110 s (his: 107.9 s at 199,307)
Standard test, sampled, temperature 1.0: 52.0 tok/s
8 requests at once, sampled: 89.6 tok/s in total
128K prompt, first token: 73 s
Mia's EXL3 4 bpw recipe, Nov 11, standard test, her own settings: 65.4 tok/s
The standard test is sampled and uses code and prose prompts, so it runs slower than his greedy counting test.
With Ash's bench tools the ring decodes up to 8 % below what Bizuayeu posted for his ring on Nov 7, same on TensorFold 1.2.6. No cause found yet, I may dig deeper when I find the time.
Quant: XSFP4, 165.7 GiB
Method: his bench, median of 3, temperature 0, drafts on, the long prompt one run. Standard test: median of 2 runs on 2 prompts, 512 tokens, thinking off
Recipe:
https://t.co/AkZ8VCCl08
Recipe and XSFP4 quant by Bizuayeu, engine TensorFold by Ash @ashxhart, GLM-5.3-Flash by https://t.co/sVNNSIeuhJ @Zai_org. EXL3 recipe by Mia @MiaAI_lab.
Now AWS itself created an emulator. What will be the future of Localstack, Floci, etc?
AWS Emulator is a local emulator of the AWS APIs.
https://t.co/MZxSOzWYhB