Philip Myers

[Philip Myers @sugarflows]

@Gini_220 3452839

[Philip Myers @sugarflows]

@zainmfj @CoreDumpped will you ship outside US ?

[Philip Myers @sugarflows]

@eliebakouch Same.

[Philip Myers @sugarflows]

1/ Excellent, careful article. Some matched data from a close cousin of the card: 4× RTX 2000E Ada (16GB, 224 GB/s, 50W) on a WRX90 / Threadripper PRO 9955WX, every card at PCIe Gen4 x8, vLLM 0.29.

2/ On section 5 (TP without P2P): our first CUDA P2P test ran at 0.95 GB/s and returned NaNs, with AMD-Vi IO_PAGE_FAULTs in the kernel log. Setting amd_iommu=off fixed it: 12.2 GB/s and verified data, then ~13.15 GB/s on every GPU pair. The NCCL_P2P_DISABLE workaround may be masking an IOMMU translation issue rather than a hardware limit — worth checking dmesg.

3/ With P2P working, TP4 scaled well at batch 1 (tok/s, TP1 / TP2 / TP4):
• Llama-3.1-8B FP8: 24.2 / 44.8 / 80.0 (TP4 = 3.30× one card)
• Gemma-4-12B AWQ: 23.0 / 41.4 / 68.9 (3.03×)
• Gemma-4-12B AWQ + MTP depth 3: 135.7 at TP4
NCCL 8 KiB all-reduce: 10.6 µs at TP2, 18.0 µs at TP4.

4/ A simple model predicted every point within a few %: time per token ≈ bytes read ÷ (TP × ~205 GB/s) + 2 × layers × all-reduce latency + ~1.5 ms fixed. Applied to your Qwen3.8-27B NVFP4 TP4 row, it gives ~53 tok/s against the 49.0 reported.

5/ Agree with your TP2-vs-TP4 reversal under load: our TP4/TP2 gain fell from 1.78× at 1 request to 1.33× at 64. Faster cards should scale less well at TP4, because communication and fixed overhead don't shrink with card speed. Different card from yours (Ada, not Blackwell), so treat these as a comparison point, not a substitute.

[Philip Myers @sugarflows]

They finally made one... https://t.co/GRZbrgTlBf via @YouTube

[Philip Myers @sugarflows]

@_davidcoles @itsjustmarky Worth dropping the power to the overheating cards ?

[Philip Myers @sugarflows]

@itsjustmarky I’ve just bought 16 of them, not what I wanted to hear. Might have to make do with TP=4 while I work out solution.

[Philip Myers @sugarflows]

@itsjustmarky Use rtx pro 4000, single slot and low power (150W each). Only need to bifurcate one pcie slot to get 192gb vram total at TP=8.

[Philip Myers @sugarflows]

@sudoingX RTX 2000E Ada Generation 16 GB in TP=4 or TP=8 config.

[Philip Myers @sugarflows]

@TechMDAI 1 x 512 GB M5 Ultra Mac Studio ~ 2 x 256 GB M5 Ultra Mac Studio ~ 4 x 128 GB M5 Max Mac Studio ~ 8 x 64 GB M5 Pro Mac Mini ~ $20k

[Philip Myers @sugarflows]

TP=4 ?

[Philip Myers @sugarflows]

@Hikari_07_jp It is worth trying Multi-Instance GPU (MIG) for 2 or 4 splits per GPU if model size allows in order to delay contention issues while maintaining a form of concurrency.

[Philip Myers @sugarflows]

Thanks, a few more questions and I leave you in peace ! What is the MTM of your P8?
(e.g. 30HHxxxxx)
What BIOS version are you running? I bought three of these machines at $8k each and they are now over $40k, I was beginning to despair about not being able to get the Max-q(3) cards I have working.

[Philip Myers @sugarflows]

@GPTWare Thanks, will try again (I got no support from Lenovo as I bought generic bulk Max-Q cards) how many cards did you manage in the one workstation, I guess two is the limit?

[Philip Myers @sugarflows]

@GPTWare I tried this but struggled with non-Lenovo rtx 6000 gpus- did you connect the power cables directly to the motherboard. Would be great to know as I will try again.

[Philip Myers @sugarflows]

@GPTWare The RTX PRO 6000s are in the Thinkstation ?

[Philip Myers @sugarflows]

@vrloom Set the gpu wattage lower during the hot weather.

[Philip Myers @sugarflows]

@ivanfioravanti With MIG you can split 1 rtx pro 6000 into 4 x 24 gb or 2 x 48 gb logical cards - gives better results than concurrent users on 1 model

[Philip Myers @sugarflows]

@Hikari_07_jp TP=4 !