Axitek

[Axitek @axitek]

@Florious95 Yes. I have this setup running in primary/delegation models for my agent. GLM as primary does a good job orchestrating, and QFN just wants to go faster!

[Axitek @axitek]

Qwen3.8-Flash-Next:IQ3-S (Strata v1.0.42) vs GLM-5.3-Flash:Q4+ (Maya v1.0.32)

State of the art in local inference - two of the best models to run locally now hit incredible prefill rates (10k tps for QFN!)

GLM is smarter - QFN is faster. Which do you pick?

[Axitek @axitek]

@loktar00 I was trying to simplify and explain Cascade mode (https://t.co/XOx43oZuAr) to my friends, and ended up calling it token ring, I know it's not quite right but it makes good sense!

[Zach Mueller @TheZachMueller]

Here’s essentially the expansion board broken down. It’s got two PCIE switches inside, meaning that you only need a singular x16 slot to power 8 GPUs (or two).

Let’s talk about how it works.

First, the main 8 GPUs slots

You have two methods of setup here:
- Direct connect via two PCIE x16 -> 4x x8 MCIO
- Cascade via a single PCIE x16 -> 2x x8 MCIO

The difference here is how much of your GPU usage do you want heading back to your CPU vs keeping everyone around one dimension for faster GPU -> GPU communication.

A quick way to think about it: model fits on 8 GPUs? Cascade. CPU offload? Probably direct connect.

But what IS cascade?

Let’s look back at the diagram.

Direct connect says that we’re going to plug in two slots worth of x8 connectors into both 0-7 and 8-15 slots (two halves of our PCIE x16) into the separate halves. This then populates the switch in such a way that we can support 8 GPUs, each on separate 4-GPU P2P groupings.

The nice part here is because the PCIE switches live in this board, you can take four additional MCIO cables and go from the output downlink into any one of the direct connect interfaces and add an additional GPU.

BUT, if you want the two groups of 4 GPUs to interact, they’ll get tunneled through a single x16 slot worth of transfer, and have to go through the CPU. Not a fun bottleneck when dealing with 64 lanes worth of GPUs on either side.

The alternative is what we call “cascade.”

If we don’t want to worry about the CPU side (e.g. model inference), we can instead only hook up one PCIE slot as the input, then take the downlink from SW1 and use it as the INPUT to SW2.

It’s still the bandwidth of a singular PCIE5 x16 slot, but you avoid going through the CPU which saves you some cycles.

For the full 12 GPUs, you would need 3-4 full x16 slots on your motherboard. With the direct connect slots running through the host CPU still (since they are direct connect), if you’re not doing cascading.

Or translation:
- Non-DC: slot 1 and slot 12 are powered through additional retimers and MCIO at x16 and don’t add a slot, it’s just moving your existing PCIE slots somewhere else. Then you power slots 2 and 11 through the “jump” mentioned earlier
- DC: slot 1, 12, and then 2 or 11 are powered through a retimer, and then the other (between 2 and 11) are powered through the “jump”

Now let’s get into how I plan to use this.

I have 9 GPUs in total (8 6000’s and a 6000 ADA), so I’ll be doing the Cascade, then using the SW2 output to power a 9th PCIE slot similar to the non-Cascade version that could power 10.

What I don’t know yet is which is better: TP4/PP2, or TP8 over the two P2P groups, but I expect that would be the difference of batching. I’ll probably accept some slowness and do TP8. I can’t predict how much slower that will be, so when it comes in I’ll do these full benchmarks in many different combinations.

What I want you to gather from this is these boards have internal PCIE switches, to help avoid needing to go through the CPU to have your GPUs interact, and bring you a larger lane budget than your CPU would allow.

[Axitek @axitek]

@CodeBuilder_ Here's the outcome you want

[Axitek @axitek]

Smart AND fast - Maya keeps getting better!

GLM-5.3-Flash:Q4+ (Maya-L) on Maya v1.1.0 at 256k context:
- Decode: 52 tps (+20% from v1.0.34)
- Prefill: 4,080 tps (+5% from v1.0.34)

Should I repeat this is at 256k context?

https://t.co/ZqpOwlDTv6

H/W 7x3090+128gb ddr4

[Axitek @axitek]

@CodeBuilder_ Lets figure it out! How are they connected together? I have full electrical x16 PCIe4.0 to each card (asus wrx80e sage II board, x16 risers to each card), and the p2p patched driver (https://t.co/cFTyB95t0s).

[Axitek @axitek]

@CodeBuilder_ It’s a function of the number of GPUs I think, I tried two configs on v1.1.0 -

4x3090 maya-M @ 256k: prefill 2753 tps, decode 44 tps
7x3090 maya-L @ 256k: prefill 4080 tps, decode 52 tps

[Axitek @axitek]

@TheZachMueller A year from now looking back, will you have wished you got more storage while it was steal cheap at today's prices? 😅

[Axitek @axitek]

@loktar00 Why does hardware fail on sundays

[Axitek @axitek]

Thanks @PeasantSmith @needmorevram ! Dflash is working

[Axitek @axitek]

@nawariokra Prefill should, decode won’t

[Axitek @axitek]

@ShankhadeepSho1 I mean it’s what the post says. Strata with Qwen3.8-Flash-Next. Download it and run https://t.co/VySdZHns1H, no special tuning. 10k prefill at 256k context. Hardware specs also shared in thread.

[Axitek @axitek]

@tteokl_ @coldniko Yes

[Axitek @axitek]

Qwen3.8-Flash-Next:IQ3_S (Strata v0.1.42) @ 256k context

Prefill: 10,007 tps
Decode: 99 tps

Great work @coldniko, lets go faster!

7x3090+128gb ddr4

[Axitek @axitek]

@HouPeiyi @unbug I stopped giving complex work to QFN:IQ3_S, it kept making mistakes and going around in circles with tool calls. Maybe Q6 would be an improvement?

[Axitek @axitek]

@knorrium @modretro Oh nice! I have two Tetris cartridges, would be awesome to flash custom roms to them.

[Axitek @axitek]

@PeasantSmith @loktar00 whatever is cheaper? the cable. lets hope it's the sata cable.

[Axitek @axitek]

@gabu3d_pl @coldniko Good idea.

[Axitek @axitek]

@geldeki at what point does it become a datacenter? https://t.co/iCyMdzg5l5

[Axitek @axitek]

I can feel the warmth of decode

[Axitek @axitek]

@geldeki Yes, this feels frontier quality, and both GLM and QFN can run on consumer hardware. This benchmark is on the high end with 7x3090+128gb ddr4 with an Q4+ (Maya-L) quant.

[Axitek @axitek]

@dogrosemk @coldniko Oh cool interesting. I'll get an agent to look at the settings

[Axitek @axitek]

@dogrosemk @coldniko This is IQ3-S, it can go faster on a smaller quant