Gv9ch

SWE 9-5 and solo entrepreneur
Building:
@llmrequirements
@agentarmory

[Gv9ch @Gv9ch]

@llmrequirements Deepseek v4 flash is literally 200gb model which runs on 2 sparks

[Gv9ch @Gv9ch]

@MiaAI_lab I was running it for some time but didn't get good results and moved to dsv4. I think the problem was not decode but cache and PP. Is it now good enough for production use?

[Gv9ch @Gv9ch]

@Teknium do your team run Hermes with codex subscription? I like how Hermes works with my local inference. Wanted to move codex agents to run on Hermes, but so far it is not functioning:

[Gv9ch @Gv9ch]

@MiaAI_lab I put sol to the task, it suggested to add VLLM_PREFIX_CACHE_RETENTION_INTERVAL=57344. So far going smooth, no 300k cold prefills one after another.

[Gv9ch @Gv9ch]

@MiaAI_lab I am currently using this one. Looks like something off with the cache. Multiple Hermes session constantly run cold if they are executed in parallel.

[Gv9ch @Gv9ch]

@Reederey Do you even need glm 5.3 if your acceptance rate is 100%?

[Gv9ch @Gv9ch]

@MrCatid Draft acceptance is actually pretty high on real agentic tasks. I don't have that benchmaxed 67 tokens but I have stable 40 single stream with high think. Prompt processing is super slow indeed, but you can put 4k batched tokens to sacrifice some kv cache pull for 80% speed up.

[Gv9ch @Gv9ch]

@MiaAI_lab Can you plus add some changelog with the fixes? I setup the initial version and kinda hesitant on touching it.

[Mia @MiaAI_lab]

Run GLM-5.3 Flash EXL3 with DFlash2 for 2x DGX Sparks with ease ✨

- 900k context
- 982,612 FP8 KV cache
- Full image and video support
- 300k stress passed, 874 prefill tok/s.
~63 tok/s in single stream structured/code

155 tok/s on 4 concurrent sessions (structured) - that means ~37 tok/s on each session!

Expect slower speeds in prose, the drafter just misses more. DFlash2 just guesses worse on natural text.

Why EXL3?

Compared to FP8, here are the KLD numbers:
FP8 - 0.0206 KLD, 328 GB
EXL3 4bpw - 0.0246 KLD, 176 GB

EXL3 4bpw is ~0.004 nats behind native FP8 at 54% of the size (!)

BF16 > FP8 ≈ EXL3 4bpw > NVFP4

Speed is about the same as NVFP4, and it seems to handle concurrent sessions better. Expect further improvements over time!

Special thanks for @BrandonMusicKy for the quant.

Get it here:
https://t.co/6o0dNhW2XS

[Gv9ch @Gv9ch]

@dominiksumer @jonathan_wilke Doesn't seem that this architecture is scalable for the event based analytics

[Gv9ch @Gv9ch]

@Code_X_Captain Deepseek v4 flash dwarfstar

[Gv9ch @Gv9ch]

@carsenklock 2 cables> 1 cable

[Gv9ch @Gv9ch]

@german_person But 550 is not power draw, it is power rating for PSU. Actual draw is lower. And how much tokens to you get for that? Looks like it is missing actual calculation.

[Gv9ch @Gv9ch]

@TheShiftReportX @NousResearch How? So far different profile = different bot

[Gv9ch @Gv9ch]

@MiaAI_lab I am maybe missing something:
https://t.co/IsCD8agqYK

[Gv9ch @Gv9ch]

@MiaAI_lab Ninfer, not sglang

[Gv9ch @Gv9ch]

@Tech2Wild Try dflash, on 3.6 it gave 40+ tps.