Consigliere

A SaaS-preneur chasing dollars by day, sneaking out of my cave occasionally to teach, and secretly indulging in my gacha addiction (shhh, don't tell my wife!)

[Consigliere @seree]

@AtlasInference @worawisut @SpectralCom @AIatAMD He surely benefits with his strix halo 😇

[Consigliere @seree]

@LeakerApple @grok how do uou think honestly?

[Consigliere @seree]

Hope you can survive this without OOM first, buddy! 🤣🤣

#DGXSpark

[Consigliere @seree]

@msitweets @nvidia @grok compare with dgx spark

[Consigliere @seree]

@1337hero @mr_r0b0t @leopardracer LoL

[Consigliere @seree]

@sakurayukiai @AtlasInference 19-32 tps, this is MTP enabled already

[Consigliere @seree]

First time benchmarking rust-based inference engine by atlas (@AtlasInference)

Model tested:
Qwen3.6 27B FP8 with MTP enabled

Things noted:
- startup speed, OMG, it's much much faster than other inference engines I've tried!
- token generation speed is great but I think it can be faster, let me play with it more and will keep this posted
- memory consumed more than vllm, probably the default KV cache is 16bit?
- crashed with long context input, yes, it crashed my DGX Spark and force shutdown without a reboot 😭

Anyway, i think the future is quite interesting for this inference engine, will playing more with it!

[Consigliere @seree]

@AtlasInference I'm adjusting various parameters to experiment it more, will DM what I found very soon.

Next station will be 35B A3B NVFP4 with the recent Qwen/Qwen3.6-35B-A3B-NVFP4 model instead od RedHatAI's one.

BTW: thanks for your hard work!

[Consigliere @seree]

@RaminNasibov Syndicate by Bullfrog

[Consigliere @seree]

@ifourth That's why we need a brand instead of us, 3AM Lab will do!

[Consigliere @seree]

Wait, my DGX Spark was cooked! 🔥🔥🔥

[NVIDIA AI @NVIDIAAI]

A new era of PC.

25.0528, 121.5990

[Consigliere @seree]

@MrPeterLMorris The latest frontier models also can't. That's the nature of LLM.

[Consigliere @seree]

@MrPeterLMorris It's not broken, I just ran a few recipes last night. 🔥🔥

[Consigliere @seree]

@mr_r0b0t @NVIDIAAI Can’t wait to get the 2nd Spark for this!

[Consigliere @seree]

It's NOT about not having enough compute.

It's about what we're WASTING on the harnesses!

We desperately need smarter harness optimization, not just throwing more power at the problem!

#AITalks

[Consigliere @seree]

@MrPeterLMorris To simplify local inferencing on DGX Spark, you gotta try sparkrun. Life Savior!

[Consigliere @seree]

AEON Ultimate Qwen3.6 27B NVFP4 DFlash - benchmarked!

Quite nice speed compared to the base model or even the MTP one. 🔥🔥

#DGXSpark

[Consigliere @seree]

Benchmarking the famous Qwen3.6 27B FP8 DFlash on my DGX Spark.

Speed boosted from 16.2 tps (MTP) to 26.9 tps (DFlash). 🔥🔥

Now testing with OpenCode to see if it will broke at tools calling or not.

Will try AEON’s NVFP4 DFlash version very soooooon!