Vikesh Chevli

AI, Local LLMs & Ad Tech

[Vikesh Chevli @vikeshchevli]

@StefanMaier @NVIDIAAI @NousResearch Speed is why I landed on Qwen 3.8 Flash Next in the end. Using @MiaAI_lab's recipe for 2 sparks. Image shows aggregate decode.

Prefill is a nice 2000+ tok/s which makes it feel snappy for Hermes.

I extended to 12 concurrent which gives 460tok/s agg on code & 280tok/s on prose.

[Vikesh Chevli @vikeshchevli]

@Reederey Great job. The cache misses and high prefill latency were the most frustrating thing about existing GLM-5.3-flash configs. Will give this a go tomorrow.

[Vikesh Chevli @vikeshchevli]

One of these LLM models resides in hardware in my home.

Hard to believe how far local LLMs have come and how much intelligence you can squeeze out from a couple hundred billion parameters.

GLM 5.3 Flash is an incredible release.

[Vikesh Chevli @vikeshchevli]

@MiaAI_lab You are some sort of miracle worker wtf

[Vikesh Chevli @vikeshchevli]

@louszbd Well done great release!

[Vikesh Chevli @vikeshchevli]

@seanhighness @tonysimons_ I've had it set up for quite a while to actively orchestrate kanban jobs to different profiles. Was hoping I could move to bot mode, it looked great for what little I could use it, but need the app to be much more stable than it is now. Default Hermes is very capable though.

[Vikesh Chevli @vikeshchevli]

@seanhighness @tonysimons_ The desktop app has been quite unstable for me since the "Bot" release, pretty unusable for me since. I don't use the desktop app much though so doesn't overly bother me.

[Vikesh Chevli @vikeshchevli]

@0xBakeer I do not see any reason why it would. Hardware is effectively identical?

[Vikesh Chevli @vikeshchevli]

@MiaAI_lab Only just got my 2nd and this is working great for me. Legend!

[Vikesh Chevli @vikeshchevli]

@ashxhart Excellent. I have #1 and #3.

[Vikesh Chevli @vikeshchevli]

@JonathanLeaders @MiaAI_lab @bozhanghaha @NVIDIAAI @sgl_project I have got it working but struggling to fit context window / KV pool much greater than 65k in a 5090. Smoke test had decode at 250tok/s though...

[Vikesh Chevli @vikeshchevli]

@ddalcu With the 3.6 models I found tool calling to be a bit faulty on MLX inference engines. Froggeric's chat template fixed it for me. Perhaps the same for 3.8? Not tried the model on Apple silicon yet so can't confirm.

[Vikesh Chevli @vikeshchevli]

Qwen 3.8 27B being actively compared to Claude Opus 4.6 Max on benchmarks by Alibaba! Unlimited use Opus level AI on your desktop PC! It's just truly astonishing!

[Qwen @Alibaba_Qwen]

Performance of Qwen3.8-27B:

[Vikesh Chevli @vikeshchevli]

If you're playing with LLMs on your own hardware! https://t.co/cM8PDVTMmr

[Vikesh Chevli @vikeshchevli]

@sfxnz Is Supergrok sub through it back to normal limits as it was prior to Grok 4.5? Or still stupidly low?

[Vikesh Chevli @vikeshchevli]

Great news for the local LLM community! Another viable RTX 3090/4090/5090 model. Eager to give this a go at some stage, 233 tok/s on an RTX 5090 with speculative drafting is really impressive. Will be surpassed by the Qwen 3.8 27B no doubt though.

[Alexandr Wang @alexandr_wang]

1/ big announcement today: we will be releasing an open weight version of muse spark 1.2 soon.

we also are releasing muse glimmer, a 30B agentic model with open weights under apache 2.0. muse glimmer can run on 24GB of VRAM without losing agentic reliability. 🧵

[Vikesh Chevli @vikeshchevli]

@composio I found it the nice balance between the minimalism of Pi and some of the wider capabilities of other harnesses. Local LLMs (eg. Qwen 3.6 27B/35B & Gemma 4 12B/31B) felt more performant with it too when compared to running smaller models in the other 3.

[Vikesh Chevli @vikeshchevli]

@sudoingX @ASvanevik @0xSero @TheAhmadOsman @MikeBradleyAI @MiaAI_lab @MichaelGannotti @mr_r0b0t @basecampbernie @KyleHessling1 @NeoAIForecast @ivanfioravanti @elliotarledge A genuinely great list, was already following all but 1 of them!

[Vikesh Chevli @vikeshchevli]

@0xSero Thanks for the excellent single DGX Spark config, it is rather quick! My Hermes spotted something. The image shipped xgrammar 0.1.27, below pinned vLLM's >=0.2.1 requirement, causing some tool-call 500s. Adding xgrammar==0.2.1 fixes it.