Bờm

Dumb guy with a small areca spathe fan.

[Bờm @quatmo]

@PeasantSmith @Tech2Wild Ask hermes directly pls.

[浮世绘 @shmily7]

Ox Alpha啊，你裤子掉了。

[Bờm @quatmo]

Actually gemini-3.7-flash is nice:
⚡ Summary Metrics
- Visible Stream Throughput: ~408.7 tokens/sec (Peak: 548.1 tokens/sec)
- Internal Thinking / Reasoning Speed: ~176.9 tokens/sec
- Effective End-to-End Throughput: ~249.9 tokens/sec (Reasoning + Output tokens combined)

[SGLang @sgl_project]

Just pushed DFlash2 (@inco_ai) recipes to the Qwen3.8 27B cookbook⚡️

https://t.co/eAe1dFO5Sr

The community has been seeing great results with NVFP4 + DFlash2, and these recipes should be some very good starting points to play with.

More Qwen3.8 27B updates on the way 🫡

[Bờm @quatmo]

@seanphan @jun_song 4600 also, crazy

[Zhijian Liu @zhijianliu_]

DFlash 2 is here! Qwen3.8-27B at 70 tok/s on an M5 Max MacBook Pro.

⚡ Up to 4.6× the speed of autoregressive decoding, with the same output.

This is the next generation of DFlash, seeded at Z Lab and upgraded at Inco AI. Get one more accepted token on every pass, for free!

https://t.co/We0lwYPSBl

[Bờm @quatmo]

Tired of editing config files every time you swap a local model?
The broker is a dependency-free, localhost OpenAI-compatible proxy that exposes one stable model ID (local-auto) while routing requests to your prioritized local model servers. https://t.co/rup2GRKI3q

[Bờm @quatmo]

@MiaAI_lab https://t.co/rup2GRKI3q small tool

[0xBakeer @0xBakeer]

Qwen3.8-27B, one DGX Spark: 7.88 -> 75 tok/s solo, 256 aggregate.

Spent the weekend trying to make Qwen3.8-27B faster on a single DGX Spark and ended up proving myself wrong about almost everything I assumed going in.

Started at 7.88 tok/s. Stock FP8, nothing tuned. My assumption was the same one everybody has: if you want speed, quantize. 4-bit, fewer bytes per token, done.

Wrong. Or at least, it was the smallest part of the answer.

Speculative decoding got me to 58.5 tok/s on the exact same FP8 weights. That's 7.4x with nothing touched, and it cannot change the output at all, because the full model verifies every drafted token and throws away anything it wouldn't have written itself. Free speed. I had been ignoring it.

Then a few things I did not expect.

The drafter architecture mattered more than the draft depth. DSpark is a separate 1B drafter that emits a block of 7 tokens at once. MTP re-runs one in-checkpoint layer 7 times and pays a full vocab projection every time. DSpark accepts FEWER tokens per pass and is still 46% faster. Cost per draft token: 0.046 vs 0.153. Cheap passes beat good guesses.

Acceptance rate is a trap. Going from k=7 to k=14 dropped acceptance from 98.7% to 68.7% and made generation 27% faster. The late draft positions are basically lottery tickets, the early ones are certain. Mean tokens per forward pass is the number that predicts throughput. Not acceptance.

Prefix caching had been silently off the whole time. vLLM disables it by default for hybrid attention models and Qwen3.8-27B reports is_hybrid=True. Nothing in the startup log mentions it. Claude helped me and only found it reading engine source. Turning it on: 14x faster prefill on a 19K shared prefix, 22x on a 53K one.

And then the result I actually care about.

4-bit is 27% faster than FP8 at one request. At 4 requests, 20%. At 8, 10%. At 16 concurrent requests the two are within 0.2% of each other. Basically identical.

It makes sense once you see it. Single-stream decode is memory bandwidth bound, so halving the bytes nearly halves the work. Batch it, and one weight read serves many sequences at once. The bottleneck moves to compute and byte count stops mattering.

So if you're serving agents in parallel, quantization buys you close to nothing. It also costs you 7-9% of prefill at long context, because 4-bit weights get unpacked before every matmul on hardware with no native FP4 path.

Final numbers: 7.88 -> 75 tok/s single stream, 256 tok/s aggregate across 16 streams.

Run your own task before you trust any 4-bit build.

Both serving recipes are public:
FP8: https://t.co/wxEcszQsLY
NVFP4: https://t.co/cldZ1VVZun

[Bờm @quatmo]

@ivanfioravanti @MiaAI_lab I have the thermal throttling issue with Mia's recipe on the new Deepseek 0731, since the new max thinking always push the temp to 86 degC in TTFT, with 16 concurent. Apply clock cap might increase TTFT abit but not much on tok/s and no thermal throttling. I use 2300 MHz.

[Bờm @quatmo]

@0xSero I mostly do the same, fable/opus make plan, orchestrate, gpt sol critic and 0731 works. Opus seems too be not so good these days working alone.

[Bờm @quatmo]

@MiaAI_lab On 0731: the reasoning_effort ladder in deepseek_v4.py has no low branch — anything not max/xhigh/none silently becomes "high". So DEFAULT_THINKING=low actually runs high. Also drop_thinking=True: if reasoning eats max_tokens you get empty content + finish_reason=length.

[Bờm @quatmo]

Claim your https://t.co/0l0BQ1nB6E name while its still available https://t.co/Ht6AESyg56

[Bờm @quatmo]

@MiaAI_lab Appreciated!

[Unsloth AI @UnslothAI]

Introducing Unsloth Desktop 🦥
The first desktop app to run and train models locally.

• Open-source. Runs on Mac, Windows and Linux
• Supports MLX, diffusion image/video, audio, GGUF
• Connect Claude Code and Codex to local LLMs
• 50% more accurate, self-healing tool calls + sandboxed code exec
• Works for CPU + multiGPU setups - NVIDIA, AMD, Intel, Mac
• Train models 2× faster with 70% less VRAM
• Private web search, deep research, RAG, MCP and exports (NVFP4, GGUF)
• Use Unsloth’s OpenAI-compatible API and cloud models
• Securely deploy LLMs remotely and access anywhere

Unsloth Desktop is now available on https://t.co/xxBDOI9ApK and GitHub.

GitHub: https://t.co/2kXqhhvLsb
Blog and Guide: https://t.co/CYosNAQHva

[0xMarioNawfal @RoundtableSpace]

THIS CHROME EXTENSION EXTRACTS ANY WEBSITE’S DESIGN SYSTEM AND TURNS IT INTO AN AI-READY PROMPT FOR CLAUDE, GROK, OR ANY CODING AGENT.

[Bờm @quatmo]

@0xSero Wow wow

[Bờm @quatmo]

Thermal throttling at Max reasoning with Deepseek v4 flash 0731