AaronAsh @[email protected]

[antirez @antirez]

DwarfStar branchk "ds4f-mxfp4" now can run the lossless MXFP4 DeepSeek v4 Flash GGUF I published on my Hugging Face account. It rocks even with SSD streaming in 128GB systems at > 20 t/s in case you want to try the *actual* DS4F weights released without any quantization.

[Anemll @anemll]

Who is excited and ready for @Kimi_Moonshot K3 weights to drop? 😎

[antirez @antirez]

Do you remember my friend's 8xL40S server? I implemented microbatches for prefill / generation in DwarfStar and now this server with DS4 Flash 4bit is basically a *fast* multi user server with 120 t/s generation (aggregated) and 2000 t/s prefill. Didn't expect to reach this point.

[antirez @antirez]

GLM 5.2, Q2_K routed experts (effectively ~2.6 bits) running with SSD streaming on an M5 Max 128GB computer.

[Mario Zechner @badlogicgames]

recommended reading. deepmind's new AI control roadmap. looks like they've given up on solving the lethal trifecta directly. the new direction seems to be a tower of LLMs. i suppose that's as good as it gets.

https://t.co/RiXTjnCJuz

[antirez @antirez]

DwarfStar now supports SSD streaming in the DGX Spark and Strix Halo, not just in Metal. You can run the Q4 quants at decent speed, and even DeepSeek v4 PRO at low speed, or you can run Q2 Flash if you have less than 128GB.

[Anemll @anemll]

Tested @ivanfioravanti’s Retro Space Invaders prompt using the native MXFP4 MoE version of DeepSeek V4 Flash in streaming mode.

This is my fork of @antirez’s repo with Apple Neural Engine support and streaming optimizations.
Hardware: M5 Max 128GB with a fast internal SSD.

Here is the result:

[Unsloth AI @UnslothAI]

DiffusionGemma can now run at 2000+ tokens/sec! ⚡

We made local DiffusionGemma inference 1.8× faster.

Run it on 18GB RAM via Unsloth Studio.

GitHub: https://t.co/aZWYAtakBP
Guide: https://t.co/wYLfJWE6kG

[Dan Woods @danveloper]

https://t.co/t9YRmiOfKv

[antirez @antirez]

Good news. Now the DwarfStar story of local inference shifted from "you need 128GB" to "you need a decent MacBook" if you can afford going slower. Indeed the usability level at this speed is different, but it is *very* good that now this is a spectrum and not a hard can/can't.

[antirez @antirez]

1. It is slow, obviously, but not unusably slow if for some reason I'm cut out of frontier AI.

2. The goal here was to make this working slow but optimize it enough to have Flash good enough on 64GB systems. I'll ask somebody to check what performances it reaches. I don't have a 64GB machine.

[Anemll @anemll]

Here is alpha release of SSD streaming for @antirez DSv4 Flash
https://t.co/1BSZKsc1GP

Model in "dense+Sidecar" format
https://t.co/MEqHhmRf1Z

[AaronAsh @[email protected] @AaronAsh2]

@anemll @antirez @liuliu Do you have this up on a public repo somewhere? I'd love to test it on my m3 max 64gb

[antirez @antirez]

Should DwarfStar support Qwen 3.6 27b dense q8 model in ds4-server, agent, cli and so forth to bring the same tooling to 64GB MacBook owners?

[AaronAsh @[email protected] @AaronAsh2]

@mitsuhiko the ux of pi-ds4 managing the inference engine is pretty damn nice. 64gb isn't enough ram for ds4 so I made a plugin for llama.cpp and qwen as a quick experiment: https://t.co/NWBnEr2yna

[antirez @antirez]

Distributing LLM inference in DwarfStar: https://t.co/Pgb548OZ3w

[antirez @antirez]

DeepSeek V4 PRO 2 bit quants, on a baroque music whose copyright expired, with the noise of the vynil record in the background, implements a small C compiler while running on a Mac Studio M3 ultra with 512gb of RAM.

https://t.co/tbvJ6Q7Ygy

[antirez @antirez]

To make this more real world practical, I just pushed a --power option that allows to balance speed and heat / fan noise / battery usage. For instance you can use --power 50 and the fan not even spins, for half the inference speed. It does micro-sleeps taking a running average of layers execution (prefill) or token generation (decoding).

[antirez @antirez]

For the DGX Spark owners. This is what you get with DS4 in your hardware. I want to post this to show how with fast prefill and not very fast generation, the system remains absolutely fine to use.

[Matt Pocock @mattpocockuk]

I don't know what the fuss is about. Anthropic's rules on using subscriptions are very simple:

Claude Code = OK
Claude's online platform = OK
Agent SDK running in personal software = OK... ish?
Agent SDK running in commercial software = NOT OK
Claude Code running in CI = ??

Oh, maybe it's not so simple...

Agent SDK running in CI = ??
claude -p running in CI = ??
claude -p running in personal software = OK
claude -p running on open source software, but run on my personal computer = ??
claude -p running on distributed sandboxes, kicked off by me = ??
Distributing open source software which relies on claude -p, and documenting how to use your subscription with it = ??
A thousand other edge cases = ??

Let me be clear. I have never before experienced, from any developer tool, such a frustrating lack of clarity over the basic terms of usage.

I personally asked, 3 weeks ago, and have received nothing but delays. The recent @bcherny announcement did absolutely nothing to clarify things.

I say this as someone who just released a Claude Code course - my incentives all align with supporting Anthropic.