Alfonso Fortunato

DevSecOps engineer · Kubernetes (CKA/CKAD/KCSA)
Local AI on a DGX Spark: benchmarks, recipes, breakages
Open to work & collabs · 🇮🇹 in 🇨🇭

[Alfonso Fortunato @devopsfortunato]

Happy to contribute to this project from day zero. Keep pushing the boundaries further 🚀

[Ash Hart @ashxhart]

TensorFold Zig engine 1.1.5 and Mac app Beta 10 are out 🙌

Beta 10 brings community-v2 benchmarks, Google sign-in fixes, updates that work through fake-IP proxies and model compatibility checks before loading. It bundles engine 1.1.4.

The standalone Zig engine is now at 1.1.5, with a lot of work from the community.

- Qwen3.8 Flash Next on GB10 with NVFP4: up to 28% faster prefill.

- Qwen3.8 Flash Next on M3 Ultra with MLX 4-bit MTP: 16% to 23% faster decode in the release tests.

- Follow-ups about the same 35k-token document: first token in 2.8s instead of 25.7s on GB10 with Flash Next FP4-Local.

- LM-5.3-Flash NVFP4 at TP=2 on two GB10s: 22% more tokens/s with two streams.

There's also Dory support on CUDA, more checkpoint formats, Drama 3 improvements, tool-call fixes and Windows CUDA build fixes.

Huge thanks to everyone contributing to the engine.
@Bizuayeu @plotarmordev @jerry543 @MiaAI_lab @agispof @softpoo @GumbiiDigital @bobclawblaw @jvr0x @ViC305 @rielstamand

And on GitHub: jschmied, BlivionIaG, harrisonfriia, drowzeys, Sapphire-Rapids and Boscoeuk.
Thanks for the reports too: @devopsfortunato, laidick, flashosophy and ss-cong.

To everyone testing, sharing access to hardware and supporting TensorFold, thank you. If I've missed anyone, I'm sorry.

@MiaAI_lab @plotarmordev @volatilemarkts @petruspennanen

[Alfonso Fortunato @devopsfortunato]

@0xBakeer @nanbeige @TandemLLM you will be disappointed

[Alfonso Fortunato @devopsfortunato]

This one makes me laugh 😆

Anyway, there’s too much hate towards Apple. Let’s see what happens over the next few months.

[Sanju Choudhary @saaaanjjjuuu]

Samsung literally mocked Apple's first foldable in every possible way 😭

[Alfonso Fortunato @devopsfortunato]

@plotarmordev Looking forward to it. Yes, I'll add my ideas for sure. Open it tomorrow :D

[Alfonso Fortunato @devopsfortunato]

I think the future is hybrid: frontier models in the cloud for orchestration and advanced planning, and a swarm of local agents doing the real work. Local agents also handle your sensitive data and can be your red team.

That's what I'm building right now: a local architecture where I hand off as much work as I can to multiple agents, each with a target outcome and clear rules.

Today @bot a big part of my life: price tracking on specific products, my subscriptions, my posting on X and other platforms, my job application tracker, and more.

But I hate vendor lock-in. What if prices double tomorrow, or a feature gets limited? I don't want to be in that spot, and I don't want my whole life sitting in the cloud.

So I'm looking for an alternative. Thanks to @plotarmordev's suggestions and guidance, I'm leaning towards Mausbot: lead agents, then specialized agents per project, plus a decision matrix.

I'll build it from the ground up. Like in infrastructure, the heavy lift isn't the implementation, it's the planning and the architecture. This week goes into drafting the best architecture I can.

My hardware:
- 2x DGX Spark
- 4x Raspberry Pi 4 (4 GB RAM)
- Desktop PC on Omarchy, 32 GB RAM, RTX 4070 Ti
- Mac Pro (2020), 16 GB RAM

The catch: Mausbot runs on a single node (minimum 2 CPUs, and more agents need more RAM). Could I shard the work across the Pis? Not sure it's possible.

Another option: rent a 16 GB VPS (I don't want my desktop running 24/7) and keep the agents' memory on the Pis.

Still exploring. How are you running your agent setups? Always happy to learn from other builds.

[Alfonso Fortunato @devopsfortunato]

@KamakuraCrypto @bot Check this: https://t.co/JTSNSiWkRX im gonna test it soon

[Alfonso Fortunato @devopsfortunato]

Even Tailscale is under attack, and new vulnerabilities keep showing up.

This gist walks your agent through enforcing your network’s policy. It doesn’t create anything without your approval.

Running it right now

[📙 Alex Hillman @alexhillman]

Hey fellow Tailscale loving friends, with all the security shenanigans going on here’s a prompt I recommend for making sure the weakest link on your talilnet can’t wreck everything else over ssh or otherwise.

Give your agent this and let it walk you thru locking down your ACL:

https://t.co/xFy0M1Adtx

[netrunner @plotarmordev]

Drama 3 now runs natively on Apple Silicon Macs!

Drama Tensorfold now supports the full model and my 20GB quant on Metal, fully local. Sound on 🔊

https://t.co/uIZLMPTTHj

[Alfonso Fortunato @devopsfortunato]

@ivanfioravanti Monster for sure, but dense model…

[Alfonso Fortunato @devopsfortunato]

Apparently the model didn't break during ablation, so there's no single reason not to run it.

Having a model that can either defend you against external attacks or help you work on kernel stuff is crucial.

Gonna try it soon.

@OrcaRouter never disappointing

[netrunner @plotarmordev]

I think a 1 point MMLU* gain after abliteration means it can't see what abliteration breaks.

While yes it gets much better at refusals and I use abliterated models myself, lots of people running abliterated models say they get worse on SOME tasks.

*MMLU is a test of general knowledge across many subjects, so it's about the last place that damage would show up.

[Alfonso Fortunato @devopsfortunato]

Huge release.

Will try the new mtp improvement for flash next.

Amazing work @ashxhart

[Ash Hart @ashxhart]

TensorFold 1.1.4 is live.

Up to 5× faster Qwen3.8-27B cold prefill on GB10, native Gemma 4 on CUDA, faster Flash Next multi-stream serving, and a long list of fixes from the community.

What's new and fixed:

• Qwen3.8-27B on CUDA has a new default prompt/decode recipe. On GB10, cold prefill improves by 54% at 2k, 107% at 8k, 258% at 32k and 416% at 64k. The 64k result goes from 203 to 1,045 tok/s; decode gains about 2% across 1 to 8 streams. Optional int8 prompt precision is also available, with bf16 remaining the default.

• Gemma 4 12B and 31B now run on native CUDA from MLX affine checkpoints, with their drafters. On GB10, the 12B's MTP assistant takes single-stream greedy chat from 30 to 67 tok/s, and z-lab's DFlash drafter takes the 31B from 12.6 to 29.9 tok/s. Image paths are implemented, but their GB10 image test cells have not run yet.

• Flash Next on CUDA verifies multiple streams' drafted windows in one pass. The GB10 served sweep gains 35%, 75% and 116% at 2, 4 and 8 streams, with tokens unchanged. Greedy MTP heads batch together; sampled heads, trees and media retain their per-stream path.

• More kernel work: fused Flash Next DeltaNet prefill, a reused PLE worker pool, Qwen attention reads that reuse KV data in L2, segmented split-K decode, and fewer Nemotron RADIO launches. GLM-5.3's Metal sparse-index scan drops from 1.59 ms to 0.14 ms in the measured cell. Startup timing tables also stop one noisy measurement from setting later performance.

• Drama 3 CUDA prompt chunking is opt-in. With 128-row chunks on GB10, first audio drops from 18.2 s to 0.76 s and the full utterance from 47.5 s to 30.2 s. The default remains unchanged; larger chunks can change rounding.

• Opt-in speech controls now include silence-tail, onset and mid-utterance guards; whole-reply redraws with budgets and deadlines; short-reply and register guards; a controls allowlist; synthesis frame limits; and attested voice catalogs. The voice tool can build those catalogs from reference audio and transcripts. Drama CUDA replay also waits for buffer fills before reuse.

• Nemotron-H MLX 4-bit now runs on the opt-in native HIP backend, including MTP, prompt caching and tensor parallelism. Unsupported flags are refused before loading, and Qwen HIP can cancel during long prompt chunks.

• Idle unload can free a model's memory after a configured idle period and reload it on the next request. A failed reload returns a retryable error; ranked GLM CUDA engines refuse this option.

• Sticky CUDA faults now make /health return 503. Driver-refused CUDA graphs can run eagerly instead of spinning, and queued writes retain their destination buffers through allocation failures.

• GLM's draw ring no longer fills after 64 completed or cancelled cold requests. Long chats retain less tokenizer/template scratch, kept-alive connections retain less memory, and a missing disk state no longer evicts cached states before its read fails.

• Memory and cancellation fixes cover Drama Metal loads that exceed the budget, Nemotron Metal Mamba-pool sizing and released slots after refused work, Qwen draft-node counts above 15, and prompt passes that now stop between chunks on cancellation or shutdown. GLM on one Mac also keeps its draft-acceptance estimates steadier after missed drafts.

• Flash Next CUDA gains POST /v1/decisions on one GPU and loads valid nested checkpoint shards. Qwen CUDA vision reads Hugging Face-named towers. Managed benchmarks can select bf16 or fp8 KV cache and record the choice. Windows builds with nvcc and cl.exe work again.

• Slide learning counts only kept lessons. App fixes cover engine-refused checkpoints, TUN/fake-IP proxies, Google's sign-in callback and visible setup errors. The docs now have a native GLM-5.3-Flash CUDA page.

• Experimental DeepSeek work adds native prompt rendering, HTTP 400 responses for template refusals and four-rank prefill planning. It remains behind the whole-model gate; release binaries do not serve DeepSeek checkpoints yet.

Release checks cover cold prompts from 2k to 64k, 1/2/4/8-stream sweeps and reference checks, with Linux and macOS host tests passing. The Qwen recipe changes prompt arithmetic, so output bits can differ from 1.1.3; its quality check matches the previous recipe at 27/40 exact cases, and concurrent/resumed checks still hold. One measured regression is documented: Nemotron Lightning greedy chat at four streams on GB10 is 3.8% slower.

All three release archives are signed and verified against the site's trust root. The nine published assets were downloaded back and checked against the signed files. Homebrew is updated and upgrade-tested, and a fresh install through the website script verified the archive signature.

A huge thank you to the release contributors:

@bobclawblaw @plotarmordev @rielstamand @MiaAI_lab @Bizuayeu @jishvenugopal @petruspennanen @wesley4546

And GitHub contributors jschmied, Mirrdhyn, wojo, heitke, BlivionIaG, harrisonfriia and paragontasx.

Thank you for the reports and requests, GitHub users Chris9121, flashosophy, Gogo6969, Idoo689, johncarterlives, mikog8tor, samwang0041-star and tonghuaxuan2016-dot.

Attribution also matters. The Gemma family builds on earlier work by gilby on GitHub and @codengod. Credit to Google for the Gemma assistant, z-lab for DFlash and the 31B drafter, MLX/mlx-lm and Hugging Face Transformers for model references, and Fish Audio, SGLang-Omni and Descript for the Drama model/codec references. Full source and licence credits are preserved in THIRD_PARTY_NOTICES.md and docs/credits.md.

And thank you to everyone testing, sharing measurements, contributing fixes and giving me access to hardware. This project would not be moving this quickly without you.

@MiaAI_lab @plotarmordev @petruspennanen @volatilemarkts

https://t.co/H2dBxCSaUV

[Alfonso Fortunato @devopsfortunato]

@0xBakeer Prefill, decode, TTFT, prompt caching in different cases (images, documents, etc.), tool-bench, sixcategory evaluation and some coding tasks for my use cases. I’m also testing how it explains complex concepts, so I can gauge whether I like its way of talking or not

[Alfonso Fortunato @devopsfortunato]

@0xWhiteMage @AgentSparko Yes, really underrated. I love this community because of people like him, and you, @0xWhiteMage

[Alfonso Fortunato @devopsfortunato]

@AgentSparko is honest and actually engages with you, even if you’re a small account. That’s rare.

The dashboard looks great.

I’m going to try it.

[AgentSparko 💥 @AgentSparko]

🚨🚨🚨

🏎️ LIMCARS — Local Inference Monitor · Cost Analyzer · Recipe Server

Your DGX Spark, finally with a dashboard that tells the truth.

📊 Live dashboard, 2 times per second
• ⚡️ Generation & prefill speed, TTFT & end-to-end latency — with honest sample counts, never percentile theater
• 🧠 KV cache usage, prefix-cache hit rate, speculative-decoding acceptance
• 🖥️ Hardware sidebar: CPU/GPU temps, power, clocks, unified RAM, SSD, network — each with its own sparkline

💰 Real money, not vanity metrics
• 🤑 Token ledgers per session and per model
• 📈 Daily savings chart vs cloud API pricing — see what you didn't pay your cloud API provider
• ⚠️ "Lost potential" alerts — your idle GPU's hourly capacity, priced out in USD
• 🧮 All dollars computed live from editable pricing — never baked into storage

🍳 Recipe server
• 🚀 One-click launch of your LLM engines from saved recipes
• 🔒 Password-gated edits (rate-limited, scrypt-hashed)

🕵️ Hermes Traces
• 🔍 Peek into your Hrmes sessions for agents and subagents, see tool calls, reasoning, chat
🔌 Speaks your engine's dialect
• 🤖 Auto-detects vLLM, SGLang, llama.cpp, EXL3, TandemLLM & more — and reads the right metrics for each

🎨 Made to be lived in
• 🌗 Dark & light themes, 24 palettes for theme and also for charts colors, tinted UI that follows your color
• 📱 Works on your phone via LAN

🕊️ Privacy by construction
• 🏠 One single Python process
• 🧬 Had another dashboard? An AI-agent migration contract (AGENTS.md) brings your history over

🧪 Tested on a single NVIDIA DGX Spark — bring your rig, open an issue, help field-test the rest.

🚀🚀🚀

[Alfonso Fortunato @devopsfortunato]

@ivanfioravanti @AgentSparko Yes, always open to discussing new things and helping out. Just like you, @ivanfioravanti

[Alfonso Fortunato @devopsfortunato]

@AgentSparko Love it, gonna test today!

[Alfonso Fortunato @devopsfortunato]

@wikiwayne I think my GF will love it haha

[Alfonso Fortunato @devopsfortunato]

@MiaAI_lab Deserved.

[Alfonso Fortunato @devopsfortunato]

Today I’m definitely switching from Windows to Omarchy on my desktop PC

I’ve been running Windows exclusively on this machine for more than 20 years, especially for gaming (I was a semi-pro SC2 player back in the day). But now I’ve realized it’s holding me back from using a full-power AI agent setup.

I’ll miss Valorant (love it), but the amount of shit Windows puts on top of a machine is unbearable.

I’ve been using WSL, but the hassle of dealing with different mount points between desktop apps and WSL isn’t worth the time spent on it.

Any tips for Omarchy?

#omarchy #Linux