Toshiaki Maki

Senior Principal Architect at @VMwareTanzu / @Broadcom | Ex @Pivotal | Bichon Frise 🍋🐩

[Toshiaki Maki @making]

@Bizuayeu inviteしました！

[Toshiaki Maki @making]

自宅TensorFoldを1.1.4ベースにアップデートした。
Mia-AiLab/GLM-5.3-Flash-EXL3-4bpw-TensorFold-Ablitを使った時のパッチ有無の比較(TP=3)。
1Mコンテキストを使う想定の設定だと、会話ごとにコンテキスト分の領域を確保しちゃうので、パッチないと厳しい。

[Ash Hart @ashxhart]

TensorFold 1.1.4 is live.

Up to 5× faster Qwen3.8-27B cold prefill on GB10, native Gemma 4 on CUDA, faster Flash Next multi-stream serving, and a long list of fixes from the community.

What's new and fixed:

• Qwen3.8-27B on CUDA has a new default prompt/decode recipe. On GB10, cold prefill improves by 54% at 2k, 107% at 8k, 258% at 32k and 416% at 64k. The 64k result goes from 203 to 1,045 tok/s; decode gains about 2% across 1 to 8 streams. Optional int8 prompt precision is also available, with bf16 remaining the default.

• Gemma 4 12B and 31B now run on native CUDA from MLX affine checkpoints, with their drafters. On GB10, the 12B's MTP assistant takes single-stream greedy chat from 30 to 67 tok/s, and z-lab's DFlash drafter takes the 31B from 12.6 to 29.9 tok/s. Image paths are implemented, but their GB10 image test cells have not run yet.

• Flash Next on CUDA verifies multiple streams' drafted windows in one pass. The GB10 served sweep gains 35%, 75% and 116% at 2, 4 and 8 streams, with tokens unchanged. Greedy MTP heads batch together; sampled heads, trees and media retain their per-stream path.

• More kernel work: fused Flash Next DeltaNet prefill, a reused PLE worker pool, Qwen attention reads that reuse KV data in L2, segmented split-K decode, and fewer Nemotron RADIO launches. GLM-5.3's Metal sparse-index scan drops from 1.59 ms to 0.14 ms in the measured cell. Startup timing tables also stop one noisy measurement from setting later performance.

• Drama 3 CUDA prompt chunking is opt-in. With 128-row chunks on GB10, first audio drops from 18.2 s to 0.76 s and the full utterance from 47.5 s to 30.2 s. The default remains unchanged; larger chunks can change rounding.

• Opt-in speech controls now include silence-tail, onset and mid-utterance guards; whole-reply redraws with budgets and deadlines; short-reply and register guards; a controls allowlist; synthesis frame limits; and attested voice catalogs. The voice tool can build those catalogs from reference audio and transcripts. Drama CUDA replay also waits for buffer fills before reuse.

• Nemotron-H MLX 4-bit now runs on the opt-in native HIP backend, including MTP, prompt caching and tensor parallelism. Unsupported flags are refused before loading, and Qwen HIP can cancel during long prompt chunks.

• Idle unload can free a model's memory after a configured idle period and reload it on the next request. A failed reload returns a retryable error; ranked GLM CUDA engines refuse this option.

• Sticky CUDA faults now make /health return 503. Driver-refused CUDA graphs can run eagerly instead of spinning, and queued writes retain their destination buffers through allocation failures.

• GLM's draw ring no longer fills after 64 completed or cancelled cold requests. Long chats retain less tokenizer/template scratch, kept-alive connections retain less memory, and a missing disk state no longer evicts cached states before its read fails.

• Memory and cancellation fixes cover Drama Metal loads that exceed the budget, Nemotron Metal Mamba-pool sizing and released slots after refused work, Qwen draft-node counts above 15, and prompt passes that now stop between chunks on cancellation or shutdown. GLM on one Mac also keeps its draft-acceptance estimates steadier after missed drafts.

• Flash Next CUDA gains POST /v1/decisions on one GPU and loads valid nested checkpoint shards. Qwen CUDA vision reads Hugging Face-named towers. Managed benchmarks can select bf16 or fp8 KV cache and record the choice. Windows builds with nvcc and cl.exe work again.

• Slide learning counts only kept lessons. App fixes cover engine-refused checkpoints, TUN/fake-IP proxies, Google's sign-in callback and visible setup errors. The docs now have a native GLM-5.3-Flash CUDA page.

• Experimental DeepSeek work adds native prompt rendering, HTTP 400 responses for template refusals and four-rank prefill planning. It remains behind the whole-model gate; release binaries do not serve DeepSeek checkpoints yet.

Release checks cover cold prompts from 2k to 64k, 1/2/4/8-stream sweeps and reference checks, with Linux and macOS host tests passing. The Qwen recipe changes prompt arithmetic, so output bits can differ from 1.1.3; its quality check matches the previous recipe at 27/40 exact cases, and concurrent/resumed checks still hold. One measured regression is documented: Nemotron Lightning greedy chat at four streams on GB10 is 3.8% slower.

All three release archives are signed and verified against the site's trust root. The nine published assets were downloaded back and checked against the signed files. Homebrew is updated and upgrade-tested, and a fresh install through the website script verified the archive signature.

A huge thank you to the release contributors:

@bobclawblaw @plotarmordev @rielstamand @MiaAI_lab @Bizuayeu @jishvenugopal @petruspennanen @wesley4546

And GitHub contributors jschmied, Mirrdhyn, wojo, heitke, BlivionIaG, harrisonfriia and paragontasx.

Thank you for the reports and requests, GitHub users Chris9121, flashosophy, Gogo6969, Idoo689, johncarterlives, mikog8tor, samwang0041-star and tonghuaxuan2016-dot.

Attribution also matters. The Gemma family builds on earlier work by gilby on GitHub and @codengod. Credit to Google for the Gemma assistant, z-lab for DFlash and the 31B drafter, MLX/mlx-lm and Hugging Face Transformers for model references, and Fish Audio, SGLang-Omni and Descript for the Drama model/codec references. Full source and licence credits are preserved in THIRD_PARTY_NOTICES.md and docs/credits.md.

And thank you to everyone testing, sharing measurements, contributing fixes and giving me access to hardware. This project would not be moving this quickly without you.

@MiaAI_lab @plotarmordev @petruspennanen @volatilemarkts

https://t.co/H2dBxCSaUV

[Toshiaki Maki @making]

@Bizuayeu 外に出せるように色々整備してからpushします

[Toshiaki Maki @making]

@Bizuayeu githubのprivate repoに移してinviteしますね。僕名義では出せないので、代理で送ってくれても良いですw

[Toshiaki Maki @making]

ローカルLLMやればやるほどモデルや推論エンジンに手を出したくなり、Claude Codeが離せなくなる矛盾(?)

[Toshiaki Maki @making]

@Bizuayeu full reportをDMで共有しますね。ローカル環境情報が入っているので...

[Toshiaki Maki @making]

@Bizuayeu はい重みの違いですね

[Toshiaki Maki @making]

markdownでは表現できない方法の一つとして、テンプレートの定義。同じ形式のスライドをテンプレートとしてコードで定義して使いまわせる。 deck as code

[Toshiaki Maki @making]

lispresen大幅に機能追加した。
マークダウンでプレゼン資料作ると発表者側は楽だけど、聴衆側は読みづらい、という課題を抱えている方に試してほしい。
読みやすいフォントサイズ、レイアウトになるように自動調整される。
lispresen自体はバイナリで配布。skillも配布。
https://t.co/mgaW1QlnSY

[Toshiaki Maki @making]

rontolispの紹介スライドを作ろうと思っていたのに、rontolispでスライドを作成するツールを作ってしまった。S式でプレゼンテーションが書ける。Deck as Code。
https://t.co/mgaW1QlnSY

[Toshiaki Maki @making]

claude codeに記事のまとめを作ってもらうとかにも有用。artifactで解説させると長ったらしい文章を書いてわかりづらいので、lispresenで長さを矯正させる。

拡張子はlispだけど、ただのマークアップなので、lisp知らなくても大丈夫。

[Toshiaki Maki @making]

TensorFold 1.1.3+独自pacthのTP=3でMia-AiLab/GLM-5.3-Flash-EXL3-4bpw-TensorFold-Ablitと比較してみました。
decode/prefillはEXL3より遅いけど、精度は明確に良いという結果。
一旦自分はEXL3に戻したけれど、実タスクで精度に不満が出たらXSFP4を再検証する。

[ビーズちゃん @Bizuayeu]

リンク先の、私が量子化した重みのダウンロード数が100を超えました。嬉しい！

step-by-step explanation を用意し、量子化手法を詳細に説明しているので、そのへんが気になる方も是非！

もちろん、商用で使いやすい Local LLM を探している方には相変わらずオススメです！
https://t.co/nvEmXA5Alp

[Toshiaki Maki @making]

TensorFold 1.1.3+独自パッチで納得できる性能までちゅんーニングできたので、upstream版で運用する。

[Toshiaki Maki @making]

tensorfold 1.1.2 + 独自パッチで本番運用できそうなくらいまでパッチ当てまった。最終テスト中

[Toshiaki Maki @making]

こんなコードも**スライド上で**実行できる

[Toshiaki Maki @making]

https://t.co/dqKStmbyKq
rontolisp(Javaで実装したCommon Lisp処理系、SchemeもClojureも読み込める)の入門スライドを作った。スライド上でプログラムを実行できる。

[Toshiaki Maki @making]

コウモリダコの展示!!?

[N_Gooo06 @Tokusathu_Suki]

真っ暗で何も見えなかったけど、奇跡に会えた
コウモリダコ、次はちゃんと見るぞ

#沖縄美ら海水族館
#コウモリダコ
#深海
#深海魚

[Toshiaki Maki @making]

rontolispの紹介スライドを作ろうと思っていたのに、rontolispでスライドを作成するツールを作ってしまった。S式でプレゼンテーションが書ける。Deck as Code。
https://t.co/mgaW1QlnSY

[Mia @MiaAI_lab]

MiniMax-M3.1-Flash TensorFold recipe for 2x DGX Sparks dropping soon!! EXL3-EAQ quant is ready.

https://t.co/3TAzNrMrNQ

[Ash Hart @ashxhart]

MiniMax M3.1 Flash with zero day support on TensorFold.

@MiaAI_lab has you covered for a spark recipe 😁