LakyFx

CEO / CTO - LLM & AI Agent Engineer / AI Agent Architect / Local LLM inference

[LakyFx @laky_fx]

The controller learns verification costs and draft acceptance rates, not specific prompts or answers. An unfamiliar task can benefit immediately if its runtime behaviour resembles previous work. When that behaviour changes, the estimates adjust using recent observations, with confidence priors and fallback guards when calibration isn’t helping. So switching tasks doesn’t reset the learner, but gains can vary with context length, concurrency and how predictable the output is. It also isn’t guaranteed to keep getting faster the longer it runs.

[LakyFx @laky_fx]

🚀 My TensorFold Recipe v1.2.6 — Smart Adaptive Speculation & Advanced KV Pool Optimization

The first results were from the initial run. I've now tested it again with the speculation controller trained on historical workload data, and the improvement is pretty impressive! 🔥

🧠 1. Smart Adaptive Speculation (K1–K5)

The controller doesn't learn to predict tokens themselves. Instead, it learns how many tokens to speculate based on historical acceptance rates.

It remembers which tools, operations, and workload patterns previously resulted in successful or unsuccessful speculative decoding.

Based on that history, it dynamically selects a speculation budget between K1 and K5.

The key is avoiding unnecessary speculative work. Higher speculation depths are only selected when the controller has evidence that they've worked well in similar situations.

This reduces verification overhead and wasted computation while maintaining high acceptance rates.

Current results after the controller has learned:

~84% speculative token acceptance

Over 90% acceptance during repetitive agent workflows, such as long-running coding, reviews, and recurring tool calls

⚡ 2. KV Pool Memory Management & Fragmentation

I've also significantly improved how the KV pool manages memory.

One problem I discovered was memory fragmentation. Cached contexts need efficient placement and allocation to maximize reuse and avoid unnecessary prefill operations.

Think of it a bit like defragmenting old FAT16/FAT32 hard drives. The principle is similar, although here we're dealing with RAM and KV cache allocation.

Traditional defragmentation introduces overhead, so I've disabled continuous defragmentation.

Instead, I developed a mechanism that efficiently relocates existing cached contexts immediately before prefill, creating contiguous space for incoming contexts.

For example, relocating a 300K-token cached context within RAM costs roughly 100 ms in my testing. It's simply moving data between RAM locations, with relatively little overhead compared to the potential cost of recomputing that context.

I've also improved prefix ID tracking and storage. There were some issues with how newly loaded prefixes were registered and reused, which I've addressed.

🎯 3. KV Cache Hit Rate: ~96% → 98.5–99.2%

This is probably one of the most interesting improvements for my actual Hermes workload.

Previously, I was getting around 96% cache hits, which was already good.

Now I'm seeing approximately 98.5–99.2% in early testing! 🔥

When agents repeatedly send context back to the model, nearly everything can be reused directly from RAM. Only genuinely new content, such as new instructions or reasoning, needs additional prefill processing.

And that's important because every unnecessary prefill introduces latency.

The goal isn't simply maximizing raw decode tokens per second. It's minimizing total overhead and making long-running agent workflows as efficient and responsive as possible.

I'm going to let this version run for several more hours under my real Hermes workload and analyze the detailed logs before drawing final conclusions.

But so far, v1.2.6 looks extremely promising! 😎🚀

git: https://t.co/sSbvOQhsev

https://t.co/C6Btp15fQI

#TensorFold #DeepSeek #DGXSpark #EXL3 #LocalLLM #AIInference

@ashxhart

[LakyFx @laky_fx]

@HandwerkAiLab With v1.2.5, you need to run benchmarks multiple times because there's a learning phase where the controller learns how to select the optimal speculation budget. It won't perform optimally right out of the box.

I repeated my benchmarks after 24 hours of real-world usage and got 30% better results!

So keep that in mind. Have your agent explain how the adaptive speculation learning works, but the key is that it needs time to learn the optimal speculation strategy for your specific workload.

A single benchmark run won't give you the full picture. :)

[LakyFx @laky_fx]

@HandwerkAiLab

[LakyFx @laky_fx]

Yeah, exactly! That's why I published multiple variants. I've just released the improved v1.2.6 and posted about it. :)

1.2.2TF has its own issues too. It's not all about decode speed.

As I mentioned before, I optimize for real-world workloads. Getting 5% more decode performance in a benchmark doesn't necessarily mean much. Prefill performance, KV cache efficiency, memory management, and overall overhead all matter. Those are exactly the areas I've been improving in the latest version.

Decode is only one part of total workflow performance. What really matters is how quickly the model completes an entire task, including prefill, tool calls, reasoning, and everything in between.

That's why I've made further improvements focused on end-to-end task completion time, rather than just raw tok/s.

But ultimately, it's up to you! Pick whichever version works best for your setup and workload. That's exactly why I published multiple variants in the first place. 😄🚀

[LakyFx @laky_fx]

@majewskizby My version of TensorFold :) for TP3 :) 👌👀

[LakyFx @laky_fx]

Your speculative decoding win on an H100 does not follow you to a MacBook.

A NeurIPS 2026 workshop paper (On-Device Intelligence + SLMs for Agentic Systems) tested four inference optimization families on four Apple Silicon machines, single-request decode, with matched RTX 4090 runs where possible. The results invert most of the folklore.

Speculative decoding: every method they evaluated reduced throughput on Apple, even though the same techniques report 2x to 6x gains on server GPUs. The draft-verify loop pays off under batching, not under a single token stream.

KV cache pruning: up to 1.75x end-to-end throughput on Apple, while on desktop NVIDIA the same trick can cost throughput. Same technique, opposite sign.

Ported CUDA kernels: transfer only partially. Public Metal has no async copy and batch-one decode makes several serving-oriented mechanisms inapplicable, so only the transferable parts deliver, up to 1.20x decode.

Quantization: gains vary machine by machine, there is no universal multiplier.

The conclusion is the useful part. A speedup is not a portable property of a technique, it is a joint function of technique, runtime, hardware contract and workload regime.

Which means a benchmark from a batched server tells you almost nothing about your laptop. The only number that counts is the one you measured on your own hardware.

Paper: https://t.co/nN5JHb3wOY

#LocalLLM #AppleSilicon #LLMInference #MachineLearning #AI

[LakyFx @laky_fx]

@filicroval I've brought ideas from NVIDIA FLARE and DeepInferer into my TensorFold recipe, combined them with my own optimizations, and I think the result is seriously fast! 😎🚀
My TP3 recipe is actually faster than a baseline TP4 setup! 🔥 Tuning inspiration: https://t.co/iOe5osxabU

[LakyFx @laky_fx]

If I can offer one piece of advice, don't focus too much on those benchmarks. They don't tell you much about real-world performance. What you really need is smooth, efficient operation with minimal overhead.

Look for sources of overhead, experiment with optimal prefill chunk sizes, and definitely explore speculative decoding. I've developed my own smart adaptive speculation, and it makes a huge difference.

I've documented everything I've tested in my recipe, including all the ideas and experiments. There are probably 14–15 separate tests in there, maybe even more.

The main value is in the ideas, because your hardware setup will behave very differently from a single Spark.

It's all about experimenting, measuring, and finding what actually works best for your setup. 😎

[LakyFx @laky_fx]

@DavidOndrej1 We need them to release Kimi K3.1 Flash! 😄 Something that fits into 512 GB, and we'd be all set! 😎🚀
Then I'll tune your Linux kernels for maximum performance, just like I did with my DeepSeek V4.1 Flash setup! 🔥

[LakyFx @laky_fx]

This is insane! 🤯 The impossible just became reality! 🔥
Looks like I'll have to buy a DGX Station so I can optimize it the same way I've been tuning my DGX Spark recipes. Ultra-fast performance, here we come! 😄🚀
Hopefully @DavidOndrej1 won't mind if I throw my own recipe into the mix. It's completely free for everyone, and I've poured hundreds of hours into researching, experimenting, and fine-tuning kernels and speculative decoding. :) Shall we give Kimi K3.1 the same treatment? 😎🚀 https://t.co/sSbvOQhsev

[LakyFx @laky_fx]

Real-world performance isn't just about decode or prefill speed.

I only use RidgeMark so I can compare my results with other people's recipes. For actual optimization, I have my own heavy-workload benchmark.

Honestly, I think the way most people benchmark today, including RidgeMark, is pretty far from real-world usage.

My benchmark is built from 24 hours of actual agent workloads. I've extracted the most demanding tasks from those runs, and that's what I optimize against.

It's about much more than raw speed: speculative acceptance rates, kernel efficiency, reducing overhead, and memory management.

Right now, for example, I'm digging deep into how the KV pool operates, optimizing RAM allocation, storage, and fragmentation to maximize KV cache efficiency.

I'm currently getting around 96.5% KV cache hit rate, but I believe I can push that to 98.8%.

That means most of the prompt can be reused almost instantly from RAM, with cache throughput exceeding 200,000 tokens/s.

And that's the interesting part: not all caching is created equal. There's still plenty of room to optimize how it works. 😎

For my real-world agent workloads, that's where some of the biggest gains are hiding. 🚀

[LakyFx @laky_fx]

@christopherowen @mxfp0 @g0blinResearch 👀Check out my SparkDash results with TensorFold 1.2.5 DeepSeek 4.1 Flash on 3× DGX Spark TP3 💻 C1 code: 126.1 tok/s 📝 C1 prose: 87.7 tok/s 🚀 C12 code aggregate: 246.2 tok/s

TP4 has an advantage here: an extra Spark and a cleaner split of the model. TP3 is a little awkward because some dimensions don’t divide evenly across three devices. Still, I think these are pretty good results!
Maybe I’ll connect my fourth Spark and give TP4 a go. It’s currently busy running MiniMax H3, Qwen Image 2.1 and Flint 40B…🙂

Recipe: https://t.co/sSbvOQhsev
Weights: https://t.co/C6Btp15fQI

#DeepSeek #DGXSpark

[LakyFx @laky_fx]

@HandwerkAiLab @calvarado2004 One thing that might help you: I'd actually say Czech is probably even more complex than German. 😄
I don't write my prompts in English at all. I use Czech 100% of the time, and everything works absolutely perfectly for me. :)

[LakyFx @laky_fx]

18 open-weight models, 291 graded runs, two RTX 5060 Ti cards: none replaced a 27B dense model.

Siim Tolk built a loop that finds new open-weight models on Hugging Face each morning, downloads the ones fitting his 32 GB of VRAM, and lets a coding agent run his 97-case suite overnight. Every case runs 3 times, graded by code, not by another model. His production Qwen3.8-27B passed 283 of 291 runs. Nemotron 3.5 Bolt 74B-A9B, 4th under 100B on Artificial Analysis, passed 247, and only 16 of 45 Estonian cases.

The serving setup is the useful part: IQ4_XS at 15.5 GB instead of Q5_K_M at 19.7 GB, same score and 20% faster, all layers split across both cards, 262k context in two 128k slots, q4_0 KV cache with flash attention, and the built-in MTP draft head. MTP took the model from 25 to 41 tok/s on one card for 1.2 GB of extra memory. n-gram drafting on top was 15 to 20% slower, because MTP already guessed most of the repetitive JSON.

The honest caveats: decode falls from 45 tok/s on a short prompt to 26.5 tok/s after 90k tokens, and prefill of a fresh 90k prompt takes about 95 s. Each card needs ~2.8 GB of compute buffer, which is why Gemma 4 31B QAT fit on paper and ran out of memory in five configurations once its draft model was added. vLLM generated up to 1.4x faster but needed ~30 of 32 GB and 4.5 to 5.5 minutes to start.

Tokens per second is not time per answer. His previous MoE at 137 tok/s moved the median real request from 2.3 to 8.6 seconds when he switched to the denser, more accurate 27B. And one "uncensored" build followed an instruction planted inside a document every single time, which disqualifies it for an agent that reads the web.

Suite and results: https://t.co/zZxOaYRzlH
Source: https://t.co/1H35lpmIfU

#LocalLLM #llamaCPP #OpenSource #AI #Agents

[LakyFx @laky_fx]

Git: https://t.co/ljUwtitfvA
Weights: https://t.co/Il5z63G23m

[LakyFx @laky_fx]

Fastest TP3 recipe yet for DeepSeek 4.1 Flash 1017 🚀
🤖 763B parameters. Weights + recipe are out.
Check it out 👇 links in coments.

[LakyFx @laky_fx]

@HandwerkAiLab @calvarado2004 Dude, I have no idea! 😄 I don't speak a single word of German, so I don't really use it. :) You try it :)
I've actually managed to push speculative decoding acceptance above 80% on my real-world Hermes workload, and I've got even more improvements in the works! 🚀

[LakyFx @laky_fx]

@VweeterLimited Offline on a phone is the easy part, bandwidth is the wall. A 4B model at Q4 reads about 2.3GB per token pass and flagship LPDDR5X gives ~68GB/s, so decode tops out near 30 tok/s before overhead. Small weights fit fine, long context is where phones fall over.

[LakyFx @laky_fx]

A 3-bit quant of an open model on one RTX 5090 scored 91.0 on the biggest coding benchmark of the year. Claude Sonnet 5 in the cloud scored 91.0 too, at about $27 per run.

Fabio Akita closed benchmark v4 yesterday: 67 runs, roughly 60 models, past $6,000 spent. The yardstick is code review vigilance, not code writing. The model builds a Rails chat app in seven sprints while 14 security and correctness defects get injected. Fixed unprompted counts full, after a reveal 40%, never zero.

Consumer result: Qwen3.8 Flash-Next, IQ3_S at about 3.4 bits, RTX 5090 with the Strata engine. 91.0, 159.2 tok/s, $0 per run. Same model in NVFP4 on a B300: 92.5, 195 tok/s, $12 of inference. Going home cost 1.5 points and 18% of the speed.

What did not survive the move home: Ornith 1.5 35B went 81.0 on a B300 (BF16) to 64.5 on the 5090 (Q8_0), GLM 4.7 Flash 33.5 to 24.0. The bits were not the main problem, the serving stack was. Serving the same checkpoint yourself moved scores 8 points up (GLM 5.3 Flash) and 67 points down (Nex N2.5 Pro).

Caveats: 42 of 67 runs hit Tier A and seven models tied at 100, so the top is saturated. A repeated GPT 5.5 run scored 91 against its own 100, so a few points are noise. One run per configuration.

The takeaway for local inference: a 125B MoE with 6B active per token, 3-bit weights and hot experts pinned to VRAM is now enough for serious review work at 159 tok/s. Below a 5090 it collapses. Qwen 3.8 27B on Strix Halo finished at 7.7 tok/s.

Source: https://t.co/eVUQBl6foI (code and logs on GitHub)
#LocalLLM #LLM #AI #OpenSource #Inference

[LakyFx @laky_fx]

Recursive's Flint: 39.8B MoE with 2.5B active per token, Apache 2.0. Vendor-run 89.7% SWE-bench Pro v2, 74.2% SWE-bench Verified. The 3-bit GGUF is 18.5GB, so 24GB RAM runs it at ~131 tok/s on an M4 Pro. Coding-agent quality on a laptop.
#LocalLLM #OpenSource #AI