David Hendrickson

CEO & Founder | PhD | Startup Advisor | @Columbia | Author Generative Software Engineering | 🔔 Follow for AI & Vibe Coding Tips 👇

[David Hendrickson @TeksEdge]

Qwen3.5-4B running CPU-ONLY at up to ~11+ tps on a Ryzen 5 laptop. No GPU. Running models in RAM and CPU is something the Qwen4 team is already thinking about.

💡 I built this standalone .exe because businesses are already deploying local models to save money and ensure privacy. (thumb drive friendly - just drag drop and doubleclick)

My llama.cpp recipe 👉
🧠 Qwen3.5-4B Q4_K_M GGUF
⚙️ Ryzen 5 7540U — 6C/12T
🧵 --threads 9
🧵 --threads-batch 12
⚡ --prio 2
🔄 --poll 50
📦 --batch-size 2048
📦 --ubatch-size 512
🚀 --flash-attn on
🧠 KV cache: q4_0 / q4_0
🔧 --repack
💾 --mmap
👤 --parallel 1
🚫 --device none
🚫 --gpu-layers 0
🚫 KV/op GPU offload
🚫 MTP OFF

Interesting result 👉 MTP=3 was slower (~10 tps in benchmarks).

Plain decode + 9 threads + Q4 KV hit ~11.4 tok/s.

That's about 10% faster just from tuning llama.cpp — on a basic laptop CPU.

[David Hendrickson @TeksEdge]

Okay so imagine you're chatting with a local LLM that has allocated most of your GPU's VRAM. Then you launch a game.

Instead of killing the model, LlamaForge gives the VRAM back.

That's the idea behind a new experimental feature in LlamaForge called Elastic VRAM.

When another application starts grabbing GPU memory
🎮 Game needs more VRAM
⬇️ LlamaForge moves model layers back to CPU
🧠 MoE experts get moved first
💬 Your existing conversation survives
⬆️ When VRAM becomes available again, layers move back to the GPU
The model gets slower while it's shrunk to CPU.
But it doesn't have to disappear completely just because you launched something else.
Underneath, LlamaForge is running upstream llama.cpp and uses llama.cpp's slot save/restore to preserve the chat while it rearranges where the model lives.

There's also a simpler Game Mode
Launch game → unload model
Quit game → reload model

Elastic VRAM is the more interesting version because the model stays available while relinquishing GPU memory.

Keep in mind that it is ...
⚠️ Experimental
🟢 NVIDIA only today
⚙️ Off by default
🧪 Not widely tested yet
🔓 LlamaForge → MIT / local

Link in ALT

[David Hendrickson @TeksEdge]

Are you getting ready for Kimi K3.1?

It may already be testing in Arena under the anonymous name Kibitz.

And there are now multiple clues pointing back to Moonshot.

🧬 Community tokenizer/fingerprint tests reportedly behave like Kimi

🗣️ In some Arena tests, Kibitz identifies itself as Kimi/Moonshot

🔀 Kimi Code's configuration now contains a model-switch target named k3.1

And the anonymous naming pattern is eye catching
K2.5 → Kiwido
K2.6 → Kiwire
K3 → Kivine
K3.1? → Kibitz

K3 itself reportedly appeared anonymously in Arena roughly 30 hours before release.

None of this confirms Kibitz = Kimi K3.1.

Moonshot has not announced anything officially yet.

But K3.1 itself looks increasingly real.

Kimi Code now literally contains a migration rule pointing existing K3 sessions toward k3.1.

If Moonshot follows K3 with open weights again, K3.1 could become the next ridiculous multi-GPU / multi-node Local AI project.

Links in ALT

[David Hendrickson @TeksEdge]

A new decision model tied for #1 on Artificial Analysis' Decision Index and its 8 points ahead of Jev 1.14.

SubQ Snap 1.1 is built to make the decisions inside an AI agent.

Artificial Analysis Stats
🥇 SubQ Snap 1.1 → 65
🥇 d1-ultra → 65
🥇 Elephant 9B → 65
Jev 1.14 → 57

And Snap has one unusual advantage, its context
📚 384K context

Subquadratic says its Sparse Attention architecture reduces attention compute from quadratic to linear scaling with context.

Snap reads the context and returns probabilities over a fixed set of allowed decisions in a single forward pass.

Subquadratic reports these benchmarks
🧰 When2Call → 96.2
📉 81% fewer confident errors on its unanswerable-query test
📚 384K context
💰 $0.04 / 1M input tokens
🆓 no output-token charge

This isn't a local model today as Snap 1.1 is currently offered through Subquadratic's API.

Layer this between the prompt and the models

Link in ALT

[David Hendrickson @TeksEdge]

Nvidia released Kumo Relational, a small purpose-built AI model to predict what happens next from your relational database.👀

👉Customer churn. | Fraud. | Future demand. | Conversion. | Risk.

This is not a replacement for SQL. SQL tells you what already happened.

Kumo is designed to predict what happens next from the relationships and history already inside your database.

And it can do this without training a separate model for every task or manually engineering features.

Instead of flattening a relational database into one giant table, it works across the relationships between your existing tables.

Give it historical examples in-context, up to 10,000, and ask it to perform classification or regression.

Nvidia tested it across 📊 41 tasks 🗄️ 15 databases

They report it beating supervised relational deep-learning systems and expert-engineered gradient-boosted-tree pipelines.

On SAP SALT, Nvidia says it beat large AutoGluon ensembles by 8% MRR.

You can download the weights and run it locally on CUDA.

The checkpoints are tiny
🧠 Classification → ~120MB
📈 Regression → ~124MB

Your model doesn't just query your private database anymore. It can try to predict what happens next from it.

🔓 NVIDIA-authored code → Apache 2.0
⚖️ Weights → OpenMDW 1.1

Link in ALT

[David Hendrickson @TeksEdge]

2 x ~$400 Snapdragon mini PCs. No GPU used and 64GB of RAM combined.

And they're running a 95GB Qwen3.8-Flash-Next at 20.3 tps. 👀

The model is 31GB larger than their combined RAM.

I have to build this.

The setup
🖥️ 2× Lenovo IdeaCentre Mini
⚙️ Snapdragon X X1-26-100
💾 32GB LPDDR5 each
💽 512GB NVMe each
🔌 connected with one USB4 cable
💰 ~$800 total

Here's the model:
🧠 Qwen3.8-Flash-Next
📦 IQ4_NL → 95GB

Reported with the author's ToRc inference engine
📥 Read → 46.2 tps
✍️ Generation → 20.3 tps
💻 Coding generation → 19.2 tps

And the Adreno GPUs aren't being used.

ToRc uses the Snapdragon CPUs + Hexagon NPUs for some of the dense math, while the model's experts are streamed from NVMe as they're needed because the weights don't fit in RAM.

The second box also isn't simply splitting normal token generation in half.

For Flash-Next, ToRc pipelines MTP across the two machines over USB4.

⚠️ ToRc isn't publicly released yet and these are the author's results, not independently reproduced benchmarks.

Link in ALT

[David Hendrickson @TeksEdge]

This may be one of the more important Healthcare specialized Local AI weight releases.

Ant open-sourced the weights for Ling-3.0-flash-Sante.

The model itself isn't new as Sante has been available through APIs since September but ...

What's new is 👉 you can now bring the healthcare model home.

Stats
🧠 124B total parameters
⚡ only ~5.1B active/token
📚 256K context
🩺 healthcare-specialized
🔎 deep research
🤖 multi-agent collaboration
🔧 tool use

It's built on the same Ling-3.0-flash architecture that already runs in llama.cpp.

For reference, base Ling-3.0-flash GGUFs already exist around
Q4_K_M → ~77GB
IQ3 → ~54GB
IQ2 → ~40–46GB
IQ1 → ~28–31GB

⚠️ Those are BASE Ling-3.0-flash quants. I haven't found an official Sante GGUF yet.

But now that the Sante weights are downloadable, this becomes a home model for
🖥️ 96/128GB workstations
🍎 high-memory Apple Silicon
⚡ DGX/RTX Spark-class unified memory
🔗 multi-GPU local systems

And healthcare is one of the places where Local AI can be useful for private documents + research + agent workflows can stay on hardware you control instead of being sent to a cloud API.

Ant also reports substantial healthcare gains in the new release, including

HealthBench Professional
45.73 → 54.29
AFUMED ComplexReasoning
65.22 → 76.92

Link in ALT

[David Hendrickson @TeksEdge]

Wow, that's a pretty nice bump for Ling-3.1-flash.

Ant introduced the new Ling-3.1-flash-1019 checkpoint, and its Artificial Analysis score jumped

Ling-3.1-flash → 41
Ling-3.1-flash-1019 → 44

The highlighted gains come from its updated agentic performance
AA-Briefcase
~1400 → 1524 Elo

AutomationBench-AA
62% → 68%

The existing Ling-3.1-flash is already open-weight under MIT and it's big.

Stats
🧠 ~560B total
⚡ ~25B active/token
📚 up to 1M context

Community GGUFs already go down to
IQ1_S → 142GB
IQ2_XXS → 174GB
IQ3_XXS → 256GB
Q4_K_M → 354GB

The existing Q4 has even been loaded across 4× DGX Sparks using llama.cpp RPC.

⚠️ 1019 is a new checkpoint/update, not a new Ling architecture announcement.

Ant hasn't released the updated 1019 weights yet. It says open weights + API are coming soon.

Link in ALT

[David Hendrickson @TeksEdge]

Ling-3.1-flash scores 41 on Artificial Analysis’s Intelligence Index versus its predecessor.

The 560B model I posted last week now has another reason to stay on the watchlist.

You can test it before the weights arrive on OpenCode because it has added free hosted access, and Vercel’s free trial runs through October 13.

Vercel currently serves 262K context. The advertised ceiling is 1M.

Still waiting on downloadable weights and confirmed local deployment requirements.

Save the tasks it handles well now, then rerun them on a local quant when available. How much of that capability survives?

Vercel link in ATL

[David Hendrickson @TeksEdge]

Apparently, private Local AI roleplaying AI is a pretty serious use case.

ReadyArt's new Wicked-Tutu family is already pulling serious downloads.

And this is not a tiny model

Stats
🧠 31B dense Gemma 4
💾 Q4_K_M → ~18.7GB
🎮 fits in 24GB-GPU territory
📚 up to 256K context
🏠 completely local

Runs directly with
⚙️ llama.cpp
🦙 Ollama
🖥️ LM Studio

ReadyArt trained it specifically as a private local roleplay model, with a purpose-built dataset and thousands of scenario categories.

The earlier v0.1 GGUF has already passed 17K downloads, and they're iterating fast enough that a Final 31B build is already out.

Local AI isn't only competing with ChatGPT but it's enabling categories where privacy + customization + no cloud at all are the product.

⚠️ The HF metadata says Apache 2.0, but the model card also adds personal/non-commercial terms, so I'd treat the licensing as unclear for now.

Link in ALT

[David Hendrickson @TeksEdge]

This small Local AI model could replace an entire document-processing pipeline.

ATH-MaaS released OvisOCR3.

It’s only a 0.8B-class multimodal model.

But one local model can now do all these things
📄 document → Markdown
🧾 document → structured JSON
🔎 OCR + bounding boxes
🧮 formulas → LaTeX
📊 charts → structured data
🔑 key-value extraction
🖼️ figure captioning
🔀 flowcharts → Mermaid
📚 cross-page table reconstruction

That last one is interesting.

Give it consecutive document pages and OvisOCR3 can determine whether a table continues onto the next page and reconstruct the complete table.

Team-reported results
OCRBench → 88.8
OCRBench KIE → 90.5

Cross-page table Relation F1
OvisOCR3 → 97.07
Qwen3.8-Max → 94.47

And it’s
🧠 built on Qwen3.5-0.8B
🔓 Apache 2.0
🏠 downloadable/local
⚙️ Transformers + vLLM + SGLang
🔌 OpenAI-compatible local server through vLLM

No official GGUF that I can find yet.

Link in ALT

[David Hendrickson @TeksEdge]

The company behind PUBG just trained its own 29B Local AI model from scratch.

And the architecture is interesting.

KRAFTON’s new Raon-1 mini is …
🧠 28.8B total parameters
⚡ only 4.3B active/token
💾 official Q4_K_M → 19.7GB
📚 262K context
🔓 Apache 2.0
🤖 reasoning + agent/tool use

18 of its 25 layers use fixed-state linear attention.

Only 7 layers maintain a growing KV cache.

KRAFTON says those layers use only ~8 KiB of BF16 KV per token.

So even at the full 262K context, that’s roughly:

👉 ~2GB of growing KV cache before other runtime/state overhead.

And in KRAFTON’s reported comparisons, this new 30B-A4B model beats Qwen3.6-35B-A3B on 11 of 18 listed benchmarks.

Lots of new models coming for Qwen3.6-35B recently.

⚠️ Preview release. KRAFTON’s benchmark results. Upstream llama.cpp support is still pending and KRAFTON provides its own branch today.

Link in ALT

[David Hendrickson @TeksEdge]

Nvidia lets you build a 1TB Local AI cluster out of DGX Sparks.

& Nvidia has a playbook for AI clustering them.

8 × DGX Spark
🧠 128GB each
↓
🔥 1TB unified memory across the cluster

Nvidia now officially supports manually configured clusters of up to 8 x DGX Sparks using their built-in ConnectX-7 networking.

And Nvidia says that 1TB is enough to run frontier-class models like GLM-5.3 locally.

They specifically call out
🧠 larger models
📚 longer contexts
🤖 multiple concurrent agents
👨‍💻 multiple developers sharing one local backend
⚙️ distributed development

They list models such as
→ GLM-5.3 Flash
→ DeepSeek V4 Flash
→ Qwen3.8 Flash Next

And later this month Nvidia Sync Model Launcher is supposed to make it easier.

Link in ALT

[David Hendrickson @TeksEdge]

@UnslothAI Right on! Let the vibe coding begin! Can’t wait to see people put this to the test!!

[David Hendrickson @TeksEdge]

AMD and Intel agreed on something pretty unusual.

A local PyTorch AI app shouldn't have to care whether the CPU underneath it is AMD or Intel.

AMD is folding its ZenDNN AI optimizations into oneDNN, the library originally developed by Intel.

AMD + Intel have now aligned on oneDNN as the default open-source x86 acceleration path in PyTorch.

ZenDNN has been working on refining for ...
🧠 LLM inference
⚡ INT4 / INT8 / BF16 / FP16
🧩 fused MoE operations
🚀 vLLM CPU acceleration
🔧 optimized attention/matmul

Now that work is being brought into the common oneDNN ecosystem AMD + Intel are also jointly developing ACE, new x86 AI matrix instructions.

The plan is for future AMD/Intel AI instructions → oneDNN → PyTorch

So local PyTorch software gets one x86 target instead of separate vendor paths.

⚠️ This does NOT make llama.cpp/GGUF faster, and AMD/Intel haven't published new performance numbers from the merger.

🎯But for CPU Local AI, the two companies that spent decades fighting over x86 are now cooperating on the AI software layer.

Link in ALT

[David Hendrickson @TeksEdge]

Here we go! Purpose built AI generated LLMs!

Then AI helped optimize the code that runs it.

Recursive released Flint, a new coding/agent model designed specifically for fast local inference.

Beats Qwen3.5/3.6/3.8 in some coding benchmarks!

Stats …
🧠 39.8B total parameters
⚡ only ~2.5B active/token
📚 262K documented context
🤖 reasoning + tool use
🔓 Apache 2.0

And there are official GGUFs:
🟢 NVIDIA NVFP4 → 22.9GB
🍎 Apple INT4 → 26.0GB

Recursive recommends 48GB-class memory for these local builds, so think:
🍎 48GB+ Apple Silicon
🟢 48GB+ NVIDIA
⚡ 64GB RTX Spark

Recursive reports its optimized runtime at
🍎 M4 Pro → >125 TPs
🟢 64GB RTX Spark → >150 TPs

And WIRED’s Will Knight actually ran Flint locally on a 48GB Mac and reported it was fast and successfully handled the coding jobs he tried.

Side Note: Recursive built Flint using its Eureka Machine, an automated AI research system that helped
🧩 design the architecture
🧪 develop training recipes
⚙️ optimize infrastructure
💻 write hardware-specific inference kernels

Nvidia says those specialized kernels improved Flint’s RTX Spark generation performance by 55% wo changing the model weights.

Recursive-reported coding results
SWE-bench Verified → 74.2
SWE-bench Pro v2 → 89.7
Terminal-Bench 2.1 → 54.9
LiveCodeBench v6 → 80.3

So this isn’t merely AI writing more AI code.

Link in ALT

[David Hendrickson @TeksEdge]

@RandyNorthrup No not yet. Some ggufs weren’t loading in llama.cpp I suspect that was the reason. Good old Bartowski worked

[David Hendrickson @TeksEdge]

This might be my new favorite all-purpose local AI model. Fits nicely into 16GB of VRAM for an RTX
5060/5070TI.

The new SparkX2.6-4B beats Qwen3.5-9B in 80% of shared benchmarks and 97% of Gemma4-12Bs.

Stats
💾 official Q4_K_M → 3.54GB
🧠 only 4B parameters
📚 native 1M context
💭 none / low / high thinking budgets
🤖 trained for local agents
🔓 Apache 2.0

It already runs with:
⚙️ llama.cpp
🦙 Ollama
🖥️ LM Studio
🍎 MLX
🚀 vLLM / SGLang

In the team's reported benchmark suite Averages:
Spark-X2.6-4B → 63.3
Spark-X2.5-4B → 52.1
Qwen3.5-9B → 49.1

So they're claiming a 4B model beats their tested 9B comparison overall.

For normal local-agent contexts, though, a ~3.5GB reasoning/agent model with this runtime support is very interesting.

Link in ALT

[David Hendrickson @TeksEdge]

A single 24GB GPU can now run a 35B scientific research agent that spawns sub-agents and checks its own evidence.

Sage-A1.5 is 35B / ~3B active, and its official Q4_K_M is just 21.7GB.

Here are the new Sage-A1.5 model stats
🧠 35B MoE / ~3B active
💾 official Q4_K_M → 21.7GB
🎮 fits a single 24GB GPU at moderate context
📚 256K native context
👁️ multimodal
🔓 Apache 2.0

And it's specifically trained as a long-horizon research agent.

So it is designed to
🔎 use tools/search/code
🤖 spawn parallel sub-agents
✅ inspect its own evidence
🔄 change direction when research stalls

No separate critic model required for its verification mode.

These are the full-precision benchmark results

PinchBench
Sage-A1.5 → 90.0
Kimi K3 → 88.2
GLM-5.3 → 86.6
GPT-5.6 Sol → 87.3

FrontierScience-Olympiad
Sage → 81.0
Kimi → 69.0
GLM → 67.0
GPT → 73.0

SWE-Pro
Sage → 61.2
Kimi → 63.3
GLM → 64.6
GPT → 64.6

⚠️ Those benchmark scores are from the BF16 checkpoint, not the Q4.

But the same model now has an official 21.7GB GGUF with llama.cpp / Ollama / LM Studio support.

Link in ALT

[David Hendrickson @TeksEdge]

@NayteThaGreat Yeah going to test it myself on my own benchmarks. So far no small model has been able to beat Qwen3.5-9B. I’m getting ~70 tps so far on this new model

[David Hendrickson @TeksEdge]

@LottoLabs 💯

[David Hendrickson @TeksEdge]

Underdog released Bark 3.0, and it's only a 163MB download. Runs locally w/Pytorch+CUDA or MLX

On their Open ASR Leaderboard evaluation, Bark wins
🐶 Bark 3.0 → 4.94% WER → 163MB
🟢 NVIDIA Parakeet → 4.94% → 2.51GB
🌀 Whisper large-v3 → 6.04% → 9.26GB

So Bark matches Parakeet's average accuracy while being 👉 ~15× smaller

And beats Whisper while being 👉 ~57× smaller

On recorded meetings check out these stats
Bark → 7.82% WER
Parakeet → 9.42%
Whisper large-v3 → 13.63%

And this isn't CUDA-server-only.
🏠 fully local
📱 phone-class
🍎 MLX + Core ML
🧠 Apple GPU / ANE
🪟 CPU + CUDA paths
🔓 CC-BY-4.0

I've checked and no GGUF. It uses its own compressed Bark format plus MLX/Core ML/Transformers runtimes.

Bark 3.0 Plus is only 180MB and improves average WER to 4.76%.

⚠️ English only

Link in ALT