fntAInhead

A tinkerer crafting innovative projects and cool creations with AI. Also, the mild ramblings of a madman.

[fntAInhead @fntAInhead]

@sattyyouneed great question... perfect competition.

[fntAInhead @fntAInhead]

@aaditsh all of our brains would break if Alex Karp joined

[fntAInhead @fntAInhead]

though I don't disagree with you, I'm going to argue that it's been a great learning investment in how AI and LLMs work more generally. there is also something to the comfort of keeping personal information private to you and the usefulness of most local models as most useful intelligence does not need a frontier model. I'll finish by saying that I'm taking this position mostly to protect my own self esteem knowing that I probably won't recoup my money lol.

[fntAInhead @fntAInhead]

Whole loop on one RTX 5060 Ti 16GB: Qwen plans/codes → Z-Image paints → UltraSharp for 4K → Ornith restored.

Local wallpaper factory. No cloud gen API.

[fntAInhead @fntAInhead]

Built a local digital-art wallpaper pipeline on an RTX 5060 Ti 16GB.

Desktop 1080p → desktop 4K → phone 1080p → phone 4K. One GPU. One theme: Blade Runner night, empty streets, flying cars only.

Agent that wrote it: Qwen3.8 27B UD-Q2 xhigh. Image model: Z-Image Turbo Q8.

🧵

[fntAInhead @fntAInhead]

The agent recipe that built this (qwen38-q2-xhigh):

Qwen3.8-27B-UD-Q2_K_XL.gguf (~9.83 GiB)
llama-server (turboquant) · ctx 131072 · ngl 999
KV: turbo3/turbo3 · flash-attn on · draft-mtp n=2
reasoning_effort=xhigh · reasoning_budget=-1 · preserve_thinking
temp 1.0 / top_p 0.95 / top_k 20
b=2048 ub=512 · text-only (no mmproj)
≈13 GiB of 16 on RTX 5060 Ti

Vision+xhigh does not fit 16GB. Text-only xhigh does.

[fntAInhead @fntAInhead]

@HenrykAbram @bojarowski @Bentyn What does the 32gb ram do? you lose all your speed if you offload anything to ram. further to that as you approach 100k context, that speed drops dramatically due to context processing.

[fntAInhead @fntAInhead]

@sattyyouneed The talent part I don't think is true anymore

[fntAInhead @fntAInhead]

@superalesha Dude, this is amazing. Thank you, I've been looking to understand the difference between quants beyond individual feelings on it and you delivered like crazy. Please keep doing runs like this if you can.

[fntAInhead @fntAInhead]

@MiaAI_lab Desperately want this to happen. For 16gb VRAM gpu's MoE models are ideal.

[fntAInhead @fntAInhead]

Sources: TurboQuant https://t.co/b8dYVPhNwI (Google, ICLR '26); KVarN https://t.co/1vxPoK6u3K; turbo3 PPL in llama.cpp PR #21307 + discussion #20969.
TL;DR — turbo3 for speed, KVarN for long-context fidelity. Don't trust anyone's "50 t/s KVarN" without a benchmark.

[fntAInhead @fntAInhead]

Benchmarked KVarN vs turbo3 the right way — same weights, same context, same MTP. Qwen3.8-27B (Q3_K_XL @ 100k, all-on-GPU): turbo3 = 30 t/s, KVarN = 10 t/s (~3×). Ornith-1.5 (Q4, CPU-offloaded experts): equal (30 t/s). The KVarN speed tax only shows when the model is fully GPU-resident. 🧵

[fntAInhead @fntAInhead]

Quality is why KVarN exists. kvarn5/kvarn4 = 5-bit K / 4-bit V vs turbo3's 3.125-bit, plus the last 1024 tokens stored exact. Anbeeld's own matrix: kvarn5/kvarn4@1024 lands within ~3% of q8_0 quality for 37% less KV memory.

[fntAInhead @fntAInhead]

@jacekdziwisz I'd be curious to understand whether there was any foresight here or not. It seems like apple is going to be in a great position to compete in the local AI space with high intelligence open weight models for years to come.

[fntAInhead @fntAInhead]

That’s the whole setup. No extra GPU. No 64GB RAM. Just a 5060 Ti 16GB tuned for long-context coding instead of short chat.

[fntAInhead @fntAInhead]

Set up a local coding stack on an RTX 5060 Ti 16GB + 32GB DDR5.
Target was simple: keep Ornith 1.5 35B-A3B loaded, use the full 262K context, and not live at 10 tok/s.
Landed at 35–40 tok/s.

[fntAInhead @fntAInhead]

Why this combo: • MoE keeps active params small enough for a 16GB card • APEX MTP is the speed bump • TurboQuant3 is what makes 262K practical on 32GB RAM • FP16 mmproj so vision doesn’t eat the quality budget