Michael Wegener

Running big models on small hardware: RTX 3060, 5060 Ti, Strix Halo. Tok/s, KV cache, quant tricks. Self-hoster, PHP dev, security nerd.

[Michael Wegener @MWsatware]

The Last V8 - Build from a single /goal prompt with @coldniko 's Strata Server running @Alibaba_Qwen 3.8-flash IQ3 XXS Quant.

Sourcecode: https://t.co/gUUXamOmJc

[Michael Wegener @MWsatware]

@Bassmaster187 @opencode Vision ist für self-verification an. Da ich ja auch meine ganzen anderen monitoring Tasks zu Hause damit mache, lasse ich es mal so laufen.

[Michael Wegener @MWsatware]

Inspired by @Bassmaster187 's https://t.co/rmuUEnIzcf post, I handed over his prompt as an @opencode /goal to create a V8 Engine - Qwen 3.8-flash on my 3060 12GB is still on it.

[Christian P. @Bassmaster187]

1/ Mir wurde mehrfach vorgeworfen, dass Lokale LLMs schrott sind und gerade in der Webentwicklung NUR Opus 5.5 funktioniert. Alles andere ist komplett unbrauchbar.
Ok, also simulieren wir einen V8 in 3D mit Kameraschwenks, Schnitt im Motor, Explosionszeichnung usw... Ja, es sind kleine Fehler drin, wie z.B. der zweite Riemen, der in der Luft ist, die Zündkerzen zünden bei OT und nicht drehzahlabhängig vorher schon usw... Falls jemand kommt und sagt die Zündreihenfolge ist falsch: ja, für einen Standard Chevy ist das richtig. Aber Rennmotoren dürfen den Zylinder 7 und 4 tauschen.

Aber Fakt ist: meine Senior Entwickler würden das vermutlich nicht in einer Woche ohne KI schaffen.
Ich habe Verwendet: Qwen 3.8 27B mit OpenCode. Hat etwa 70 Minuten auf meiner AMD R9700 AI Pro gearbeitet.
Und ja, mit großer Sicherheit (hoffentlich) kann das Opus 5.5 besser und schöner. Aber ob das immer notwendig ist?

[Michael Wegener @MWsatware]

Local cost is marginal electricity only: no hardware amortization (the rig is ~EUR 2k retail), no idle draw, no labor. APIs stay infinitely faster in aggregate - one 3060 is one slot at a time; this is a marginal-cost-class plus data-ownership claim, never "beats the API". Cache-read math assumes our 88.8 % reuse maps 1:1 to provider prefix caches; eviction is theirs. No API publishes Wh/token, so no per-model energy claim exists. For scale, the artifact's flagship row: the same day on Claude Opus 4.6 list prices would have been $135.80 = 126x.

[Michael Wegener @MWsatware]

A day of local agent inference on the RTX 3060 box (measured Nov 9 to 10): 2.6 kWh, about EUR 1.00 of electricity at the 37.97 ct/kWh German household average (Destatis H1-2026), 0.9 kg CO2 direct - and the prompts never left the building. The same day on the hosted twin of this architecture, Qwen3.8-Flash at list prices with its cache-read rate applied to our 88.8 % prompt reuse: 3.90 USD, 3.6x. Serving cost, not output tier: local runs an IQ3_XXS quant, the endpoint is FP8-class.

[Michael Wegener @MWsatware]

Real traffic, not a bench: 1,096 requests, 111.10 M prompt tokens (98.71 M reused from parked conversation snapshots = 88.8 %), 12.40 M new prefill at ~890 tok/s wall median, 976 K generated at 45-52 tok/s by segment. Energy: prefill 1.10 kWh + decode 1.41 kWh + snapshot restores ~0.07 kWh = ~2.6 kWh. Power basis: GPU 137-149.5 W measured (nvidia-smi 1 Hz); steady-serving system figures (~250 W decode / ~285 W prefill) are a component-level bound from measured GPU plus CPU class-share, not a wall-meter read.

[Michael Wegener @MWsatware]

@Israfilv2 It's on a daily limit and data will run into training. Nothing for free but a cool offer to try things out.

[Michael Wegener @MWsatware]

Exact model: Qwen3.8-Flash-Next, quantized IQ3_XXS (GSQ-RCO, thanks @mhelcig ) - 71 GB across two
GGUF shards (44 G + 27 G), plus a BF16 mmproj for vision and an MTP module for
speculative decoding.

Server: Strata engine 0.1.56 by @coldniko (local build 06698c6), OpenAI-compatible API on :8080,
model id "qwen3.8-flash-next-iq3_xxs".

Key settings:
- Context: 196,608 tokens
- KV cache: k8v4 (8-bit keys / 4-bit values, lossless 4-bit tier) with KVA split
- Prefill: auto, expert cache 1200
- Speculation: MTP, 4-draft spec, min-p 0.70, stream hnorm
- VRAM: 960 MiB reserve, elastic, pcie-frac 0.20
- Caches: 12-slot prompt cache (tail), 8 GB conversation cache
- Adaptive expert swapping: every 2 steps, 192 swaps, decay 0.92
- Vision: strata-vision on CPU, 8 threads

Hardware: one RTX 3060 + 8-core AMD Ryzen 7 3800X + 64 GB (4x 16GB DDR4 @ 3000MT/s)

[Michael Wegener @MWsatware]

@Bassmaster187 @Wbackdown @opencode 62 GB Ram, nvme Auslastung ist zu vernachlässigen.

[Michael Wegener @MWsatware]

Hi, vielen Dank für das Feedback. Würde das jetzt so mal als Issue in das Repo übertragen lassen und schauen was eine zweite Runde aus dem Block macht.

Gleicher stack wie für den v1.0.0 release. So kann man sehen, wie sich das Modell bei Codeänderungen schlägt.

Goal: Verbessere die Note 3- (noch befriedigend) in eine 2+ (voll gut) ausgehend von https://t.co/PVXcozzlf9

[Michael Wegener @MWsatware]

@boardyai @coldniko @Alibaba_Qwen v16.

[Michael Wegener @MWsatware]

The Last V8 - Build from a single /goal prompt with @coldniko 's Strata Server running @Alibaba_Qwen 3.8-flash IQ3 XXS Quant.

Sourcecode: https://t.co/gUUXamOmJc

[Michael Wegener @MWsatware]

@boardyai @coldniko @Alibaba_Qwen I could tell Qwen3.8-flash UD-Q4_K_XL on gufo Server (AMD Strix Halo 395+) to fork and refactor to v12.

[Michael Wegener @MWsatware]

@Bassmaster187 @opencode Here you are: https://t.co/gUUXamOUyK

[Michael Wegener @MWsatware]

@Wbackdown @Bassmaster187 @opencode Done: https://t.co/gUUXamOmJc

[Michael Wegener @MWsatware]

@HermesAgentTips When I'm running on solar and battery overnight it's gufo server on a Strix Halo 395+ Box with ~120W tdp

It serves regardless of internet connectivity and quality is enough for dev/debugging even larger apps thx to vision tower and 256k ctx.

[Michael Wegener @MWsatware]

You can see the open /todo opencode is just working through. The goal is still active.

Another reminder: This is my homeoffice PC, I work on it during this experiment and I do things like writing posts like this one on it.

It even deployed this additional experiment: https://t.co/6x45Bkfvch so let's wait until the engine is ready.

[Michael Wegener @MWsatware]

@Wbackdown @Bassmaster187 @opencode It is on the way finishing it, it just takes some time to reach the goal.

[JanTec @jan_tec]

Back to the roots of my channel: filament testing! 🧪

Testing Poticon PA STD1 (potassium-titanate whiskers filament) for safety, strength, stiffness and more!

Watch now: https://t.co/c38i2EcpP1

[Michael Wegener @MWsatware]

@Bassmaster187 @opencode I will publish it on github as soon as it's done.

[Michael Wegener @MWsatware]

Widened the NVMe expert cache from 28 to 40 GiB: DeepSeek-V4.1 decode on the RTX 3060 rig went 7.32 to 7.86 tok/s = +8.49% (A/B/A, control drift 2.21%), prefill flat within drift. The price is 9.2 GiB RAM, and NVMe stays a cache-miss tax, not the floor.