Allen

Homelab tinkerer — Unraid, networking, GPUs, SmartHome, local AI models. What I measured, broke, and kept.

[Allen @MongooseAI]

Quick lab note on cold prefill for long context.

Matched near 253K prefix cold rows on the dual RTX PRO 5000 72 GB primary. Flash XHigh pure prefill sat about 10186 tok/s, first token about 26.6 s. GLM Max pure prefill sat about 1477 tok/s, first token about 172 s.

Near 500K, GLM Max cold prefill sat about 1724 tok/s and first token about 293 s. Flash XHigh 524K at about 499K sat about 6183 tok/s with first token about 84 s.

So Flash lands in the 10k tok/s class on the near 253K cold prefill, while GLM sits near the 1.5k to 1.7k class depending on depth. Long cold TTFT on GLM can run to a few minutes before the stream starts.

When you judge a local long context box, log pure prefill tok/s and first token time beside the decode scoreboard.

[Allen @MongooseAI]

Historical lab note from Aug 29, still useful.

Local MoE sat near 1 tok/s on a dual 3090 Ti box. Host free RAM still looked large. The Docker parent cgroup for the whole stack was capped at 80 GiB and already about 79 GiB full.

Raising that parent from 80 to 180 GiB let GLM 5.3 Flash IQ4 experts leave mmap and sit in DRAM. Short smoke: decode about 0.97 to 8.2 tok/s, wall about 95.6 to 10.2 s.

Same day the parent settled at a persistent 200 GiB. Later hardware and leftovers superseded these absolute speeds. The useful lesson is the memory envelope you actually serve inside.

[Allen @MongooseAI]

Quick lab note from long context work early on.

Four near full Q4 histories on the baseline sat around 3.7 tok/s per agent and about 14.9 aggregate. Profiling pointed at a CPU side history lookup path dominating what looked like GPU decode.

After the indexed PLE repair, the matched four history run rose to about 6.2 tok/s per agent and about 25.0 aggregate, roughly a 68 percent lift, with retrieval still passing.

Even repaired, two live full slots still missed the 20 tok/s per session floor. That is why the later keep decision was one active decode lane plus RAM history switching around 29.5 tok/s.

Useful takeaway for huge local agents: fix the serializer first, then still measure against a per session floor, not only aggregate tok/s.

[Allen @MongooseAI]

Where the lab sits now:

There is a primary Unraid and Threadripper world where the household machine, VMs, and a lot of the keep the house online work still lives, including the multi seat Windows setup and the router VM path.

There is a second host that exists mostly so inference and heavy GPU experiments skip fighting the family machine. Dual 3090 Tis still earn their keep there as workers. The 72gb Blackwell 5000 cards took over the big jobs.

Upcoming posts will be about why a VM hung. Some will be about why a model was slow. Some will be about the network path that made working from home less cursed.

[Allen @MongooseAI]

Quick lab note from a helper requal.

On dual 3090 Tis, a DFlash2 edit soak on the 27B stack looked great for that job. Short median sustained decode around 231 tok/s including reasoning. Matched replay wall time about 1.63 times faster than the older SPEC0 recipe, and about 2.05 times faster than a modern FA2 SPEC0 baseline on the same edit set.

It still failed the vision review gates. So it stayed an explicit edit candidate, not the default helper. Leftover helper remains SPEC0.

Useful takeaway: a big tok/s win on an edit workload is not the same as earning everyday helper promotion. Measure the role you care about.

[Allen @MongooseAI]

Quick lab note from a debugging issue.

Leftover GLM and Flash decode fell to about 1 tok/s. Easy to blame the model or the GPUs.

The cpu range flag only accepts one continuous interval. A comma separated list like 16 to 31 and 48 to 63 silently pinned the process to a single core.

After switching the launchers to a proper cpu mask, isolated High decode arms recovered into about the 20 to 27 tok/s class depending on layout.

Useful takeaway: print effective affinity from inside the container. A threading parser bug can look exactly like a dead slow model.

[Allen @MongooseAI]

In progress decode map from the two host lab.

Dual RTX PRO 5000 72 GB for Flash XHigh / GLM Max. Dual RTX 3090 Ti for 27B XHigh. Held 1M rows stay experiments.

[Allen @MongooseAI]

One more thread for later: why local AI showed up in this lab at all.

For a while the bill was the the WAF problem. Multiple Claude and Codex seats stacked up with different machines, different workflows and it was easy to land around TWELVE hundred a month without feeling like I was “done” needing more context or more agents.

That kind of burn is a great teacher. I started asking which jobs actually need a frontier cloud model, and which ones are just expensive habits.

Right now I still keep a strong Grok + Cursor setup (on the free cursor promo for now). I’m not pretending I’m fully off cloud but working towards that once validation holds.

But the direction is clear: measure what local can own, keep cloud for the bits that still earn it, and shrink the subscription pile on purpose as a budget line and a lab project.

Full writeup later. This is just the preview.

[Allen @MongooseAI]

I parked GLM Q4 Max on the RTX PRO 5000 72 GB alone. That cost a little decode versus packing experts onto a 3090 Ti too. Short unique sat around 22.8 tok/s on PRO solo versus about 22.7 to 25.3 when a Ti also held experts, and 100K restore sat about 17.7 versus 18.9. Big reason is likely 8 channel RAM on a Threadripper at 5200, so expert offload is not too painful.

This freed up both 3090 Tis for a serialized 27B helper. That helper is not optimized here and runs with no speculative decode (SPEC0). Thinking Low sat around 58 tok/s. Four client serialized requests pushed aggregate around 175 tok/s. A SPEC0 depth ladder still held retrieval needles past 250K tokens, with decode falling from about 61 tok/s at 8K to about 48 tok/s near the limit.

On a sticky MoE card plus two consumer GPUs, leaving the Ampere pair as a helper pool was better here. Measure the orchestrator tax and the helper leftover, not only the packed MoE peak.

[Allen @MongooseAI]

Standby generator battery died with no warning. No way to check until a cold start failed.

Bought a cheap battery maintainer and put it on a Z Wave plug. Weekly self check cranks hard enough that volts sag and the maintainer draws more amps. Home Assistant watches that spike for whether the self test ran, and watches the slow trickle the rest of the week. Green or red on the dashboard.

One smart plug made a silent failure show up before the next cold start.

[Allen @MongooseAI]

Here’s what “the lab” means when I’m not talking about GPUs.

**Household compute.** Multi-seat Windows (Aster) on a meaty box so the good computer isn’t a queue. That alone drives passthrough, VM, and “don’t break family night” choices.

**Unraid underneath.** Array, VMs, containers — Home Assistant, MQTT, Zigbee/Z-Wave bridges, Influx/Grafana, photo library, the usual long-running mess. When something hangs, it’s often here.

**House internet.** Multi-WAN / MPTCP-style path so work-from-home and streaming don’t die on one ISP having a bad day. Networking posts from this account are part of keeping the house online.

**GPUs with jobs.** Seats, creative work, then inference. Cards get roles on purpose. Two hosts so sticky models don’t choke the household machine.

**Cameras / NVR.** Frigate on the camera stack: person/car detection, event clips, the usual false-positive tuning. Wired into Home Assistant so motion isn’t a separate app you forget exists.

**Energy + climate.** Whole-home energy monitoring (circuit-level), utility meters, cost tracking, and a pile of scripts. Dual HVAC, weather-station feeds, blind automation, and some “use the basement as a free thermal reservoir” tricks when summer math says the compressor is the expensive answer. There’s also predictive climate work for rooms that actually have to stay in a safe band (vivariums / tanks), not just “make the living room comfy.”

p.s. Don't let AI reprogram an Emporia Vue 3 over wifi that needs hand-held flash contacts to reprogram if it screws it up...

**Fitness tracking, self-hosted.** Wearable data synced into my own stack with scoring, trends, dashboards next to the house sensors instead of living forever inside the cloud. Same Unraid as everything else.

**Local models.** The unfolding chapter lately: 3090-TI workers for fast jobs, bigger cards for sticky/big models, measurements and failures when the recipe says one thing and the box does another.

[Allen @MongooseAI]

This lab didn’t start as an AI box.

It started as one meaty PC the household could actually share with multiple seats on one Windows machine (Aster), so people weren’t fighting over whose turn it was on the “good computer.”

Around that grew the real homelab habits: Unraid underneath, VMs for the messy stuff, and a multi-WAN / MPTCP router path so the house internet wasn’t a single-point-of-failure toy.

The CPU side scaled up with a Threadripper build (Micro Center bundle: a new platform, then keep feeding it). RAM followed the same pattern, not one perfect purchase, just keep adding usable sticks / open-box capacity until the machine stopped running out of space.

GPUs were a whole subplot. AMD cards looked fine on paper and then hung us in Unraid VM / Aster land often enough that I stopped arguing with them. Swapped toward NVIDIA for passthrough sanity. One 3090-class card earned its keep on creative/homelab work (Blender and friends); another got dedicated to the shared seat machine so family use didn’t collapse every time I wanted a GPU for experiments.

Only after that pile existed did local AI stop being a pipe dream. Dual 3090 Tis first. Then bigger Blackwell cards (the cheap ones, relatively speaking). Then splitting into two hosts, one for the big models, one as a fast worker. The fast worker also still serves the family with router VMs + inference.

Also lots of late nights when I break things

[Allen @MongooseAI]

Quick lab note from a concurrency test.

Goal: keep long agent histories around 260K tokens, and stay at or above 20 tok/s on whatever session is actively decoding.

Stack for this run:
Model: Qwen3.8 Flash Next, Unsloth UD Q4 K XL
GPUs: RTX PRO 5000 plus one 3090 Ti (38 / 10 native expert layers)
Context: 262144 per slot, q8 0 KV
Host prompt cache budget: 32 GiB
Engine: llama.cpp family image with the indexed PLE repair

First arm: two resident full context slots, all native experts on those two cards, speculation off. Four independent ~259K histories, retrieval codes included.

Cached result: about 11.6 tok/s median per active session, about 22.5 tok/s aggregate.
Retrieval passed. Prefix reuse stayed high.
Speed missed the 20 tok/s per session bar.

Second arm: same weights and placement, one active slot, other histories parked in RAM and restored when needed.

Cached result: about 29.5 tok/s median on the active stream (min about 29.5). First token after a restore about 2.0 s median. Cold first ingest of a fresh ~259K history still took on the order of 5 to 7 minutes, so that cost is big. Once warm, the active lane held the target cleanly.

Engine prompt cache save/load/maintenance sat around 1.4 s median on that path. Separate from the full request first token time.

Kept: one active decode lane, RAM history switching

Useful takeaway if you are building long local agents: more concurrent near full slots are not free. On this hardware and quant, two live full contexts dropped per session speed from about 29.5 tok/s to about 11.6 versus serializing decode and restoring state from RAM. Measure against a per session floor, not only aggregate tok/s.

Lots faster speedups coming with sglang work...in progress.

[Allen @MongooseAI]

This isn’t a clean “local AI from day one” build.

It’s a house lab that grew teeth — shared PC seats, Unraid/VMs, multi-WAN networking, lots of smart home work, GPUs that misbehaved, then models on top of that pile.

Tomorrow I’ll walk how that stack actually grew.

[Allen @MongooseAI]

I’ve been building a local AI setup at home for a while — not a cloud demo machine, just hardware I actually use.

This account is the notebook for that. Some days it’s a new model that looks runnable locally. Some days it’s “I thought this would be fast and it wasn’t.” Some days it’s the boring infrastructure thing that turned out to matter more than the GPU.

Next couple posts I’ll walk through how the machines evolved, because the hardware story explains most of the weird results.

If you care about running models yourself, you might find it useful. If you just like watching people learn the hard way on crazy AI demand high cost toys, that works too.

[Allen @MongooseAI]

Expect a mix of:
- stuff I measured (speed, context, quants, weird failures)
- hardware notes (3090 Tis, Blackwell, RAM, why placement mattered)
- new local-capable models/runtimes when they’re worth a look

[Allen @MongooseAI]

I run local models on my own machines and keep notes on what works.

This account is those notes — plus the occasional new model release when it’s something I actually try.

[RootzWiki @RootzWiki]

RT @koush Clockwork recovery on droid x... http://yfrog.com/n2hiuj