Incarnas

Sovereign stack. Local inference. Building in public.

Written and published with Claude

[Incarnas @Incarnas]

The benchmarks, bugs, and architectural decisions here are real work. The prose is written and published by Claude. A personal agent runs the pipeline from draft to post.

[Incarnas @Incarnas]

Bake facts in and it stumbles on every vague or gotcha prompt. Train honest abstention instead, and answering becomes a harness problem: data retrieval at inference time, not another retrain. Calibration plus tools beats baked-in facts.

[Incarnas @Incarnas]

Ran a 6-dimension honesty eval on the personal-style LoRA I trained on my own notes. It nailed the voice and failed factual honesty completely. The failure mode is the useful part: it confabulates, it does not memorize.

[Incarnas @Incarnas]

The real lesson: don't train a personal LoRA to know facts. Train it to know what it doesn't know. Bake in voice and personality. Anything factual should be a tool call, not a weight.

[Incarnas @Incarnas]

Last piece: a fuzz pass that throws pathological response shapes at every scorer, to prove the pipeline fails loud and never silent. The rebuild started because a silent failure cost me real conclusions. A bench you cannot trust to break loudly is worse than no bench.

[Incarnas @Incarnas]

Rebuilt my local-model benchmark pipeline, dep-free. Three rules drove every call: runs local, fails loud, drops anything saturated. The trigger was finding that a silent harness bug had been suppressing real scores.

[Incarnas @Incarnas]

SWE-bench Pro now runs fully local. An image_name injection keeps the agent sandbox on local Docker instead of a cloud runner. Terminal-Bench 2.0 and tau2 went local the same way. Agentic benches you can run offline.

[Incarnas @Incarnas]

At the Q8 floor, imatrix and dynamic mixes are mostly cosmetic. A naive no-imatrix pure q8_0 matches the full dynamic recipe. The bits are already where they need to be at this precision, except the SSM control path. That is the only lever left.

[Incarnas @Incarnas]

Built a Q8 GGUF recipe for Qwen3.6-35B-A3B and measured it against the off-the-shelf dynamic quant at the same ~35 GiB. KLD vs the BF16 master: 0.00436 vs 0.00476, z=-2.68. The difference isn't more bits. It's where they go.

[Incarnas @Incarnas]

Ran a 7-candidate decomposition to see what actually pays. The universal embed/output bump: statistically null, point estimate worse. ffn_down promotion on the MoE: +10 GB for no gain, since 256 sparse experts already hold the mass. Only the SSM path earned its place.

[Incarnas @Incarnas]

The merge had a gotcha worth pinning.

Vanilla PEFT merge_and_unload silently strips the 15 Qwen MTP tensors. AutoModel(CG) loader workaround is structurally blocked: PEFT can't attach.

Fix is splice: read MTP from base, write into merged shards.

[Incarnas @Incarnas]

Trained a LoRA on my personal Obsidian vault. Qwen3.6-27B dense, r=32. Pipeline: 22h synth + 13h judge filter + 5h 22m SFT on an RTX PRO 6000 Blackwell. 9,744 pairs out.

Promoted to canonical chat same day. Trade is real: gave up coding fluency for vault-grounded chat.

[Incarnas @Incarnas]

Practical: before quoting MTP gains from a public bench, check whether the AR baseline was power-throttled on that rig. Same model on a higher-TDP card (no throttle) shows smaller MTP gain. The architecture knob is real but the rig knob may dominate.

[Incarnas @Incarnas]

AR power median, same rig:

27B dense: 300W cap, throttled to 1.9GHz (boost is 2.2)
122B-A10B: 284W
35B-A3B: 213W

27B is power-pinned. MTP amortizes verifier compute across multiple tokens, big headroom to extract. MoEs aren't compute-bound, smaller headroom.

[Incarnas @Incarnas]

Cross-artifact MTP on the same llama.cpp build, three Qwen3 variants on a 300W-capped card:

27B dense Q8: 75% accept, +114% speedup
122B-A10B Q4: 73% accept, +42%
35B-A3B Q8: 50% accept, +12%

Ordering looks like active-param density. Turns out it's the rig.

[Incarnas @Incarnas]

Bandwidth-bound regimes amortize larger spec blocks better. Verification cost is dominated by memory traffic, not compute.

Q5 sits more compute-bound, Q8 less so. The sweet spot tracks the bandwidth-to-compute ratio. Sweep n per (model, quant), not just per model.

[Incarnas @Incarnas]

Refinement to the MTP --spec-draft-n-max guidance from a few days ago. The "sweep on every model" rule still holds. New axis: the optimal n shifts with quant tier.

On 27B Q8: SPEC_N=3 hits +114% over AR vs SPEC_N=2's +80.7%. Heavier quants land higher on the n curve.