Primary

Local LLM Lab.
Applied AI, Local inference, Agentic systems.
DMs open.

[Primary @i_am_primary]

Rig notes:
Dynamics of box have changed significantly since deployment of vLLM on 9/30 (marked by white vertical line). Pinning models to cards offers less flexibility to route requests for balance.

Chart 1 measures utilization of each GPU. Green (bottom) card immediately became workhorse due to pinned model. Other two cards drop off in utility due to being secondary models and quicker to execute.

Chart 2 measures average temp of cards under load. With deployment of vLLM, loads are much less balanced due to model pinning (as illustrated pre/post white line). Although unbalanced, vLLM superior at handling concurrency via batching, which helps with thermals under load. Green bar (workhorse) barely moves off the average.

Chart 3 measures peak temp of cards under load. Peak dynamics inconclusive here, hard to tell if better/worse. 7 day rolling average peak temp 67C. But mostly due to mechanical effect of averaging, under-utilized cards pulling average down. Peak temps look more volatile than under load temps.

Workhorse on bottom is qwen3.8-fp8 (dense). Secondary workhorse on top is qwen3-coder-next-fp4 (moe). Ancillary models shared middle -- designed for less usage due to sandwiching.

[Primary @i_am_primary]

Spent days playing whack-a-mole with software factory -- every patch led to more edge cases. Micromanaging agents limited their ability to complete tasks.

Realized the patches were applied to wrong node. Strictness was removed from execution node, and placed onto planning and review nodes.

Planner prepares specs and clearly articulates pass/fail objectives. Execution node has freedom to implement the plan. Review node verifies work against plan objectives.

Seeing far less escalation in thinking. Suspect prior escalations were due to agent's inability to complete tasks within rigid framework. Reducing guardrail bloat gave agent freedom to arrive at solutions.

[Primary @i_am_primary]

@JoshARosen 100% agree -- faced bloat in skills library in the past, had to implement lifecycles for skills to surface, then move through promote/demote pipeline.

[Primary @i_am_primary]

@JoshARosen Cool part is with a learning loop, agents can self-manage library of skills relevant to specific systems and workflows. Skills and memory become more modular.

[Primary @i_am_primary]

@BoboLinNotes 100% agree with you. Tuning performance for inference stack has so many variables to consider. Various layers have different optimization objectives too.

[Primary @i_am_primary]

Wonder if it's possible to use VRAM pooling to spread model weights across cards, but with utilization routing.

If middle card hits thermal limits, backoff and re-route to bottom/top cards.

Not sure if possible -- portion of the weights sit on a card not engaged. Can model calculate with partial weights (maybe slower)?

[Primary @i_am_primary]

Rig notes:
Dynamics of box have changed significantly since deployment of vLLM on 9/30 (marked by white vertical line). Pinning models to cards offers less flexibility to route requests for balance.

Chart 1 measures utilization of each GPU. Green (bottom) card immediately became workhorse due to pinned model. Other two cards drop off in utility due to being secondary models and quicker to execute.

Chart 2 measures average temp of cards under load. With deployment of vLLM, loads are much less balanced due to model pinning (as illustrated pre/post white line). Although unbalanced, vLLM superior at handling concurrency via batching, which helps with thermals under load. Green bar (workhorse) barely moves off the average.

Chart 3 measures peak temp of cards under load. Peak dynamics inconclusive here, hard to tell if better/worse. 7 day rolling average peak temp 67C. But mostly due to mechanical effect of averaging, under-utilized cards pulling average down. Peak temps look more volatile than under load temps.

Workhorse on bottom is qwen3.8-fp8 (dense). Secondary workhorse on top is qwen3-coder-next-fp4 (moe). Ancillary models shared middle -- designed for less usage due to sandwiching.

[Primary @i_am_primary]

Due to stacked cards configuration, model pinning was chosen. Placed workhorse model on GPU with best airflow.

VRAM pooling is another option -- spread model weights and utility evenly across cards. Initial testing in box found although utility ramps up evenly, thermals unfortunately do not (middle card becomes thermal ceiling for all cards/pool).

[Primary @i_am_primary]

@MrrrOzi I agree -- vLLM significantly increased throughput of model instances through concurrency (switching from Ollama), while maintaining acceptable temp ranges.

Do miss those balanced load measures though 🙂

[Primary @i_am_primary]

Correction: Chart 3 dashed line says middle avg peak 7 days. Label is incorrect. 7 days avg peak of all cards, as titled

[Primary @i_am_primary]

Rig notes:
Expanded telemetry at every level of stack. Model, inference engine, routers, queues, etc. This was long overdue.

Useful in tuning and debugging systems. Also helpful in drawing sharper conclusions about overall system performance and resource draws.

Refactoring applications to new inference engine provided great opportunity to harden coding factory. Various jobs passing through factory revealed limitations of local llm.

Overall system went through major changes, especially pre/post vLLM. Data draws interesting pictures.

[Primary @i_am_primary]

Stock fans:
4x Antec Tranquil 140mm
2x Antec P12R 120mm

Replacement fans:
4x Noctua A14x25 140mm
2x Noctua A12x25 120mm

[Primary @i_am_primary]

Rig notes:
Swapped stock fans and added 140mm intake under shroud.

Front -- 3x 140mm (intake)
Back -- 1x 140mm (exhaust)
Top -- 3x 140mm (cpu exhaust)
Bottom -- 1x 140mm (intake)
Shroud -- 2x 140mm (intake)

Skeptical on results -- due to stacked card config of box, not enough space to pass cool air through. Sandwiched cards barely have surface area to shed heat.

[Primary @i_am_primary]

*Note: qwen3-coder doesn't actually think. When coder requires thinking, its escalated to qwen3.8 dense model for additional reasoning.

[Primary @i_am_primary]

Results:
Deployed escalated thinking to dynamically set required reasoning levels. Laddering starts from none to high.

Test:
File jobs spanning across 15 files, n=10; planner vs. coder; default vs. ladder

Planner:
With default, planner tends to over-think. T/out is ~4x higher than ladder while T/in approximately same.

With laddering, same set of plans are produced with 75% reduction (seconds and T/out), because it starts from least amount of reasoning -- none. Of n=10, no plans escalated.

Coder*:
With default, coder tends to under-think. 40% of jobs were escalated to higher reasoning when ladder available.

When escalated, reasoning increased ~25% as measured by T/in (from 67K to 83K). Runtime on average stayed same, but tails expanded due to escalation.

While runtime looks like regression, its not. Far cheaper to spend time upfront executing correctly, than review node to send back to rerun in revision cycle.

[Primary @i_am_primary]

Findings:
Review node faster with thinking off; no reduction in accuracy.

For simple True/False judgements, thinking leads to unnecessary reasoning. Tok/out shows scale of difference. Thinking requires far more tokens and time to produce same set of judgements.

One would assume reviews require thinking. Testing showed otherwise. Accuracy across all models held. Thinking significantly slower with tail reaching 177 seconds.

[Primary @i_am_primary]

Rig notes:
Swapped Ollama for vLLM. Gained throughput, lost model zoo.

Finally deployed vLLM. Ollama was great for quickly load/unload models as needed. But throughput was painfully low.

Ran into concurrency issues in coding factory. When CF churned, it backlogged other applications waiting to request inference engine.

Switching to vLLM wasn't easy. Applications rely on range of models to function. Models range Qwen and Gemma, Dense and MoE, vision, coder, etc.

Can't load them all using vLLM, so had to consolidate application to use fewer models. Mainly qwen3.8-fp8 and qwen3-coder-fp8, with gemma4:27b-fp4 for fallback and diversity.

Did lose some flexibility around thermals and routing (due to model pinning), but concurrency and throughput improved overall -- allowing jobs to hop off cards quicker. Thermals under control with stacked card constraint using routing.

[Primary @i_am_primary]

Correction on util figure -- subtitle says 86% across all cards. Running avg is 82% as displayed. 86% is 9/25 snapshot.

[Primary @i_am_primary]

Thermals showing improvements:
Avg GPU utility: 82%
Avg GPU temp: 70C
Avg Max temp: 81C

Usage data shows that with addition of 3rd card, GPU has headroom. Load is balanced across all three cards with back-offs around temp thresholds using router.

I'm able to better hold certain temperatures, but throughput is reduced as cards wait to cool.

If cards stressed to 90%+ utility, middle card will hit 90C.

[Primary @i_am_primary]

https://t.co/TIhxaOVQ2F

[Primary @i_am_primary]

Prior tests showed that with n=50, thinking OFF was faster and produced more usable plans in my coding factory (CF).

Lately jobs have grown in complexity. Requests sent to CF touch multiple files, modules, and/or repos. Tasks are no longer atomic.

Churn appears again. We saw churn before from thinking ON, overthought for small atomic tasks.

I suspect this time its under-thinking for more complex tasks. Jobs get stuck in cycle of revisions, which hints at this.

[Primary @i_am_primary]

Findings:
1) Planner tends to over-think
2) Coder tends to under-think

When coder under-thinks, it either produces no edits (gives up). Or it churns -- caught in a revision cycle of under-thinking a solution to an under-thought attempt.

Solution:
Thinking escalation ladder.

Coding factory attempts to plan jobs with minimal thinking. If plan review fails, then re-tries with higher thinking. Prevents over-thinking on small jobs, and under-thinking on large. Same for coder. If first attempt not pass, re-attempt with higher thinking.

Thinking level also defined as param in job. Saves effort by defining thinking levels upfront (will escalate if insufficient).