kapicode

Building in public. Currently working on a custom Ralph implementation—harness-agnostic and has a TUI for progress

[kapicode @kapicode]

My #ralph implementation

[kapicode @kapicode]

Benchmarking rule I’m trying to follow:

If a result is inside run-to-run noise, call it noise.

Not “breakthrough.”
Not “secret flag.”
Not “RDMA is worse.”

Noise.

Local LLM work needs more boring honesty and fewer victory laps.

[kapicode @kapicode]

👀

[kapicode @kapicode]

@binh Are you sure you don’t have a config problem? What quants are you using? What is causing the stoppage in your harnesses?

[kapicode @kapicode]

If you run local LLMs: what is your actual setup?

Not the dream build. The daily driver.

GPU/APU?
Memory/VRAM?
Model size?
Serving stack?
What breaks most often?

I’m trying to compare practical local AI systems, not leaderboard screenshots.

[kapicode @kapicode]

@manibatra That’s an awesome setup!

[kapicode @kapicode]

DeepSeek-V4-Flash on one Strix Halo box was more interesting than I expected.

Not because it was the fastest thing in the world.

Because it made 256K context feel plausible on a local machine.

[kapicode @kapicode]

@whatup9911 What is the wrapper?

[kapicode @kapicode]

I want to push local hardware to be as capable as possible.

And I want to push my harness to be productive with the smallest/dumbest models possible.

No leaning on the model for my harness.

Caveat: there is a minimum model quality that I simply can’t avoid. Qwen3.5-9b breaks at q4 but not at q8 (for now)

[kapicode @kapicode]

I actually want to try baking RL into my harness lol.

Small model gives two options, big model chooses better option, over time small model makes better choices?

I already have multi-model requests built-in so you can compare the decisions various models make (one main “real” model, and many “ghost” models which don’t actually have the ability to run tool calls).

It’s proven somewhat effective at sussing out the strengths/weaknesses of models. But uhh… can be expensive.

[Taelin @VictorTaelin]

RL is a mistake, thinking is a mistake, and if we just put all the money into crafting an astronomically good, massive dataset, we'd pretrain a model that outperforms everything that exists by a considerable margin

source: my ass (I have no idea what I'm talking about)

[kapicode @kapicode]

@LottoLabs If someone has the bandwidth, I’d be curious to see if it could be a potential runtime optimization.

Also:
- Tiered K/V (hottest in vRAM, medium hot in RAM, cold on SSD)
- The CPU inference optimizations noted by Sakura Yuki

I think there is yet meat on the bone for local AI.

[kapicode @kapicode]

Is it possible to eGPU 3090 for prefill, then generate on the Strix Halo?

[Akash Alpha @akashalpha_]

@Code_Fault @kapicode Hey! Are these two connected? eGPU? Would do again?

[kapicode @kapicode]

@LottoLabs Yeah I’ve seen them—can you do the prefill split about which I asked? Could I pull it off with an eGPU and my existing BeeLink?

[kapicode @kapicode]

Pushing the idea further... manual REAP based on which tasks get used more and/or require more intelligence.

[kapicode @kapicode]

Thinking out loud.

Could domain-specific small dense models (SLMs) be trained and then combined together as a MoE?

Almost certainly you could put a manual router model in front, but it would be kind of cool if the whole thing could be consolidated into one model.

Imagine cherry-picking the experts you need for your use case.

[kapicode @kapicode]

Thinking out loud.

Could domain-specific small dense models (SLMs) be trained and then combined together as a MoE?

Almost certainly you could put a manual router model in front, but it would be kind of cool if the whole thing could be consolidated into one model.

Imagine cherry-picking the experts you need for your use case.

[kapicode @kapicode]

Strix Halo is vindicated by price/performance. 4x256GB Mac Studios barely beat it. You can essentially get the same speed on 1xSH with q5 with chadrock.

[Lifetimize @lifetimization]

4 node Mac Studio 256 GB cluster achieves 1.92x the speed with Qwen3.6 27B 8bit model via exo RDMA over thunderbolt 5.
@exolabs @0xSero

[kapicode @kapicode]

@0xSero Honestly this is just showing me that for 27b Strix Halo is still under-appreciated.

[kapicode @kapicode]

Coming soon...
@FlyCockpitApp

[kapicode @kapicode]

@0xRaghuboi Idk if this applies with nvidia hardware (I have a 3090 but have experimented more with Strix Halo) and I find that compressing KV less is actually faster, and if you have the headroom to not compress, it is often better to not.

Results may vary fwiw

[kapicode @kapicode]

@sakurayukiai I remember you posting about CPU cache optimization for running models on CPU. What is the best CPU-only setup you’ve seen so far?

[Whimsicali @Whimsicali]

I am currently running two split, separate models. Qwopus3.6-27B-v2-Q4_K_M with vulkan. I get better results with vulkan but I need to revisit rocm again. Both 7900xtx's hold their own in developing my projects. MTP is a big help. Not quite the speed of a 3090 but about $400-500 cheaper than is currently available. They are solid GPUs for local models.

[kapicode @kapicode]

@bakerybr02 @jack Llama? What kind of results do you get? What is the experience like?

[kapicode @kapicode]

@pierrelezan You are a baller