Punch Taylor

Local AI builder. Home mesh, hardware benchmarks, llama.cpp.
🦅 6 finger patriot. Hoosier.
Home AI mesh, build your own:

[Punch Taylor @punchtaylor]

@leopardracer Love this. Starting from scratch on your own data instead of bolting onto the big frontier models is where real control and privacy come from. Karpathy’s nanochat approach + Obsidian fine-tuning sounds like a killer weekend project. How’s the obsidian side working in practice?

[Punch Taylor @punchtaylor]

@Teknium

[Punch Taylor @punchtaylor]

@loktar00 NEED

[Punch Taylor @punchtaylor]

16 t/s on the dense at Q6 is about right for that machine, so nothing's wrong there.

And good news on your question — go Q4_K_M, you'll barely notice the difference. Most of the quality loss from quantization happens below Q4; from full precision down to Q4 you're giving up very little, which is exactly why Q4_K_M is everyone's default. Q6 is technically a hair sharper, but in practice it's mostly costing you RAM for quality you won't feel.

Here's the part that'll actually help you: on a Mac, generation speed is bottlenecked by how many bytes it has to read per token — so a smaller quant isn't just lighter, it's faster. Dropping that dense 27B from Q6 to Q4 should push it from ~16 t/s up into the low-20s, free up 5–6GB of RAM, and give you more context headroom and room for your other apps. Win across the board.

Keep a Q6 around if you ever want max quality for something specific, but Q4 is the right daily driver.

[Punch Taylor @punchtaylor]

Good instinct grabbing Unsloth's settings — those are solid, leave them be for now.

On 27B vs A3B for coding: there's a real gap, and it's the classic trade-off. A3B is a mixture-of-experts (MoE) — only ~3B of its 30B params actually fire per token, which is exactly why it feels so much faster. But coding rewards depth, and the dense 27B has all its weight working on every token, so on the hard stuff — gnarly debugging, anything that needs to hold a lot in its head — it'll usually out-reason the A3B.

How I'd run it: A3B as your daily driver (fast, light, leaves room for your other apps), dense 27B in your back pocket for when the fast one stalls. You've got the RAM to swap. For most day-to-day code, A3B is honestly plenty — you mostly feel the dense model's edge on the tricky stuff.

In my own setup I run a dense model for coding and an A3B for fast chat, for exactly this reason.

[Punch Taylor @punchtaylor]

Honestly the M2 Max 64GB is a great machine to learn on - more capable than most people realize. You can run 30B-class models comfortably, even 70B at 4-bit if you push it. (You probably didn't need to shrink your context as much as you did, btw - you've got headroom.)

Two things that'll save you weeks:

-Use LM Studio with the MLX runtime turned on. On Apple Silicon it's meaningfully faster than the default llama.cpp backend — I clocked +36% on my M4.
-Don't sweat quant or temperature yet. Start at Q4_K_M, temp ~0.7, and just use it. You'll feel what to tune once you've got a real task in front of you.

The model you pick matters way more than the settings. Have fun with it 🤙

[Punch Taylor @punchtaylor]

@sudoingX i’ve had problems with my claude code cli for the last 3 days. finally just said “fuck it” and canceled my sub. my hermes agent is far more capable on glm than claude is on opus anyway from my experience.

[Punch Taylor @punchtaylor]

the gotcha that bites everyone: writes hit disk instantly but the curated block is frozen at session start. so it "knows" but also doesn't until next session. learned this the hard way — anything that needs to land mid-session has to go through a live tool call, not the memory file.

[Punch Taylor @punchtaylor]

@nimabuyin @sudoingX actually i’m running it at Q4 on a m4 mac studio with a few other models running in parallel. why use one big model when i can run a few smaller models that punch above their weight?

[Punch Taylor @punchtaylor]

@sudoingX mostly local here. just dropped the claude sub but holding onto the Nous Portal sub for when i need something with a little more uumph.

[Punch Taylor @punchtaylor]

@loktar00 For what i’m doing currently, 152GB is totally enough. but it’s always nice to future proof. why settle when there’s a bigger dragon out there to chase?

[Punch Taylor @punchtaylor]

I made the switch to telegram from imessage recently. I love how I am able to build tg specific tools and workflows. much easier process than just imessage though i still keep my Blue Bubbles server up for just in case — never hurts to have a backup or 2 just as long as you’re not using them simultaneously on the same agent.

[Punch Taylor @punchtaylor]

Batch memory operations are a game changer for the overflow problem. Right now when agents hit the 22k builtin cap, they either freeze trying to save or lose context entirely.

Having batch save/edit/remove means the agent can actually clean house instead of choking on one entry at a time. That's not just convenience - it's reliability.

Just ran into this exact issue last night. About to hermes update and see if it fixes the bottleneck.

[Punch Taylor @punchtaylor]

Batch memory operations are a game changer for the overflow problem. Right now when agents hit the 22k builtin cap, they either freeze trying to save or lose context entirely.

Having batch save/edit/remove means the agent can actually clean house instead of choking on one entry at a time. That's not just convenience - it's reliability.

Just ran into this exact issue last night. About to hermes update and see if it fixes the bottleneck.

[Teknium 🪽 @Teknium]

We have just merged an expanded form of the memory management tool Hermes Agent uses to save/edit/remove memories so that it can do batch operations, saving many turns of tool calls in most scenarios!

Run `hermes update` in your CLI or update in your GUI to start saving now.

[Punch Taylor @punchtaylor]

@aiport_trade https://t.co/ADBE71CHgk

[Punch Taylor @punchtaylor]

5 PRs open against @NousResearch/hermes-agent — all mergeable after clearing conflicts this morning:

#29459 MQTT platform adapter (observational by default) — events log to file, no agent loop per message. Born from a feedback loop that nearly took down my mesh on first flip.

#29460 `hermes mesh` fleet provisioner CLI — codifies the 9-node pattern I've been running.

#30584 fix(aux/vision): respect explicit user-configured model in _try_nous

#41565 fix(desktop): don't auto-open remote-URL link previews

#46351 streaming-content skill — transcribes Twitch/Kick/Rumble/X video into structured JSON via yt-dlp. Added X video support today (authless via yt-dlp guest token, verified end-to-end).

@Teknium @NousResearch

https://t.co/uqL1p2E4de

[Punch Taylor @punchtaylor]

@aiport_trade dug into current main — the frozen fallback's actually a facet of an open per-turn-routing unification (NousResearch/hermes-agent#41190), which already lists failure-recovery as in-scope. added the gateway specifics there rather than a standalone PR. that's the thread to watch.

[Punch Taylor @punchtaylor]

The asymmetric context problem is real. You can strip 80% of tokens and feel great about the savings until you realize you just turned a recoverable error into a silent failure mode.

The agent doesn't know what got cut. It sees a hole and fills it with confidence. That's worse than no optimization at all — at least raw output fails loudly.

Best token optimizers I've seen don't regex-blindfold the context. They preserve error boundaries and stack frames even if it costs more tokens. Debuggability > compression ratio.