Jeff Geiser 🛡️

Four kids. Aggie. Dream about the mountains. Work with a great team at Zenlayer. edge inference, custom models.. wicklee, ARP, WES..

[Jeff Geiser 🛡️ @jgeiser]

ran first LoRA fine tune on my account intelligence model.. 519 training examples and loss dropped and initially looked good but it broke

essentially, every gradient update was teaching the model to predict the system prompt.. not account briefs

good lord..

round 2 tonight..

[Jeff Geiser 🛡️ @jgeiser]

still tweeking the model fit section of https://t.co/8oQyB5mOYQ

[Jeff Geiser 🛡️ @jgeiser]

10 miler in Connecticut last weekend.. great route - Hop River Trail

[Jeff Geiser 🛡️ @jgeiser]

Open sourcing the ELM lab I've been building — dataset generation pipeline, schema-first design, eval harness for account intelligence synthesis. First model in progress.

https://t.co/qEiziG0Dtm

[Jeff Geiser 🛡️ @jgeiser]

So, range is probably between Q4 to Q6.. anything below Q4 you get too much quality degradation..

above Q6 you start to pay a performance penalty and don't gain much in quality..

[Jeff Geiser 🛡️ @jgeiser]

Building a fine tuned account intelligence model and wanted to find out which llama3.1 8B is best on a mac mini (16GB).. I think this is a good reference architecture overall. Built a benchmark harness and ran it through all 11 quants.

Short version is Q4_K_M was clear winner..

[Jeff Geiser 🛡️ @jgeiser]

With perplexity:

Quant | PPL | vs Q8_0
Q2_K | 11.15 | +29% worse
Q3_K_M | 9.21 | +6.4% worse
Q4_K_M | 8.80 | +1.7% worse ← sweet spot
Q5_K_M | 8.73 | +0.9% worse
Q6_K | 8.68 | +0.4% worse ← ceiling
Q8_0 | 8.66 | baseline

You get 98.3% of Q8_0 quality with Q4_K_M

[Jeff Geiser 🛡️ @jgeiser]

snuck one in before the rain hit

[Jeff Geiser 🛡️ @jgeiser]

running some evals on the mac mini

[Jeff Geiser 🛡️ @jgeiser]

ok, replaced openclaw with claude but the imessage plug in does not work with Tahoe 26..

Looks like the bug is already submitted on github, but until then you can bypass with a python/bash script

Monitored chat.db for a message, pipe it to claude and have claude use osascript for final response.. key is keeping it in a loop..

It's sweet.. i am currently using it for huge productivity gains

[Jeff Geiser 🛡️ @jgeiser]

The tetris of fitting a 32B model and BGE-M3 embeddings on the same spark....

Hardware squeeze is the biggest hurdle for local AI.. hell, maybe all of ai..

[Jeff Geiser 🛡️ @jgeiser]

Now we will do a full eval with those changes above so we can assess this.. I will update as things progress..

I think we are going to be able to close the gap on routine synthesis and just use claude for the hard stuff.

[Jeff Geiser 🛡️ @jgeiser]

We have a dgx spark at the office and are using it for some automation - peers into salesforce, confluence, internal apis to generate workflows and data in an internal ui -- goal was to keep the sensitive data on the local llm and, of course, avoid some api costs..

Some of what we learned:

[Jeff Geiser 🛡️ @jgeiser]

We also added an embedding model (BGE-M3) to handle some semantic search plus some multilingual support.. much needed, but makes it even tighter on the vram