Fargo

[KISA aka Copenzafan.eth @copenzafan]

Got my local models running 4x faster

I wrote a framework that gives an agent the skills and docs it needs to deploy local models on your machine.

But here's the cool part: these are standalone models that get plugged straight into a convenient harness, from Pi to Hermes.

- Picks a model that fits your hardware
- Chooses the best configuration for MoE, Apex, and the inference engine
- Tunes model weights and hardware load distribution for your setup
- Includes MTP, which can add another 3–4 tokens per second

The result is around 15 tokens per second on average for lightweight models with roughly 30 billion parameters that can handle full agent workflows.

ChatGPT Astra runs at 54 t/s, but for short replies, 10–20 tokens per second means roughly 1–2 minutes per response.