RogerAI

Founder of RogerAI .fyi ( two-way radio for GPUs
Tune in to the world's
home GPUs.
In under 30 seconds.

[RogerAI @RogerAI_fyi]

Independence Day Frontier Freedom! RogerAI. #localllm #llm #ollama #lmdstudio #vllm #openclaw #hermes Join the Frequency.

[RogerAI @RogerAI_fyi]

vs the quant this box ran all month: +8.7 MMLU at the same speed.

Quant lesson, compressed: shrink downward, never sideways.

Whole story, every number and command:
https://t.co/KZ8fqAMWRn

roger that. ((o))

[RogerAI @RogerAI_fyi]

Every DeepSeek-V4 GGUF on the internet ships with its biggest speed feature ripped out.

The model has a built-in speculative decoder. No file keeps it. llama.cpp can't run it.

So we built the first one that works. Numbers inside.

[RogerAI @RogerAI_fyi]

Every community Q4 GGUF is that lossy sideways re-encode. The ceiling was already on our disk, labeled "intermediate".

So production runs the master now:

88.3 MMLU (community Q4: 85.0, from a file 15 GB bigger)
best tool scores we've recorded
26-27 tok/s

[RogerAI @RogerAI_fyi]

@Raullen 25x TTFT off prefix reuse is the underrated win, everyone chases decode tok/s and forgets prefill is where agent latency actually lives. curious why native MTP's parked, draft accept-rate not landing yet on the 295B?

[RogerAI @RogerAI_fyi]

before you buy more VRAM for context: quantize the KV cache. q8_0 KV in llama.cpp (or fp8 in vLLM) roughly halves it for basically no quality loss. at long context the KV quietly gets bigger than the model itself, so it's the first thing to shrink.

[RogerAI @RogerAI_fyi]

@parthsareen yep, anything that mutates the system prompt each turn (timestamps, a rotating tip, whatever) invalidates the whole prefix and you re-prefill from scratch. put the volatile stuff last or in the user turn and the cache survives. bites everyone once.

[RogerAI @RogerAI_fyi]

Stop asking which model is best. Start asking which node should run it.

The weights are free. The node is the product.

We're building the open marketplace for it, free to try: https://t.co/FAanqy5lll

[RogerAI @RogerAI_fyi]

We ran the exact same open-weight model on two different home rigs.

The answers came out identical. The speed did not: ~172 tok/s on one box, 79 to 93 on the other.

Same model. Same score. Roughly 2x the speed on one machine. That gap is the whole story.

[RogerAI @RogerAI_fyi]

This is the bet behind what we're building. The model layer is commoditizing fast.

The value moves to the layer that matches demand to the right idle node and proves what ran with a signed receipt. Own the routing, not the weights.