Ryan | MAGNVS

Engineered software for the businesses that make, move and maintain the real world.

[Ryan | MAGNVS @magnvsltd]

Everyone shrinks big models to run AI at home.
Everyone also pays for it in quality.📉

And the loss doesn't stay small…

A model writes one word at a time, and every word builds on the last. One slip, and everything after it is built on that slip. Over a long answer, or a long conversation, it adds up 📉📉

That's why people still end up back on cloud APIs for full quality.

I wanted that level of AI on hardware people already own. That's PROMETHEVS.

Think of the compressed model as a master's student 🎓
Brilliant, but slips on the finer details.

So I let the student ask for help:
1️⃣ the student (compressed copy) drafts each word
2️⃣ it flags the words it isn't sure of
3️⃣ only those go to the expert, the full original weights, to be redone

Each slip gets caught as it happens, before the next word builds on it 🎯

I call it Compress-then-Correct.

Matched against the full model, on 960 audited words:
• compressed copy alone: 97.4%
• compress-then-correct: 98.3%

510 GB DeepSeek-V4.1-Flash-1017. One home PC. RTX 3060 12 GB. PROMETHEVS Engine.

[Ryan | MAGNVS @magnvsltd]

@MatthewBerman @jeremyberman 👏🏽 An hour on how these are actually built is a proper gift. Does it get into what quantisation does to a model?

[Ryan | MAGNVS @magnvsltd]

It’s my own engine :) but it retains a focus on quality and speed over just speed.

It runs via quantised 2bit for the speed but utilises a correction method that runs the full weight whenever the quantised model is split between 2 words, in an effort to steer wherever the full weights would side.

Speed wise, 7.4 toks warm on code (11.2 on V0.3 (next release))

Full details below if you’re interested👇🏼

[Ryan | MAGNVS @magnvsltd]

Everyone shrinks big models to run AI at home.
Everyone also pays for it in quality.📉

And the loss doesn't stay small…

A model writes one word at a time, and every word builds on the last. One slip, and everything after it is built on that slip. Over a long answer, or a long conversation, it adds up 📉📉

That's why people still end up back on cloud APIs for full quality.

I wanted that level of AI on hardware people already own. That's PROMETHEVS.

Think of the compressed model as a master's student 🎓
Brilliant, but slips on the finer details.

So I let the student ask for help:
1️⃣ the student (compressed copy) drafts each word
2️⃣ it flags the words it isn't sure of
3️⃣ only those go to the expert, the full original weights, to be redone

Each slip gets caught as it happens, before the next word builds on it 🎯

I call it Compress-then-Correct.

Matched against the full model, on 960 audited words:
• compressed copy alone: 97.4%
• compress-then-correct: 98.3%

510 GB DeepSeek-V4.1-Flash-1017. One home PC. RTX 3060 12 GB. PROMETHEVS Engine.

[Ryan | MAGNVS @magnvsltd]

@c0deNinja @Nikshayyadav1 @dhh Correct, but I feel you’re iterating the same point as me?

Build the AI into a system, that you control - don’t base that system around Claude/gpt - stay in control.

I use Claude, it’s great, but I outright refuse to have a system setup around these in isolation.

[Ryan | MAGNVS @magnvsltd]

@ShushantoK @X 👋🏽 all of the below!

[Ryan | MAGNVS @magnvsltd]

because you’d be putting all of your eggs into one basket that you have no control over.

What if the prices hike and you can no longer afford the subscription?

You would have built your entire infrastructure around the product of someone else..

I think that’s why local compute is so so so important.

[Ryan | MAGNVS @magnvsltd]

Seven hours of agents is a proper stress test for open weights, since every early slip gets built on for the rest of the run.

On my 3060 the 2-bit copy of V4.1 Flash agrees with the full weights on 97.4% of 960 audited words, and over a run that long a small gap has plenty of room to grow. Will you run V4.1 Flash at full weights or quantised?

[Ryan | MAGNVS @magnvsltd]

Tok/s means very little if the run goes round in circles, and 2-4x longer to finish is the number that actually matters. One early slip can be all it takes to start the loop, which is why I care more about fixing the unsure words than chasing speed.

Are you running both at the same quant, or is the faster one squeezed harder?

[Ryan | MAGNVS @magnvsltd]

@ashxhart @MiaAI_lab Enjoy😉

[Ryan | MAGNVS @magnvsltd]

@RezAIT00 @riffydays What’s your workflow? I’m finding the unreal MCP virtually useless. Very stop start.

[Ryan | MAGNVS @magnvsltd]

@Locotir_ Looks great man, I’ll check it out tomorrow💪🏽

[Ryan | MAGNVS @magnvsltd]

I built an engine that treats quality and speed equal, neither has more importance:

Quantised weights get you most of the way. My 2-bit copy picks the same word as the full model 97.4% of the time.

But that gap compounds. Every word builds on the last, so the session drifts further from what the real model would have done as the task, code or agent run goes on.

That’s why I built PROMETHEVS on my compress-then-correct idea 🫡

Simple version: whenever the quantised weights are unsure of the next word, I route it to the full experts, steering the session back to where the full weights would take it. That lifts it to 98.3% on the same 960 audited words.

Runs on my PC (64gb DDR4, 12gb 3060, 2 SSDs) at up to 7.4 tok/s warm on a code prompt.

Not headline speeds (testing on my potato pc😂), but I hope the method can sit on top of the fast inference engines, with compress-then-correct making up for the quality loss.

What would you measure instead of tok/s?

[Ryan | MAGNVS @magnvsltd]

Everyone shrinks big models to run AI at home.
Everyone also pays for it in quality.📉

And the loss doesn't stay small…

A model writes one word at a time, and every word builds on the last. One slip, and everything after it is built on that slip. Over a long answer, or a long conversation, it adds up 📉📉

That's why people still end up back on cloud APIs for full quality.

I wanted that level of AI on hardware people already own. That's PROMETHEVS.

Think of the compressed model as a master's student 🎓
Brilliant, but slips on the finer details.

So I let the student ask for help:
1️⃣ the student (compressed copy) drafts each word
2️⃣ it flags the words it isn't sure of
3️⃣ only those go to the expert, the full original weights, to be redone

Each slip gets caught as it happens, before the next word builds on it 🎯

I call it Compress-then-Correct.

Matched against the full model, on 960 audited words:
• compressed copy alone: 97.4%
• compress-then-correct: 98.3%

510 GB DeepSeek-V4.1-Flash-1017. One home PC. RTX 3060 12 GB. PROMETHEVS Engine.

[Ryan | MAGNVS @magnvsltd]

That snow on the trees is a lovely touch :)

Long one-shots like this are where small slips stack up. What tripped it up on the first few tries, losing the thread halfway or broken code from the start?

Are you running full weight or quantised?

Compress-then-correct should help eradicate those slips if quantised🫡

[Ryan | MAGNVS @magnvsltd]

@MiaAI_lab Blank replies sending agents round in circles is such a sneaky one. Keeping broken calls as plain text is the right fix. Were most of the strays coming from the no-tools turns?

[Ryan | MAGNVS @magnvsltd]

@WinstonLBrown1 @ashxhart Fair question. Hashes only prove you’ve got exactly what DeepSeek published, not that what they published is clean.

That’s a trust call with any open model🤷🏼‍♂️

[Ryan | MAGNVS @magnvsltd]

@xenomayo @0xSero Hahaha, I think we’ve all got regrets like this.

Still, a 16gb upgrade is way easier than 64gb!😉😂

[Ryan | MAGNVS @magnvsltd]

The weights and the stores built from them sit on two ordinary NVMe drives, about 1.1 TB. Every word runs on a 2-bit copy of the experts, and only the ones it’s unsure of are recomputed from the original weights, so it matches the full model on 98.3% of 960 audited words. Up to 7.4 tok/s warm on a code prompt, and yes, it runs as an OpenAI-compatible server on the same PC.

More info below if you’re interested 🫡

[Ryan | MAGNVS @magnvsltd]

Everyone shrinks big models to run AI at home.
Everyone also pays for it in quality.📉

And the loss doesn't stay small…

A model writes one word at a time, and every word builds on the last. One slip, and everything after it is built on that slip. Over a long answer, or a long conversation, it adds up 📉📉

That's why people still end up back on cloud APIs for full quality.

I wanted that level of AI on hardware people already own. That's PROMETHEVS.

Think of the compressed model as a master's student 🎓
Brilliant, but slips on the finer details.

So I let the student ask for help:
1️⃣ the student (compressed copy) drafts each word
2️⃣ it flags the words it isn't sure of
3️⃣ only those go to the expert, the full original weights, to be redone

Each slip gets caught as it happens, before the next word builds on it 🎯

I call it Compress-then-Correct.

Matched against the full model, on 960 audited words:
• compressed copy alone: 97.4%
• compress-then-correct: 98.3%

510 GB DeepSeek-V4.1-Flash-1017. One home PC. RTX 3060 12 GB. PROMETHEVS Engine.