GooGZ AI

AI research | 30 years in tech & business | Agent tinkerer | Hermes Agent | Ex-dev turned AI dad | Latest initiative:

[GooGZ AI @PaulGug]

Sharing my latest initiative - the Nerfwatch Index

What is Nerfwatch Index?

Like many, I have often questioned whether models from the big AI labs are dialed back (i.e. 'nerfed') over time. I presume this might be to practically manage ongoing model delivery commitments within finite compute. Maybe it's just to reduce costs over time? Either way- it's challenging to prove, but we sometimes just sense things 'back off a bit' as we use them.

I'm contributing some of my subscription capacity ongoing for a while to test for this phenomena (started just before Sonnet 5.5 launched), and will see whether it is indeed true or not.

How does it work?

Nerfwatch Index tracks whether closed AI models maintain their original performance over time. Twice daily, models face the same 20 tests, with the outputs judged against strict quality standards (measurable / empirical), despite challenging constraints and conditions.

Link: https://t.co/lTgdHTJpdC

Do you believe the big labs nerf their models over time?

[GooGZ AI @PaulGug]

@LightchainAI @liquidai @OpenBMB @SparkLLM @PrismML @NVIDIAAI I know right. Something that works like they do would have been inconcievable. I have a Macbook M2, which wasn't built for this stuff yet runs these models great. Can you think of any others that I'm missing? Always looking for recommendations..

[GooGZ AI @PaulGug]

The most exciting ‘all rounder’ small model releases in recent weeks:

@liquidai LFM3 1.7B
@OpenBMB MiniCPM5 2B
@SparkLLM Spark X2.6 1.7B
@PrismML Bonsai 2 2B/4B
@NVIDIAAI Nemotron 3.5 Bolt 4B

These run on just about anything.

You have no excuse - build something amazing that packs local AI!

Any small-model wonders that I’m missing?

[GooGZ AI @PaulGug]

Thanks, it's been the 9B to beat on my kit.

Where I'd love to see it grow: holding a lot of moving parts across a long multi-step task, and re-planning cleanly when something changes midway. On well-defined tasks like scheduling and drafting, it's excellent.

A bigger size would be very welcome too 👀 .. I think 'the best that fits on consumer grade GPUs' are very interesting (think <12GB VRAM). MoEs are also compelling, as are ternary weights.

Keep us posted! Looking forward to your next releases 🙏

[GooGZ AI @PaulGug]

Tested Mythos-V2-9B, the uncensored Qwen3.5-9B fine-tune, on my personal-agent suite (RTX 3080 10GB, Q4_K_M, reasoning off).

The thinking is there. A1 (light agentic) scored 9/10 on every run, ahead of NeoHorse 1 9B's 8.3.

Tool calls are where it slips: T1: 27/30 (NeoHorse and Ornith 1.5 both 30/30)

Poor tool calling leads to further issues:
- PA1 failed all 3 runs on malformed tool-call JSON
- PA2: 88 (vs NeoHorse 92, Ornith 88.4)
- PA4: 46.3 avg. Ahead of Ornith but behind NeoHorse.

It's built for storytelling and NPCs, and I'd happily use it there. For an agent that runs your day, NeoHorse is the 9B to beat right now.

Got a new version of NeoHorse coming soon @jarvis_pei ?

[GooGZ AI @PaulGug]

@LOFI911 You tried it?

[GooGZ AI @PaulGug]

@LOFI911 Interesting! No I was not familiar thanks for the recommendation. I'll check it out for sure.😎

[GooGZ AI @PaulGug]

@LeoHarttx you mean it has to be an uncensored model? or any 9B? Ornith 1.5 9B or NeoHorse probably my picks.. and you can find albiterated versions

[GooGZ AI @PaulGug]

@mark_k @bot Grok bot is awesome

[GooGZ AI @PaulGug]

@mark_k Who is the final ‘distillee’ distilling however? Each other ?

[GooGZ AI @PaulGug]

@IAmNickDodson @liquidai @OpenBMB @SparkLLM @PrismML @NVIDIAAI 100% I'm super excited about the potential of ternary models. Hoping PrismML work on a few more variants before too long. Bonsai 2 27B is amazing for the size.

And as someone who has a day job and a family and then dabbles with this on the side when possible - i hear you :)

[GooGZ AI @PaulGug]

@IAmNickDodson @liquidai @OpenBMB @SparkLLM @PrismML @NVIDIAAI Spark is an interesting one - I posted about it a couple of times in recent days (one example below) - they all have they strengths and weaknesses. You are right that Qwen has been a surprising stayer - so many months after its release. It's crazy. https://t.co/EiWubqynGn

[GooGZ AI @PaulGug]

Small models are going through a real growth spurt right now.

On my personal-agent tests (PA Core, reasoning on, RTX 3080), three models at 2B and under now land within 21 points of each other, and rubbing shoulders with older 4B/9B weights.

Newest on the bench: Spark X2.6 1.7B from @SparkLLM

- PA total: 172.3/300
- Tool calls: 30/30 in every run
- Context recall: 9.3 at 8K, 8.7 at 16K, best of the three!
- Schedule and draft (PA2): 98 and 100 in two runs, one 54 pulled the mean to 84
- Daily organization (PA1): 35.7, the weak spot

Under 2B, three different strengths. @liquidai LFM3 plans, @SparkLLM Spark reads, @OpenBMB MiniCPM5 is the balanced one.

Leave thinking on. With it off, Spark is a much weaker model on these tasks.

[GooGZ AI @PaulGug]

@IAmNickDodson @liquidai @OpenBMB @SparkLLM @PrismML @NVIDIAAI It is Nick! What's your pick (or can you think of another)?

[GooGZ AI @PaulGug]

@itsKamadh They can be right then, and right now I think - just comes down to how capable the tech is. It's funny though when you look back at Jobs crapping on the idea :)

[GooGZ AI @PaulGug]

@elshayib_ It continues on but a quick check of the Grok says it has to be like this for GDPR anyway:

[GooGZ AI @PaulGug]

@Greg_GL_87 @liquidai @mlech26l @Franzferdinan57 Yeah I know amazing right?

[GooGZ AI @PaulGug]

LFM3-1.7B from @liquidai on my personal-agent suite (RTX3080), reasoning off.

PA Core Compact /300:
- Gemma 4 12B: 234.7
- NeoHorse 1 4B: 172.7
- LFM3-1.7B: 164.3
- MiniCPM5 2B: 164.0
- Qwen 3.5 2B: 159.7
- Qwen 3.5 4B: 138.4

At 1.7B it's level with MiniCPM5 as the best 2B I've tested, ahead of Qwen 3.5 4B and right behind NeoHorse 4B.

Tool use was a perfect 30/30 every run. Schedule and draft (PA2) is its standout at 78.3.

Where it isn't there yet: it didn't finish a change-of-plan loop (PA4, 0/3), and it drifted on scope in A1 (7/10).

So it's not a 12B replacement for my use case yet, but it's excellent for its size. One day I hope a 1B can do what Gemma 4 12B does on these tests, and releases like this keep closing the gap.

Really nice work from the @liquidai team!

[GooGZ AI @PaulGug]

LFM3 just landed from @liquidai, and the 1.7B text model is going straight onto my personal agent test bench!

256K context, about 1GB at 4-bit, and pitched for agent work. Strong claims at this size.

Can it handle a real personal-agent loop (clean tool calls, recalling context, planning a day, and adapting when the plan changes)?

Running my PA Core suite on the 10GB RTX 3080 and 16GB M2 Air. Results soon.

[GooGZ AI @PaulGug]

@EphemeralTao @liquidai @mlech26l @Franzferdinan57 The perfect tool protocol is a real standout at this size

[GooGZ AI @PaulGug]

@datachad @liquidai @OpenBMB @SparkLLM @PrismML @NVIDIAAI I know what you mean .. but they are all rounders compared to, for example, VL or OCR models.

[GooGZ AI @PaulGug]

@RightSideOfAI @NeoAIForecast @matt_teeixeira heads up!

[GooGZ AI @PaulGug]

@ViC305 @GoogleDeepMind I'd like to see them distill these down to a new Gemma 5 series !