GooGZ AI

AI research | 30 years in tech & business | Agent tinkerer | Hermes Agent | Ex-dev turned AI dad | Latest initiative:

[GooGZ AI @PaulGug]

Sharing my latest initiative - the Nerfwatch Index

What is Nerfwatch Index?

Like many, I have often questioned whether models from the big AI labs are dialed back (i.e. 'nerfed') over time. I presume this might be to practically manage ongoing model delivery commitments within finite compute. Maybe it's just to reduce costs over time? Either way- it's challenging to prove, but we sometimes just sense things 'back off a bit' as we use them.

I'm contributing some of my subscription capacity ongoing for a while to test for this phenomena (started just before Sonnet 5.5 launched), and will see whether it is indeed true or not.

How does it work?

Nerfwatch Index tracks whether closed AI models maintain their original performance over time. Twice daily, models face the same 20 tests, with the outputs judged against strict quality standards (measurable / empirical), despite challenging constraints and conditions.

Link: https://t.co/lTgdHTJpdC

Do you believe the big labs nerf their models over time?

[GooGZ AI @PaulGug]

@Greg_GL_87 @liquidai @mlech26l @Franzferdinan57 Yeah I know amazing right?

[GooGZ AI @PaulGug]

LFM3-1.7B from @liquidai on my personal-agent suite (RTX3080), reasoning off.

PA Core Compact /300:
- Gemma 4 12B: 234.7
- NeoHorse 1 4B: 172.7
- LFM3-1.7B: 164.3
- MiniCPM5 2B: 164.0
- Qwen 3.5 2B: 159.7
- Qwen 3.5 4B: 138.4

At 1.7B it's level with MiniCPM5 as the best 2B I've tested, ahead of Qwen 3.5 4B and right behind NeoHorse 4B.

Tool use was a perfect 30/30 every run. Schedule and draft (PA2) is its standout at 78.3.

Where it isn't there yet: it didn't finish a change-of-plan loop (PA4, 0/3), and it drifted on scope in A1 (7/10).

So it's not a 12B replacement for my use case yet, but it's excellent for its size. One day I hope a 1B can do what Gemma 4 12B does on these tests, and releases like this keep closing the gap.

Really nice work from the @liquidai team!

[GooGZ AI @PaulGug]

LFM3 just landed from @liquidai, and the 1.7B text model is going straight onto my personal agent test bench!

256K context, about 1GB at 4-bit, and pitched for agent work. Strong claims at this size.

Can it handle a real personal-agent loop (clean tool calls, recalling context, planning a day, and adapting when the plan changes)?

Running my PA Core suite on the 10GB RTX 3080 and 16GB M2 Air. Results soon.

[GooGZ AI @PaulGug]

@EphemeralTao @liquidai @mlech26l @Franzferdinan57 The perfect tool protocol is a real standout at this size

[GooGZ AI @PaulGug]

@datachad @liquidai @OpenBMB @SparkLLM @PrismML @NVIDIAAI I know what you mean .. but they are all rounders compared to, for example, VL or OCR models.

[GooGZ AI @PaulGug]

The most exciting ‘all rounder’ small model releases in recent weeks:

@liquidai LFM3 1.7B
@OpenBMB MiniCPM5 2B
@SparkLLM Spark X2.6 1.7B
@PrismML Bonsai 2 2B/4B
@NVIDIAAI Nemotron 3.5 Bolt 4B

These run on just about anything.

You have no excuse - build something amazing that packs local AI!

Any small-model wonders that I’m missing?

[GooGZ AI @PaulGug]

@RightSideOfAI @NeoAIForecast @matt_teeixeira heads up!

[GooGZ AI @PaulGug]

@ViC305 @GoogleDeepMind I'd like to see them distill these down to a new Gemma 5 series !

[GooGZ AI @PaulGug]

@HermesAgentTips 94% off is crazy good! Works so well with Hermes Agent too.

[GooGZ AI @PaulGug]

@DevinSoto @bot Yeah pretty much.. for me anyway

[GooGZ AI @PaulGug]

@DevinSoto @bot It would be enough for me to drop codex and consolidate around grok / grok build

[GooGZ AI @PaulGug]

@kimmonismus Nice. All the regional restrictions are a shame- even today’s phone text messaging ability being limited to US as well. Unfortunate.

[GooGZ AI @PaulGug]

@Franzferdinan57 35B MoE*

[GooGZ AI @PaulGug]

@Franzferdinan57 I get my agent to help with my tests on the macbook - one helps with llama (or forked llama) config, the other runs and saves test results. I want to use a 9B for the tests, and use 25MoE (or frontier) for coordinating the inference setup and helping interpret results.

[GooGZ AI @PaulGug]

@OpenBMB Cheers. It’s a lot of fun to do it. 😊

I’d imagine a build/training collab between your three labs would unleash a beast of a model. All have their strengths - would love to see them come together! Easier said than done I’m sure…

[GooGZ AI @PaulGug]

@BuiltByStephen @SparkLLM @Tylerwtzhang Yeah anything technical like that - I would still steer clear - anything under your 27B dense or 35B MoE types. I'm sure there are some that can but I personally wouldnt do that with a 1.7B model - not yet anyway.

[GooGZ AI @PaulGug]

This is why I get excited with each new small model increasing in capability. Becoming super capable for everyday tasks and, on my old RTX3080, @SparkLLM Spark X2.6 1.7B absolutely screams! Instant responses..

Nice work @Tylerwtzhang and team!

[ModelScope @ModelScope2022]

Spark-X2.6-1.7B has landed: an open-source 1.7B model built for capable local agents.@SparkLLM
🤖 https://t.co/jdNjiU77rL

🧠 Think only as much as the task needs: Switch between no-thinking, low-thinking, and high-thinking modes to balance answer quality, latency, and inference cost.
🛠️ Built for everyday agent work: Designed for tool use, multi-step planning, coding, and reasoning workflows in a compact local model.
⚡ Made for constrained hardware: The 1.7B scale makes it a practical starting point for private, responsive, resource-conscious local deployment.
🔧 Ready for the local AI stack: Use it with supported runtimes and quantized formats to fit different hardware and serving setups.
📦 Open to adapt: Start with the chat model for agent workflows, or use the Base checkpoint for continued training and domain customization.
📜 Apache-2.0 License.

[GooGZ AI @PaulGug]

@BuiltByStephen @SparkLLM @Tylerwtzhang Correct! it All depends on what you use it for. Coding is a higher bar but if you need to sort data in a certain order, or to read/interpret an image, a 1B or 2B is more than enough. What is the minimum you work with for it to be 'useful enough'?

[GooGZ AI @PaulGug]

@kimmonismus Easier to distill rather than ‘read the entire internet’ again I guess…

[GooGZ AI @PaulGug]

@CertainLogicAI Indeed- but they are still bit-part with the r&d (from what I can tell at least). The bigger weights are more alluring and promotable

[GooGZ AI @PaulGug]

Small models are going through a real growth spurt right now.

On my personal-agent tests (PA Core, reasoning on, RTX 3080), three models at 2B and under now land within 21 points of each other, and rubbing shoulders with older 4B/9B weights.

Newest on the bench: Spark X2.6 1.7B from @SparkLLM

- PA total: 172.3/300
- Tool calls: 30/30 in every run
- Context recall: 9.3 at 8K, 8.7 at 16K, best of the three!
- Schedule and draft (PA2): 98 and 100 in two runs, one 54 pulled the mean to 84
- Daily organization (PA1): 35.7, the weak spot

Under 2B, three different strengths. @liquidai LFM3 plans, @SparkLLM Spark reads, @OpenBMB MiniCPM5 is the balanced one.

Leave thinking on. With it off, Spark is a much weaker model on these tasks.

[GooGZ AI @PaulGug]

@CertainLogicAI When they get focused on.. great things happen