That’s astute observation. To make bench comparison fair, I have both 27B and V41F set to 131k context per stream. 27B have zero issues working with that, but V41F simply will not commit to saying finished. When set context to 300K+, zero issues, which adds another headache to benching these things
Multiple thinking modes are bane of my existence, makes benching a complete nightmare.
To give you an idea, designed a new bug suite hard enough to challenge V41F class models. V41F used x3 context as 27B did, compacted multiple times, and each time loses track and goes into validation loop. Eventually it’ll score 23/24 compared to 27B’s 20/24
So which model is better? Score says V41F, but it took 20min longer, used x3 context, and nearly hit 520 tool call limit multiple times. That’s just one thinking modes, and another mode behaves completely different.
There are things bench scores simply do not capture, and for someone who wants to find the most optimal serving config, it’s almost a losing battle when multiple thinking modes are involved and hope they stop fucking around and stick to on/off for my sanity
It’s actually driving me quite mad. All I want to do is run a bench that I trust and built, get a score, and publish it but no, that’ll be too easy.
Everytime, there’s some fuckery with reasoning that makes me wonder if results are even comparable. Lazy person would just publish the score but I literally can’t do that when I see what it took a model to get there and just doesn’t sit right not putting that into consideration
@someRandomDev5 That’s assuming it finished the task. More often than not, it loops until it fails or give unfinished results. I had to keep raising my tool call limit to insane 520 just to see if it’ll actually tell me it finished
Like who is it even for? How many end users have the time and inclination to switch thinking modes to see what works for them? I’m certain 99% times when someone complains about a model, it’s set to xhigh or off by default, both suck, and conclude model is worthless when moving down to low/medium would’ve fixed tic
@fullctx It has to stop, plus the highest thinking modes are almost always completely broken compared to medium in too many tasks. Not once I’ve benched a model in real use cases and got more optimal results, don’t even understand why it’s there
When you hand a model long tasks, what makes you stop trusting it? Not benchmark scores, but a moment where you decide you have to check everything it does
@MuffMuffington For this experiment, it’s just four small pieces.
-Frozen question sets
- A runner that asks every questions exactly same way and writes one row per answer
-A scheduler that launches fresh server per job
- Analysis script for grading the rows based on written rules
191 million tokens burned for document retrieval experiment. Compared 17 small 2-10B models against Q4 27B in screening, reranking, extraction and depth.
Full results once I do my due diligence with all this data, but can already tell you certain 2B and 9B model outperforms 27B on some categories easily
@JunJun0321A Plan is to provide a combination of small models that can either match or outperform 27B in each category, using significantly less VRAM if loaded one at a time per stage. Current have 3 model combo that looks promising but I’ll share the full results when I can
I don’t want to get ahead of myself since there’s so much data to analyze, plus there are 5 different types of documents that impacts scored as well.
Reranking and screening is where certain 2B models are showing remarkable scores. Extraction, no small model out right beats 27B but gets pretty damn close using 1/3 of vram. Depth is more complicated so I’ll hold judgement until later
Even with 1TB VRAM at my disposal, I’m constantly looking for ways to use smallest model at lowest quant for any given work. Realized running a swarm suitable for a job beats running few huge model, even decode is irrelevant after certain point if it compromises quality and results.
It definitely requires more effort to lock it in, but highly recommend testing different models/quants for your specific use cases. You can end up with much more efficient, versatile fleets on limited VRAM
@MuffMuffington I’ll keep that in mind. Not sure how you do it, but I tried going through Nvidia -> partner/resellers and they couldn’t help at all during my search. Looks like supply relaxed a bit now so things might be different