I could imagine with int4 quantization, llama.cpp, running batch-size 1, and sub-50k token ctx you could hit those numbers.
But again, that's a bad experience imo - I don't want to quantize it to 1/4th the memory footprint, run one session at a time, & otherwise brick my machine for anything else.
For reference - I get 75 toks/sec on astra/fable tier models, running 10s-100s of parallel sessions, independent of the machine I'm on.
You're right, it'd be an unfair comparison, and unrealistic to expect local models to be on par with models OOMs larger. That's the wrong goal post.
I'd also disagree with the 4.6 comparison, but that's vibes, so who knows.
I don't think it's any good, but yes, for it's size it's great.
It is significantly worse than it's benchmarks would suggest, and running it locally at ~25 tok/sec on my machine is a horrible experience. I'd never want to use it in place of any frontier model.
These small local models do have usecases as utility models, but aside from that, they're neat toys.
@RhysSullivan nah, even if you're running expensive aws bare metal at 2vcpus 8gbs per user you're well below this. like $30/mo. if you're meta it's your own machines anyways which have a much lower amortized cost basis.
its tokens.
@PaulKlayVC well in some sense that's the point dawg.
they may not be right, but if their thesis is the same as everyone else's then they have to compete on dimensions, that, if they're in fund II, they probably lose on.
@Austen ironically he's not completely wrong.
he's basically describing circa '22 days SFT instruction tuning, and is only wrong about the part where it happens at inference time lol.