@witcheer You’re missing out by not using Strata
You could hit 100+tps on Qwen3.8 Flash Next and 10+tps on DeepSeek V4.1 Flash 1017 (estimates)
Flash Next is much better than 27b as well
@Tech2Wild No.
Qwen3.8 Flash Next will, because of Srata
Seriously, hats off to @coldniko, 70tps+ on hardware under $3k with models less than 6 months behind the frontier is unreal
It’s because large language models are trained on the average of all ideas. When prompted correctly, especially with Claude models, you can get them to come up with new things, but it’s difficult
The AI he’s describing is something that is not an LLM, but we already have all of the technology to run it
@scaling01 If it’s faster and cheaper per token I don’t really mind the token consumption, it still ends up faster and cheaper overall
Even if the official provider is slow there are others (baseten) with 5x the speed of sol per token, it balances out
And as you mentioned, swarms