@dusangran@ivanfioravanti Maybe, but it is very rare for your initial prompt to have 1m in context, usually you build up to this level of context in multi-turn sessions where the previous turns context is already cached. So realistically you wont wait 16min for first token in any normal use, chat or agent
@ddalcu@RichardSocher@Recursive_SI From your data it seems that Qwen3.8-27B despite being a dense model with 10 times more active parameters it beats Flint on decode speed due to drafting. On evals they are similar, so, for local AI Qwen3.8-27B remains the king
@bnjmn_marie Benchmarking competitive models using a quant version of them and comparing it to your own model without mentioning the use of quants, is a bit unethical, because bad quants exist and each model degrades at a different pace under quantization
@liuliu@zhijianliu@ivanfioravanti Personally, I avoid any quant on KV. I had a few cases where my coding agent made silly mistakes that wouldn’t do with full precision KV.
And tbh it doesn’t worth it anymore to quant the KV cache. New archs have basically solved it (DS4.1 takes ~1GB for 1m context)
@liuliu@zhijianliu@ivanfioravanti I agree. Amazing speed, but a flat 4-bit quant and int8 on KV cache probably botch the output quality of the model.
Looking forward to using Spash with a more sophisticated quants though (hybrid precision for the model and full precision for KV cache)
@Youssofal_ You’re right to stress the quant technique used here. I would prefer not to have the inference engines starting to race each other for speed in the expense of quality.
Also, quantising KV cache is never a good idea in my experience.
@ddalcu I would suggest to remove it if it helps you with the development going forward.
Personally, I avoid KV quant schemes and go for the full precision, as I noticed it severely affects the quality.
Also, new archs seem to have solved the KV cache size (DS4.1 ~1GB for 1m tokens)
@stevibe It’s unlikely a quantitized model, even at Q4, to do better than it’s full precision version (BF16 in this case).
Usually, a higher score demonstrated by a quantitized model compared to its full precision version is margin error.
@ibragim_bad@MoZayed007@Shevan05@agolubev13 Qwen3.5-27B, Qwen3.5-122B-A10B and Nemotron 3 Super are among the most popular open-weight models for coding right now and are missing from the benchmark.
@ivanfioravanti@drrbosconovitch Droid is perhaps the best coding agent harness right now, it’s unfortunate that it’s not open source though.
In some tests of mine, it outperformed opencode and claude code when using the same underlying model.
@ivanfioravanti Cerebras state that REAP is better and they even share bechmarks to back it up, but as we know that can be tricked by having some of the benchmark in their calibration dataset.
Personally, I believe that REAP technique is interesting but I tend to prefer quantization so far.
@ivanfioravanti The question is what has greater penalty in intelligence of a model, REAP or Quantization. Because there is a certain tradeoff between hardware requirements to run a model and intelligence.
@ivanfioravanti Is it a case of benchmaxing? If not, then this model will become very popular. Even tool use seems very high which makes it ideal for light coding agent or running openclaw