ہزاروں خواہشیں ایسی کہ ہر خواہش پہ دم نکلے،
بہت نکلے مرے ارمان لیکن پھر بھی کم نکلے۔
محبت میں نہیں ہے فرق جینے اور مرنے کا،
اسی کو دیکھ کر جیتے ہیں جس پر کافر دم نکلے۔
♥️✨
The efficiency bar just moved again. IFM says K2-Horizon-36B-A4B is matching models that are over 20× larger, and AA ranks it #4 out of 142 comparable models, while only activating around 4B parameters per token.
Worth being precise here: 36B total parameters, 4B active, 25 on the AA index, and #4 out of 142 in its class. It’s not #1, but the interesting part is getting that kind of result with only 4B active parameters.
If you were testing a model like this, what would you look at first—the benchmark results, how the KV cache holds up with long contexts, or whether the training-stability claims actually check out?
At 1.1M steps, it’s only about 1.2–3.7 points behind the 32B dense model on MMLU, GSM8K, HellaSwag, and HumanEval. Dense still comes out ahead, but the real question is whether that small gap is worth using 8× more active compute
MoE has been living in the FFN for years. MoVA basically takes that same idea and applies it to the value vectors in attention. Same concept, just a different part of the model.
Cutting gradient clipping by 50% might not sound like a big deal, but it can be the difference between leaving a training run overnight and having to babysit it.
The FFN has been the main place we’ve been scaling sparsity for a while. Was there a specific reason attention was left alone, or was it just harder to make it work there?
It’s a different way to think about sparsity: instead of adding more experts to the FFN, you’re putting the experts into the value computation. I’m curious to see how far this approach can actually go.