Abdarahmane Traore

Multimodal LLM researcher. Small models, big ideas (SmolRGPT, SCOUT)
Nights: making Qwen fast on 4x RTX 5090, 203 tok/s
Receipts:

[Abdarahmane Traore @abtraore_]

Also : Field note from a 92k-token agent session (pi): acceptance fell to ~0.3 there and the fixed 7-token window drafted 2.6x more than it landed. dynamic_draft_tokens=True with dynamic_draft_skip_ema=0 restored 0.65+ at the same shallow speed. The 0.3 skip default parks easy jobs at plain-decode speed, might be worth a look.

[Abdarahmane Traore @abtraore_]

Confirmed on a 5090: first run bootstrapped itself, compiled the sm120 kernels and pulled the weights unattended in under 25 min, then 193 tok/s decode on code (dflash2 acceptance 0.70) with the full native 262k resident in 25.4 GiB. Prefill 2,244 tok/s at 16k. The RTX bandwidth scaling your README predicted holds. Did fork your repo tho: https://t.co/7ZAfD8uTgr

[Abdarahmane Traore @abtraore_]

@sfxnz Got it working on x6 5090 at 125 tokens/s with lama.cpp did some patching tho: https://t.co/NkCmyuwVX7

[Abdarahmane Traore @abtraore_]

@alexandr_wang Muse 1.2 is so fast that it rate limit muse code... AGI coming soon lol

[Abdarahmane Traore @abtraore_]

@mr_r0b0t Got it working on x6 5090 at 125 tokens/s... Now visual tasks are viable : https://t.co/NkCmyuwVX7

[Abdarahmane Traore @abtraore_]

@ModelScope2022 Made it work day 1 on x6 5090 at 125 tokens/s with llama.cpp (I did some patching tho): https://t.co/NkCmyuwVX7

[Abdarahmane Traore @abtraore_]

@sgl_project @deepseek_ai Working on x6 5090 at 125 tokens/s on llama.cpp (I did some patching tho): https://t.co/NkCmyuwVX7 , pretty good model 🚀

[Abdarahmane Traore @abtraore_]

@victormustar Got it working on x6 5090 at 125 tokens/s on llama.cpp (I did some patching tho): https://t.co/NkCmyuwVX7

[Abdarahmane Traore @abtraore_]

2x from tuning alone is the real story with this model, the kernels are young: rebuilding llama.cpp at the merged PR head nearly doubled decode for us too. Next big one for your card: MTP spec decode, once GGUFs get re-exported with the head (PR #27836). Recipes: https://t.co/WhUQRzPcVa

[Abdarahmane Traore @abtraore_]

@malikwas1f 18 on GLM and 37 on Flash-Next from 2x 3090 is solid, that lines up with the host-bandwidth math almost exactly. How much does the hot-expert cache buy over naive offload? From the all-GPU end: Flash-Next at 203-211 tok/s on 4x 5090, vLLM + MTP: https://t.co/UNZK4gLMJr

[Abdarahmane Traore @abtraore_]

53 on one 5090 with everything exact is impressive. Which stack, llama.cpp with expert offload to RAM? Asking because on 4x 5090 the vLLM path with the built-in MTP head holds 203-211 tok/s flat to 186K, and I'm curious how far the single-card ceiling goes: https://t.co/WhUQRzPcVa

[Abdarahmane Traore @abtraore_]

@analogalok Other end of the same curve: 4x 5090, vLLM patched nightly, MTP n=3 + fp8 KV for QSA. 203 tok/s decode, flat to 186K ctx (MTP acceptance ~0.9 on code), prefill 2,690 tok/s at 99K. Exact flags and patches: https://t.co/UNZK4gLMJr

[Abdarahmane Traore @abtraore_]

Nice numbers. Cousin datapoint: 4x 5090, TP4 + expert parallel, RadixArk NVFP4, MTP n=3: 203-211 tok/s on code, flat from 99K to 186K. One card-count note for PCIe rigs: prefill prefers fewer cards. On Qwen3.8-27B we measured TP=2 at 4,299 tok/s prefill vs TP=4 at 1,315. Recipes: https://t.co/WhUQRzPKKI

[Abdarahmane Traore @abtraore_]

@vllm_project @Alibaba_Qwen Verified on 4x RTX 5090: 203 tok/s decode with MTP n=3, flat to 186K context: https://t.co/1Gr1pZVA13

[Abdarahmane Traore @abtraore_]

@loktar00 it increased In Canada by +1000 lol

[Abdarahmane Traore @abtraore_]

@elonmusk Can you please reset the usage so we can test it ? lol

[Abdarahmane Traore @abtraore_]

I ran out of ctx the first time... The model card says 130k~ ctx but I was able to interpolate in llama.cpp:

--ctx-size 524288 --override-kv muse-glimmer.context_length=int:524288 --rope-scaling yarn --rope-scale 4 --yarn-orig-ctx 131072

Needle test was positive... But need real world usage to confirm if it is stable.

[Abdarahmane Traore @abtraore_]

@alexandr_wang Nice release ! I tried Glimmer but the KV is just too small, the architecture enable cheap KV but we are limited to 130k~

[TIME @TIME]

"Commoditize the petaflop." @__tinygrad__ founder George Hotz says more competition—not monopolies—is the path to cheaper AI compute at @AMD's Advancing AI event.

[Abdarahmane Traore @abtraore_]

@elliotarledge why you didn't go for the SXM... Did you weight speed/price ratio ?