El Deffo

[El Deffo @eldeffo]

@bnjmn_marie @WaleedAhmad1a10 any plans to do larger models (122b mtp, stepfun 3.7, v4 flash, 397b nex, mimo etc.) at low (iq1-q3) quants?

would be nice to be able to squeeze more intelligence into commodity hardware and have it be usable still.

[El Deffo @eldeffo]

@0xSero REAP is idiotic. replicate SlimQwen instead.

[El Deffo @eldeffo]

@RyanNg26101 @Montrey82631182 donbas "locals" only joined after russia already occupied crimea and RU started the same shit there. video clearly shows well trained, professional soldiers in matching uniforms.

also, finding 1000 local bandits in a region with millions of people, doesn't justify an occupation.

[El Deffo @eldeffo]

@HououinTyouma @0xSero it's been done and it works (e.g. ExpertFlow)

[El Deffo @eldeffo]

@0xSero sorry, ExpertFlow predictor paper: https://t.co/R6djY0fUuc

[El Deffo @eldeffo]

@0xSero read the FlashMoE predictor paper.

but it's possible to just use statistics, e.g. https://t.co/ra8acHJGZE

[El Deffo @eldeffo]

@0xSero well, ram+cpu can be probably be just ranked as well, as it's possible for them to be faster than gpus for inference

[El Deffo @eldeffo]

@0xSero it would be much easier/better to figure out layered caching instead:
L1 - GPUs, ranked separately both compute for prefill & vram speed for generation
L2 - ram+cpu
L3 - ssd
there has been quite a bit of research & implementations too, but vllm and llama wouldn't let the PRs in

[El Deffo @eldeffo]

@bnjmn_marie there's some talk about broken chat templates in 3.6 on hf

[El Deffo @eldeffo]

@leftcurvedev_ @JamesNumb3rs connect your display to motherboard iGPU, you'll save yourself 1-3GB of VRAM and will be able to use Q4

[El Deffo @eldeffo]

@Posledniskaut v slove kokot tam mas nejak privela preklepov.

[El Deffo @eldeffo]

@mudler_it you should compare UD_Q4_K_M instead

[El Deffo @eldeffo]

@bnjmn_marie even with -np X and perhaps several server copies at the same time?

maybe it would be nice to have a proper comparison of the engines too.

[El Deffo @eldeffo]

@bnjmn_marie generally the most interesting question is what can you fit into 11GB, 15GB, 23GB, 31GB... past that, it's just macs and rtx pros, and those can run almost anything anyway.

[El Deffo @eldeffo]

@bnjmn_marie how so? llama with quants is consistently faster than vllm, at least every time I tried.

also, maybe the battery could be reduced, small models - Q4_K_M, maybe IQ4_NL & IQ3_XSS, + some smarter Q2s on 200B+ models?

those are probably the only ones that need to be tested really

[El Deffo @eldeffo]

@LenSeaside @0xSero you can fit 27B UD-IQ3_XSS into 12GB, but only if you connect the display to motherboard.

[El Deffo @eldeffo]

@LenSeaside @stevibe 27B UD-IQ3_XXS [-ngl 65 + 36K Q4 kv cache] 1100-1200pp 36-37 t/s

but only if you connect the display to motherboard/CPU's iGPU, that will get you 1-3GB VRAM back.

[El Deffo @eldeffo]

@0xSero btw. https://t.co/ra8acHJGZE

[El Deffo @eldeffo]

@0xSero if llama had api to move layers in and out of VRAM, with this info the performance gains could be quite substantial