alex @robloxdemon69 - Twitter Profile

im writing my own inference engine. what i find funny is that big project like vllm and others most used inference engine don't support what retail can actually afford to use in 2026. sm_75 is still supported by cuda 13.x+ and these software cant even handle more than 6.6 token/s on most model. 8x 2080ti gets you 88gb vram, faster compute than a single gb10 and you can load very good model. from my testing, current inference engine leave ~4-6x performance on the table for old hardware and even current gen hardware. will share progress soon. my mvp will support fp16, bf16, fp8, fp4 a bunch of model (text, code, no image inference) including qwen3, deepseek and more. each model require specific configurations and kernel and it need to be optimized for every single architecture so its taking a bit of time. anyways, "shitware" is never the target of the people that get the funding and the funds to optimize for every single hardware possible, better to make people spend money on shinny new hardware instead... did you know that a 2080ti had better perf in most compute / llm tests than the 5060? and im not even talking about amd here. i think amd will be even more crazy once i get to it. image if everyone old 2021 era rigs could run big model. woudnt this be crazy? marketing around you needing crazy bandwidth is mostly fake for retail, you are not going to handle 900000 user/s on your local api. you could run LLM on pcie gen 1 if you wanted to. will let you know about progress.

2

11

1

825