Co-Founder @Superfloat | AI chips and number formats | SF16 fixed-point inference | I help AI teams cut inference cost and latency (vLLM, quantization, GPU kern
@spheron fwiw in my support bot test an L40S was 2.7x cheaper per conversation than the best L4 even at nearly 2x the hourly price, but only when kept busy
@lucaslocal7 same in my support bot test but worse lol. n-gram spec decoding didn't help at all bc the replies rephrase the docs instead of copying them
@renandnzsantos didn't measure that tbh. it was simulated support traffic and i only looked at cost per conversation, no ttft target. your burst test by prompt length makes sense though.
@macayaven Qwen3-8B on vLLM, support bot workload, graded with our own answer quality check. and yep, AWQ alone passed, the drop only showed up once FP8 KV went on top. method + code: https://t.co/wpnL3HJS0W
@nicbstme in my support bot test self-hosting only beat the cheapest cached APIs above ~0.9 req/s on an L4. below that, just reselling tokens was cheaper tbh.
@dfrodriguezl no medimos el hit rate por turno tbh, solo vimos que poner el prompt fijo primero bajΓ³ el coste ~40%. y sΓ, fecha o nombre arriba puede romper ese prefijo.
@arpit_bhayani in our one support-bot test, self-hosting only beat the cheapest cached APIs above ~0.9 req/s on an L4. do you measure batch eligibility by requests or actual spend?