Rudra

MLE, Interested in Generative models, RL, Video Streaming, Gaming, Video Compression

[Rudra @rrpharale]

@omarsar0 Check out https://t.co/SXzmCuMlXf . We also have an introductory blog at https://t.co/CZNue5DF2n

[Rudra @rrpharale]

@p_nawrot Quantization techniques like zero point quantization or absmax quantization leads to drop in accuracy at lower precision. Hence new techniques like LLM.int8() are used at lower precision. Check out paper on LLM.int8() for more details.

[Rudra @rrpharale]

@p_nawrot @kaffyou Because unlike 32 bit -> 16 bit quantization at lower precision is not straight forward. If we simply quantize 16bit -> 8 bit, there will be significant loss in performance. To overcome this, quantization -> dequantization happens on demand and this process leads to slower gen.