Mohammad Mozaffari

Ph.D. Student at the University of Toronto - Working on sparsity in training and inference of LLMs.

[Morad Vaisi @RezaVaisi]

@elonmusk
Dear Elon,
In these critical days, the people of Iran need internet access. Please help them stay connected as they struggle to reclaim their country from the rule of the mullahs. Stand with the Iranian people. Iran will not forget its friends.

[Mohammad Mozaffari @mohammad_mmzf]

I'm excited to have our work featured on the official PyTorch blog!

In this post, we explore how combining quantization with 2:4 sparsity can push the boundaries of efficient LLM compression and deployment.

Check out our blog post: https://t.co/jKC7VTeJnp

[PyTorch @PyTorch]

Quantization alone reaches its limits in compressing large language models. Combining it with 2:4 sparsity enables greater compression and efficient, hardware-accelerated deployment while maintaining accuracy.

Our latest community blog from Mohammad Mozaffari, Jesse Cai, and Supriya Rao explores the advantages of hybrid compression, key results on LLaMA models, and the software gaps that must be addressed to fully unlock its potential.

🔗 https://t.co/aAiXmeLVNM

#PyTorch #OpenSourceAI #LLMs #LLaMA

[Maryam Mehri Dehnavi @MaryamDehnavi]

Check out our paper “SLiM: One-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compression” to appear at ICML 2025!
📄 Read the paper: https://t.co/wjW1PlJ5Cd
#ICML2025 #AICompression

[Amir Yazdan @ayzddzya]

How semi-structure sparsity and low-rank adapters can be used productively together during pretraining and inference? In this work at #ICLR2025, Mohammad presents a new way of accelerating pre-training and inference by jointly using sparsity and low-rank adapters.

[Amir Yazdan @ayzddzya]

Our custom CUDA kernel yields up to 1.25x speedup and 0.63x less memory during training and up to 1.54x and 0.51x memory reduction in inference. Check our paper: https://t.co/a2DKbRqr2R

[Amir Yazdan @ayzddzya]

It is great to see a flavor of our work is implemented and replicated in Pytorch. In our work (https://t.co/a2DKbRqYSp) we show how to use semi-structured N:M sparsity in pretraining and inference. 𝐒𝐋𝐨𝐏𝐞 leverages N:M sparsity in the fwd and bwd passes.

[Maryam Mehri Dehnavi @MaryamDehnavi]

Its great to see parts of our work, SLoPe (https://t.co/pKQFpk2egj), has been implemented in Pytorch as one of the first successful releases for sparse DNN training.

[Maryam Mehri Dehnavi @MaryamDehnavi]

Second-order optimizers hold great promise for speeding DNNs. Check out our paper “MKOR: Momentum-Enabled Kronecker-Factor-Based Optimizer Using Rank-1 Updates” to appear at #Neurips 23. BERT on MKOR is 2.5X faster than first-order optimizers!

Paper: https://t.co/I2Bt04cE6F