Millend Roy

Graduate student at Columbia Engineering || SCAI Research Fellow'23 at MSFTResearch || 21 Graduate at IIT ISM Dhanbad

[Infini-AI-Lab @InfiniAILab]

🧠 How can we keep scaling LLMs with minimal changes in their serving energy?

DeepSeek’s Engram and our concurrent work STEM showed the promise of static sparse memory: large embedding tables with efficient access. But can mature pretrained models learn to use a newly added memory within a limited training budget, avoiding costly pretraining from scratch?

🚀 Introducing STAPLE: a recipe to extend pretrained models' abilities with static sparse memory. Serving energy per token stays close to dense and >50% below MoE upcycling in high-throughput regime.

Scalable capacity, interpretable memory, and compatibility with MoE upcycling - without any meaningful increase in serving cost. 🧵

Paper: https://t.co/qwVh1k9KAC
Code: https://t.co/7nn5b08Qpv
Website: https://t.co/m5CCP0gw9l