ayan sengupta

PhD student at IIT Delhi

[ayan sengupta @ayans007]

🔬 Parmanu (Hindi for Atom) is live.

Parmanu is part of the Computational Social Systems (LCS2) @lcs2lab at IIT Delhi, led by Prof. Tanmoy Chakraborty @Tanmoy_Chak , and is our dedicated home for Efficient Large Language Models (LLMs) and Small Language Models (SLMs).

We’re at a turning point in AI. The future won’t be defined by scaling alone - it will be shaped by efficiency, accessibility, and real-world deployability. Parmanu is our effort to push this efficiency-first vision forward. ✨🤖

🔗 Explore the project page: https://t.co/siVwaKY1GN

Why Parmanu matters 🔥

• 📚 A centralized hub for our research, with papers accepted at ICLR, ICML, NeurIPS, TACL, ACL, and TMLR
• 🛠️ Open access to tools, code, and artifacts spanning model compression, KV efficiency, PEFT, inference optimization, knowledge distillation, and model coordination
• 🧠 A growing ecosystem focused on making strong language models smarter per parameter, not just larger

What this means for the community

For researchers 👩‍🔬👨‍🔬

A curated, evolving resource tied to top-tier venues

Reproducible artifacts and principled problem formulations

A shared space to advance efficiency-centric LLM research

For practitioners 👩‍💻👨‍💻

Practical techniques to deploy LLMs under tight latency and memory budgets

Faster paths from paper → production

Tools that actually work under real deployment constraints

What’s coming in 2026 🚀🔮

• 📊 Efficient LLM/SLM leaderboards
• 🧪 Open-sourced efficient LLM artifacts
• ⚙️ More tools for compression, distillation, and inference
• 🤝 Deep integration with Hugging Face and other popular libraries

If you’re excited about efficient, sustainable, and scalable AI, check out Parmanu, share feedback, and collaborate with us. The next wave of LLMs won’t just be bigger - they’ll be leaner, faster, and more impactful. 🌟

#EfficientLLMs #SLMs #ModelCompression #InferenceOptimization #KnowledgeDistillation #AIResearch #NLP #ICLR #ICML #NeurIPS #ACL #TACL #TMLR #IITDelhi #LCS2 #Parmanu

[ayan sengupta @ayans007]

Three 2026 papers (AOPD, SOD, OPSD) are patching on-policy distillation one failure mode at a time.
Our 2023 work, MPDistil (ICLR 2024 poster), already had structural answers to all four.
A short article on the convergence. @Tanmoy_Chak @lcs2lab

[ayan sengupta @ayans007]

https://t.co/RaHXIzI1c6

[ayan sengupta @ayans007]

@ysu_ChatData @Tanmoy_Chak We introduce Global Eviction Ratio (GER), which tracks whether answer-critical tokens remain reachable across attention heads. GER spikes before benchmark accuracy drops and strongly correlates with the hallucination safety cliff.

[ayan sengupta @ayans007]

@ysu_ChatData @Tanmoy_Chak We observe stronger routing rigidity with global head-wise pruning methods (e.g., AdaKV), which increase head-level consensus. Chunk-based pruning (e.g., FINCH) tends to preserve more routing diversity and flexibility.

[LCS2 Lab @lcs2lab]

🚀 New Podcast Alert 🎙️
Prof. Tanmoy Chakraborty @Tanmoy_Chak joins Rudraditya on The Inner Circle With Rudraditya podcast for a sharp, no-hype conversation on the realities, risks, and future of AI. #AI #MachineLearning #SLM #AIGovernance #NLProc
https://t.co/KSUw5bYoge

[ayan sengupta @ayans007]

@MeatigoOfficial didn’t realize you guys are into fraud, my order Mea41Vmm72034 is showing delivered, but the rider called and said he can’t deliver it, as he is far from the location and he wants me to pick it up. Apparently he is now returning home and deliver it tomorrow. Wow!!

[Aradhye Agarwal @AradhyeAgarwal]

New paper on test-time scaling! We analyze 30 billion tokens from 8 LLMs (32B to 235B) and 4 reasoning datasets and propose a practical recipe for effective scaling of inference-time compute.

[LCS2 Lab @lcs2lab]

🚀 #NeurIPS25 Sneak Peek! 🚀
Thrilled to present a KV cache compression method for LLMs, now a part of @nvidia KVPress library.
📄 Value-Guided KV Compression for LLMs via Approximated CUR Decomposition
👥 @ayans007, @codetalker07, @Tanmoy_Chak
🔗 Arxiv: https://t.co/tJZdqicXHq

[Aradhye Agarwal @AradhyeAgarwal]

Excited to see LoRA back in the spotlight! LoRA –– and PEFT methods in general — have been key for efficient train-time adaptation of foundation models. In our recent TACL paper, we introduced a new PEFT method that outperforms LoRA across a range of NLP and math tasks!

[ayan sengupta @ayans007]

Soon we will submit PR to https://t.co/5Er4ZCcULV

[ayan sengupta @ayans007]

Excited to share that our recent paper on value-guided key-value compression for LLMs got accepted at NeurIPS 2025! Our paper is motivated by a simple observation - value-guided KV compression can achieve lower post-eviction loss. Preprint available - https://t.co/kuSrstj6G1. @Tanmoy_Chak

[ayan sengupta @ayans007]

To ensure that same indices are evicted from both key and value matrices, we use an efficient CUR decomposition of the composite matrix.

[Tanmoy Chakraborty @Tanmoy_Chak]

𝐎𝐮𝐫 𝐰𝐨𝐫𝐤 𝐨𝐧 𝐊𝐕 𝐜𝐚𝐜𝐡𝐞 𝐂𝐨𝐦𝐩𝐫𝐞𝐬𝐬𝐢𝐨𝐧 𝐟𝐨𝐫 𝐋𝐋𝐌𝐬, 𝐚𝐜𝐜𝐞𝐩𝐭𝐞𝐝 𝐢𝐧 #NeurIPS2025 .

Our constant attempt to design small models continues -- This time, we focus on 𝐊𝐕 𝐜𝐚𝐜𝐡𝐞 𝐜𝐨𝐦𝐩𝐫𝐞𝐬𝐬𝐢𝐨𝐧 for LLMs. Many SOTA approaches rely on "attention scores" to decide which tokens to evict -- assuming that higher scores indicate greater importance. While effective, this overlooks the role of "value vectors" in shaping the final attention outputs.

💡 To address this, we introduce 𝐂𝐮𝐫𝐃𝐊𝐕 -- a novel method inspired by the classic CUR decomposition from matrix sketching theory. Instead of focusing only on keys, CurDKV selects the most important keys and values together using leverage scores.

🔑 Highlights:
- Outperforms leading baselines (including SnapKV and adaptive variants) across a range of compression ratios.
- Achieves up to 40% reduction in generation latency while maintaining strong model quality.
- Grounded in elegant low-rank matrix approximation theory, yet highly practical for modern LLMs.

👉 Preprint: https://t.co/jdqmoyC0ws
@lcs2lab @iitdelhi

[Tanmoy Chakraborty @Tanmoy_Chak]

Just as I advocate for 𝘥𝘰𝘸𝘯𝘴𝘤𝘢𝘭𝘪𝘯𝘨 𝘓𝘓𝘔𝘴, I also call for 𝐝𝐨𝐰𝐧𝐬𝐜𝐚𝐥𝐢𝐧𝐠 𝐀𝐈 𝐜𝐨𝐧𝐟𝐞𝐫𝐞𝐧𝐜𝐞𝐬.

Growing to 20k+ submissions is not a success metric -- it incentivizes negative science, overwhelms the community, and dilutes quality.

𝐁𝐢𝐠𝐠𝐞𝐫 𝐢𝐬𝐧’𝐭 𝐚𝐥𝐰𝐚𝐲𝐬 𝐛𝐞𝐭𝐭𝐞𝐫.

There is no pride in saying: “We received 20k+ papers -- 3x more than last year -- look how popular our venue is!” Popularity does not equal progress.

Requesting big leaders to think about it and put more restrictions on the paper submissions.

@RealAAAI @emnlpmeeting @NeurosamaAI @icmlconf @iclr_conf @CVPR @IJCAIconf @aclmeeting

[LCS2 Lab @lcs2lab]

📝 Robust and Efficient Fine-tuning of LLMs with Bayesian Reparameterization of Low-Rank Adaptation
👥 @ayans007, Vaibhav Seth, @arinjay_pathak, Aastha Verma, Natraj Raman, Sriram Gopalakrishnan, Niladri Chatterjee, @Tanmoy_Chak
https://t.co/vPAvMvXUBD

[ayan sengupta @ayans007]

@rosinality In 2023 we showed that indeed it is possible to design Transformers with constant Lipschitz bound with suitable activation functions and parameterization. Lipschitz continuity helps in efficiency as well as interpretability. https://t.co/wWJ0ysnwga

[ayan sengupta @ayans007]

@AradhyeAgarwal Sorry not linearly, but exponentially

[ayan sengupta @ayans007]

@AradhyeAgarwal This is predominantly due to >1 activation factor of the non-linear activation functions (ReLU, ELU has activation factor 1, most of the other functions have >1). As we pile up more layers, the activation factor increases linearly. We explored this in https://t.co/wWJ0ysmYqC

[Aradhye Agarwal @AradhyeAgarwal]

Big news!
Our paper "Step-by-Step Unmasking for Parameter-Efficient Fine-tuning of LLMs" has been accepted to TACL — a top-tier ACL-sponsored journal (Impact Factor > 9)! 🎉

📄 Paper: https://t.co/iv03ftyu10
🔧 Code: https://t.co/SuPQqc2t3S

🧵Thread below 👇