Shashata Sawmya

Graduate Research Assistant of the Department of Electrical Engineering and Computer Science of Massachusetts Institute of Technology.

[Shashata Sawmya @shashata005]

I wonder if the land of the free has forgotten its principles of upholding freedom, equity and justice and whether the true land of the free is somewhere else!

[Shashata Sawmya @shashata005]

Doing a talk on MI in LMs. Excited with a big E!!

[UBC NLP Group @UBC_NLP]

🚨 Talk Alert!
We’re excited to host Shashata Sawmya (MIT) on May 29 @ 11AM (ICCS 246) for a deep dive into mechanistic interpretability in LLMs — from emergent features to UTune. Don’t miss it! 🧠
#NLP #LLMs #AI #Interpretability #UBC

[Shashata Sawmya @shashata005]

🔟
💡 Dive deeper & share thoughts! 👉 https://t.co/8O05MqzlbZ

[Shashata Sawmya @shashata005]

6️⃣
🌌 Spatial surprise: early-layer concepts vanish mid-stack but re-appear in the final layer—challenging the neat “simple → complex hierarchy” story.

[Shashata Sawmya @shashata005]

9️⃣
Why care? Pinpointing when “knowledge” is born helps:
• smarter checkpoint selection
• safer audits & red-teaming
• better pruning/distillation
• sharper scaling laws

[Shashata Sawmya @shashata005]

5️⃣
⏰ Temporal emergence:
<3 % concepts fire in the first 1 k steps.
Big surge at 5 k, then two bigger leaps (10 k & 30–40 k).
By 143 k steps > 99 % of concepts are active.

[Shashata Sawmya @shashata005]

4️⃣
With AUTOINTERP, we label 512 SAE neurons—so each neuron fires for a crisp, human-readable concept. Then we probed for 9 topical concepts — from Physics to Philosophy using a vector database. The framework is called EyeSee.

[Shashata Sawmya @shashata005]

3️⃣
We probe 3 axes:
• ⏰ Time – 25 checkpoints of Pythia-12B
• 🌌 Space – all 36 transformer layers
• 📏 Scale – 10 sizes from 14 M → 12 B params

[Shashata Sawmya @shashata005]

2️⃣
🤔 Why? LLMs are powerful but opaque. We use sparse auto‑encoders (SAE) to watch interpretable features flicker to life inside Pythia models as they train, grow & traverse layers.

[Shashata Sawmya @shashata005]

1️⃣
🚀 New paper alert! “The Birth of Knowledge: Emergent Features across Time, Space & Scale in Large Language Models.”
Curious how & when LLMs learn? Thread 👇
#Interpretability #Emergence #LLM #ML

[Shashata Sawmya @shashata005]

The objective connotation of "truthfulness" is thought to be profound, in fact it's not even superficial.

[Shashata Sawmya @shashata005]

Is it just me, or have mathematicians unlocked truths deeper than most of us can dream?

[Shashata Sawmya @shashata005]

@DAlistarh @sparseLLMs Congrats @DAlistarh

[Shashata Sawmya @shashata005]

What have you done to be the 🐐?

[Linghao Kong @linghaokong76]

Happy to announce that our work "Wasserstein Distances, Neuronal Entanglement, and Sparsity" with @shashata005, Ilia Markov, @DAlistarh, and @ShavitNir has been accepted into ICLR 2025 as a Spotlight Presentation! Paper link: https://t.co/5aBm3CaZ25

[Brian Hie @BrianHie]

We trained a genomic language model on all observed evolution, which we are calling Evo 2.

The model achieves an unprecedented breadth in capabilities, enabling prediction and design tasks from molecular to genome scale and across all three domains of life.

[Sam Altman @sama]

today we launch deep research, our next agent.

this is like a superpower; experts on demand!

it can go use the internet, do complex research and reasoning, and give you back a report.

it is really good, and can do tasks that would take hours/days and cost hundreds of dollars.

[Shashata Sawmya @shashata005]

Attending NeurIPS for the first time!

Down to chat about anything from LLM Optimization to Computational Neuroscience (Yep that's the two things I work on :v)