Sidak Pal Singh

Research Scientist at Google DeepMind, working on Gemini.
(prev. PhD at ETH Zürich & MPI-IS Tübingen.)

No second-hand opinions. They are absolutely my own ;)

[Sidak Pal Singh @unregularized]

📢I'll be presenting two posters, at #ICML2024 HiLD workshop (Straus 2) today (assuming no further ✈️ delays):
- Closed form of the Hessian spectrum for some neural networks https://t.co/qQTWOZUsQw
- Landscaping Linear Mode Connectivity https://t.co/w09UyJ2Wyf

[Sidak Pal Singh @unregularized]

quite good actually. coding’s so smooth!

https://t.co/gprJA4ODgh

[Google DeepMind @GoogleDeepMind]

This is Gemini 3: our most intelligent model that helps you learn, build and plan anything.

It comes with state-of-the-art reasoning capabilities, world-leading multimodal understanding, and enables new agentic coding experiences. 🧵

[Sidak Pal Singh @unregularized]

sure the numbers are great, but screw that! it’s your phd, try wild and original ideas even if you fail once or twice at least.

https://t.co/Y1XisXjGXd

[Gabriele Berton @gabriberton]

A few numbers from my PhD:

8 first-author top-conference (CVPR/ICCV/ECCV) papers
100% acceptance rate per paper
80% acceptance rate per submission
1 invited long talk at CVPR tutorial
5 top-conf demos (acceptance rate 100% vs ~30% average)
~2k GitHub stars

[Sidak Pal Singh @unregularized]

when you go beyond linear mode connectivity, interesting things happen 😮👇

https://t.co/hxuaI94OiD

[Alexander Theus @TheusResearch]

1/ 🚨 New paper alert! 🚨
We explore a key question in deep learning:
Can independently trained Transformers be linearly connected in weight space — without a loss barrier?
Yes — if you uncover their rich symmetries.
📄 arXiv: https://t.co/wVoLYNzk0m

[Sidak Pal Singh @unregularized]

Belated life update:
🎓 PhD — done
🔬 Joined Google in NYC 🗽as a Research Scientist
♊️ Gemini: now more than just my star sign :)

[Sidak Pal Singh @unregularized]

🚀 TOMORROW afternoon at ICLR: Learn about the directionality of optimization trajectories in neural nets and how it inspires a potential way to make LLM pretraining more efficient ♻️

(Poster# 585, hall 2b)

[Sidak Pal Singh @unregularized]

Ever wondered how the optimization trajectories are like when training neural nets & LLMs🤔? Do they contain a lot of twists 💃 and turns, or does the direction largely remain the same🛣️? We explore this in our work for LLMs (upto 12B params) + ResNets on ImageNet.
Key findings👇

[Sidak Pal Singh @unregularized]

Don't miss out our spotlight ✨paper at ICLR 🇸🇬 about the loss landscape of Transformers and their special heterogeneous structure, done together with great collaborators!

https://t.co/VyQiHGM7CE

[Weronika Ormaniec @wormaniec]

Ever wondered how the loss landscape of Transformers differs from that of other architectures? Or which Transformer components make its loss landscape unique?

With @unregularized & @f_dangel, we explore this via the Hessian in our #ICLR2025 spotlight paper!

Key insights👇 1/8

[Sidak Pal Singh @unregularized]

@savvyRL :) I think the email address used there seems to suggest somebody doing it for him.. but you never know haha

[Alice Bizeul @AliceBizeul]

✨New Preprint ✨ Ever thought that reconstructing masked pixels for image representation learning seems sub-optimal?

In our new preprint, we show how masking principal components—rather than raw pixel patches— improves Masked Image Modelling (MIM).

Find out more below 🧵

[Sidak Pal Singh @unregularized]

@mayank98shri @TheGradient @ynd @baharanm The bulk + outliers notion isn't wrong. The key is to understand how sharpness reduction is happening. GN can lead to spurious sharpness reduction; while NME reduces sharpness through adapting the geometry of the model itself. In vision, these ways are balanced, but not for LLMs.

[Yann N. Dauphin @ynd]

Don’t miss our poster shedding more light on sharpness regularization at NeurIPS tomorrow https://t.co/GtB9E2VeCG

[Sidak Pal Singh @unregularized]

@agarwl_ @pratyushmaini It would be interesting to see the rank these models get haha. Predictions? :)

[Sidak Pal Singh @unregularized]

Reinventing things has a bad rep in today's age. But is it really that bad? Maybe it's something to be even cultivated, like selectively?

The second post in this series of blogs is now out. Let's have a deeper look at this overused trope!

https://t.co/BIlzZKepuU

[Sidak Pal Singh @unregularized]

I’m exploring a new form of writing—threads of human curiosity woven through the circuits of AI, crafting reflections that are, in the end, fully machine-generated, yet in a way profoundly human.

https://t.co/zJgGaePdDL

[Sidak Pal Singh @unregularized]

Come, let's scale up the building one floor,
And, layer up the neural networks once more.

Soon our buildings will touch the sky,
And, our computers will bear AGI.

A quaint little hut in the mountains is out of fashion,
Satisfaction has no gradients for backpropagation.

~Fitoor

[Sidak Pal Singh @unregularized]

@kellerjordan0 @bozavlado QK params and V params have very different behavior in terms of their curvature. So grouping them together is not ideal. I believe you could still try preconditioning QK params together, and keeping V separate.

[Sidak Pal Singh @unregularized]

At this paper count, recalling all the paper names would already be a big feat :)

https://t.co/7wlFCvOYzf

[Peter Richtarik @peter_richtarik]

Source: https://t.co/4KbFaflENO

[Sidak Pal Singh @unregularized]

“Hypotheses are nets: only he who casts will catch.” - Novalis

[Sidak Pal Singh @unregularized]

At last some attempts to change the status quo: authors with three or more papers are obligated to review for ICLR https://t.co/wp3yVNCfMo

[Preetum Nakkiran @PreetumNakkiran]

Review requirements! (And 10pg limit!)

[Sidak Pal Singh @unregularized]

Come to our posters today at 3:30 pm (Straus 2) to know more! :)

[Sidak Pal Singh @unregularized]

📢I'll be presenting two posters, at #ICML2024 HiLD workshop (Straus 2) today (assuming no further ✈️ delays):
- Closed form of the Hessian spectrum for some neural networks https://t.co/qQTWOZUsQw
- Landscaping Linear Mode Connectivity https://t.co/w09UyJ2Wyf

[Sidak Pal Singh @unregularized]

Poster 1: Sharpness/Flatness are much talked about: better minima, Sharpness aware minimization, Edge-of-Stability, and so on. But what really is sharpness? What exactly does it quantify, besides the surface-level definition? How are the eigenvalues and eigenvectors really like?

[Sidak Pal Singh @unregularized]

Poster 2: Linear Mode Connectivity (LMC) is yet another popular feature of neural loss landscapes. But how does LMC arise in the first place? How should the landscape be structured to allow LMC? Are barriers present just at the end, or do they start much early?