Sunny Guha

Moving from physics to deep learning.

[Sunny Guha @sol0invictus]

An interactive map of humanity's origin myths — from every civilization, and how they connect to each other.

Go get lost in it: https://t.co/8mRAVpMn4p

[Sunny Guha @sol0invictus]

Wow Xi Yin (who is now at OpenAI ) is using GPT to generate a book on QFT - https://t.co/Ni0QXTNsfQ
Have mixed feelings about this one.

[Sunny Guha @sol0invictus]

Claude Fable available to pro+ subscribers till June 22th

[Sunny Guha @sol0invictus]

@scaling01 Training more epochs on the same data likely won't yield much. I am sure they are currently overtraining to capacity already.

[Sunny Guha @sol0invictus]

@scaling01 Isn't this inflated, assuming 40 Trillion token budget the number of active parameters with this compute is huge.

[Sunny Guha @sol0invictus]

@eliebakouch Among other things, I see peri-layernorm ( or sandwich layer norm) being a common strategy among recent models. I have seen it to have low variance growth which makes it good for models.

[Sunny Guha @sol0invictus]

Spent a few hours digging into RMSNorm. The magic isn't the forward rescaling, it's the backward gradient coupling across channelsforcing gradients orthogonal to x.
Detach the denominator → training collapses. That's why point-wise norms are suboptimal.
https://t.co/YD5O86DaOP

[Sunny Guha @sol0invictus]

@snarkyzarky If you liked the series, you should read the books as well. Anyway nothing beats star trek TNG :D .

[Sunny Guha @sol0invictus]

@Dubaloooooooooo Yayyyyyyy. This is better than hayday

[Sunny Guha @sol0invictus]

If you've got time to kill, here is a simple DDPG implementation in TensorFlow 2 from scratch without any bulky library overhead. #ReinforcementLearning #DeepLearning #DDPG

https://t.co/mDNtJseL9H

[Sunny Guha @sol0invictus]

@hardmaru I wonder if it's recycling the same ideas from causal set theory and tensor networks or if it's completely different. Will give it read this weekend.