Akshay Kumar

PhD student @UMNews | Undergrad @IITKanpur. Research interest: deep learning theory, optimization

[Akshay Kumar @aksh0135]

Excited to share our recent work introducing 𝗡𝗲𝘂𝗿𝗼𝗻 𝗣𝘂𝗿𝘀𝘂𝗶𝘁 (𝗡𝗣) - a greedy algorithm for training neural networks.
https://t.co/tqSivwVq8J

[Daniel Kunin @KuninDaniel]

Had a great time visiting @TTIC_Connect last year -- the ideas I presented have progressed quite a bit since then:
- https://t.co/QqZC9EgxgW
- https://t.co/uqvXVOSdfT

[Journal of Machine Learning Research @JmlrOrg]

'Towards Understanding Gradient Flow Dynamics of Homogeneous Neural Networks Beyond the Origin', by Akshay Kumar, Jarvis Haupt.

https://t.co/4Yyf6rWoHU

#gradients #gradient #flow

[Akshay Kumar @aksh0135]

ii) Although NP is inspired by the GF dynamics in the small init regime, it is NOT equivalent to GF (the reasons are technical — see Sec. 5.3). Nonetheless, it could offer valuable insights into how neural networks build features during training. 13/n

[Akshay Kumar @aksh0135]

Excited to share our recent work introducing 𝗡𝗲𝘂𝗿𝗼𝗻 𝗣𝘂𝗿𝘀𝘂𝗶𝘁 (𝗡𝗣) - a greedy algorithm for training neural networks.
https://t.co/tqSivwVq8J

[Akshay Kumar @aksh0135]

A few important caveats - i) NP currently applies to homogeneous activations, which includes (Leaky) ReLU and its higher powers (max(x,0)^p). It also does not yet cover ResNets or Transformers, but we plan to extend NP to more general architectures. 12/n

[Akshay Kumar @aksh0135]

Finally, we leverage these theoretical insights to build a greedy algorithm for training deep networks. More about that in the next thread.

[Akshay Kumar @aksh0135]

In a series of works, we study the dynamics of gradient flow arising from minimizing the training loss in supervised learning.
We focus on homogeneous neural networks trained with small initialization.
This (long) thread summarizes our key findings. 🧵

[Akshay Kumar @aksh0135]

Extending the theory rigorously to ReLU is an important future direction. Another important direction is to study training dynamics of non-homogeneous architectures such as ResNets and Transformers.