Akhilan Boopathy

Founder @ Kuai, building faster and cheaper training algorithms for foundation models

[Jaedong Hwang @jaedong_hwang]

We learned the bitter lession that a poster should be checked before the poster session #ICLR2025.
Thank you all for coming and we are delight that you enjoyed our mistakes.
We are also highly appeciate authors of MMSearch allowing us to use their panel. @_akhilan

[Akhilan Boopathy @_akhilan]

@jaedong_hwang @abhiramiyer @FieteGroup Please come by our poster at ICLR today (Saturday, April 26th) if you're interested in hearing more: 15:00-17:30 SGT, poster #320

Paper: https://t.co/eEZDC98kKf

7/7

[Akhilan Boopathy @_akhilan]

Excited to share our #ICLR 2025 paper on Breaking Neural Network Scaling Laws with Modularity! Joint work with Sunshine Jiang, William Yue, @jaedong_hwang, @abhiramiyer &
@FieteGroup

https://t.co/i2ZlpT68xN

1/?

[Akhilan Boopathy @_akhilan]

@jaedong_hwang @abhiramiyer @FieteGroup We find that our approach outperforms a non-modular (monolithic) baseline and a modular architecture without our initialization scheme. The advantage is greater for higher dimensional inputs

6/?

[Akhilan Boopathy @_akhilan]

@jaedong_hwang @abhiramiyer @FieteGroup We test our approach on the Compositional CIFAR-10 task in which a network must simultaneously classify a sequence of images. Importantly, we *do not* provide a partition of the full input to the individual images: this must be learned by the network

5/?

[Akhilan Boopathy @_akhilan]

@jaedong_hwang @abhiramiyer @FieteGroup Unfortunately, practically speaking, modular neural network architectures often fail to properly exploit their theoretical advantages due to difficulty optimizing modular loss landscapes. We propose a novel initialization rule to mitigate this

4/?

[Akhilan Boopathy @_akhilan]

@jaedong_hwang @abhiramiyer @FieteGroup Fortunately, we can *break* this scaling law using modular networks: networks that break up a high dimensional input into lower dimensional subspaces. Theoretically, modular networks on modular tasks require *constant* training samples with dimension

3/?

[Akhilan Boopathy @_akhilan]

@jaedong_hwang @abhiramiyer @FieteGroup Neural networks suffer from the curse of dimensionality: as the intrinsic dimension (m) of a neural network's input increases, exponentially many training points (n) are required for good generalization

2/?

[Akhilan Boopathy @_akhilan]

@RylanSchaeffer @jaedong_hwang @abhiramiyer @FieteGroup Good question! Inductive bias is quantified as the amount of information required to specify the generalizing subset of the hypothesis space; a standard unit of information is a bit

[Akhilan Boopathy @_akhilan]

Excited to share our #IJCAI2024 paper on quantifying inductive bias in machine learning models

Towards Exact Computation of Inductive Bias

Joint work with William Yue, @jaedong_hwang, @abhiramiyer & @FieteGroup

https://t.co/bhRbciIfrl

1/?

[Akhilan Boopathy @_akhilan]

@jaedong_hwang @abhiramiyer @FieteGroup Please come by our presentation on Wednesday, August 7th if you're interested in hearing more: 15:00 KST in room 202B

Paper: https://t.co/bhRbciINgT

9/9

[Akhilan Boopathy @_akhilan]

@jaedong_hwang @abhiramiyer @FieteGroup Tasks with higher dimensional inputs require more inductive bias to solve

8/?

[Akhilan Boopathy @_akhilan]

@jaedong_hwang @abhiramiyer @FieteGroup Harder tasks also extract more inductive bias from a fixed architecture

8/?

[Akhilan Boopathy @_akhilan]

@jaedong_hwang @abhiramiyer @FieteGroup We find that our inductive bias estimates are more precise than prior upper bounds

7/?

[Jaedong Hwang @jaedong_hwang]

Excited to share our @TmlrOrg (#FeaturedCertification) paper on "Grid Cell-Inspired Fragmentation and Recall for Efficient Map Building"!
Inspired by grid cells' remapping pattern in neuroscience, we propose a fragmentation-and-recall framework for spatial exploration.#gridcell