Kshitij Gupta

Working toward AGI | Interested in Scaling Laws, Multimodal Foundation Models, Memory & Reasoning! @Mila_Quebec | prev @DeepMind, @Microsoft

[Ethan Caballero @ethanCaballero]

Want to Superforecast AGI?

Stop by "Broken Neural Scaling Laws" (https://t.co/lpHQfm8VOM) poster
at ICLR Conference poster session at MH1-2-3-4 #27 at 11:30AM - 1:30PM on Monday
(https://t.co/0CPFcaZklN)
& at ICLR Me-FoMo Workshop poster session at AD10 at 1PM - 2PM on Thursday
(https://t.co/RHS2cYZ28Y)

#ICLR2023 @iclr_conf

[Ethan Caballero @ethanCaballero]

New version of Broken Neural Scaling Laws (BNSL) is out with accurate extrapolation results for the scaling behaviors listed in this attached picture:

https://t.co/HGNMfHi8g6
https://t.co/MLovkJzYTh

Plots of all extrapolations are in this 🧵.

Any other extrapolations you want?

[Kshitij Gupta @kshitijkgupta]

@kanzure After trying out various problems, it looks like it usually gets it right between 2-4 iterations. Experimenting with more complex problems currently! Interestingly sometimes it does get stuck in a loop as well! Will be adding this analysis and examples in Part 2!

[Kshitij Gupta @kshitijkgupta]

Excited to share a sneak peek of what I have been exploring lately: How LLMs can use external tools and memory to iteratively design, implement, and debug code. Even more exciting results, features, and analyses coming out soon!
https://t.co/RYpCG3pfuX

#LLMs #ChatGPT #code

[Kshitij Gupta @kshitijkgupta]

@madan_ankapura Thanks for trying this out! Yes, I recently got access to GPT-4 and from my brief experimentation, it looks like GPT 4 generally understands the problem statement better and avoids extra/unnecessary edge test cases which are beyond the possible inputs given in the problem!

[Kshitij Gupta @kshitijkgupta]

@BrandoHablando gpt-3.5 turbo is the most capable and cost effective model in the GPT-3.5 family optimized for chat. https://t.co/3bkzINHmM9. As far as I remember, it is the same one that powers ChatGPT as well!

[Kshitij Gupta @kshitijkgupta]

Happy new year everyone! 2022 was a wild ride, but I can’t wait to see what 2023 has in store as we work toward AGI. Looking forward to building multi-modal agents that can interact with external worlds and tools, and reason and solve new tasks! Here’s to an even wilder new year!

[Kshitij Gupta @kshitijkgupta]

The importance of optimally using data and data engineering! In various domains requiring real-world interactions, we are going to be bottlenecked by data. It is more and more important to investigate methods that effectively use and additionally automate data augmentation!

[Kshitij Gupta @kshitijkgupta]

I am happy to share that I had a wonderful time interning at @DeepMind the last few months! I feel lucky to get an opportunity to work with amazing people like @allisontam_, @FelixHill84, and the rest of the team, at the intersection of Embodied agents and Large-scale Modelling!

[Kshitij Gupta @kshitijkgupta]

This is a super exciting time to be a Research Engineer! As seen recently, large-scale models have demonstrated impressive capabilities! Engineering along with Scaling laws will play a key role in optimally training agents at scale, and developing algorithms that scale better.

[David Krueger 🦥 ⏸️ ⏹️ ⏪ @DavidSKrueger]

A new paper from my student @ethanCaballero, @kshitijkgupta, @irinarish and your's truly!

I'm really impressed with the empirical results.

The TL;DR is that we replace "linear on a log-log plot" with "piecewise linear on a log-log plot".

[Kshitij Gupta @kshitijkgupta]

Very excited to share Broken Neural Scaling Laws!
We decompose scaling trends and model them with smoothly broken power laws. This gives SotA extrapolation results on a wide set of tasks! Work done with amazing collaborators - @ethanCaballero, @irinarish, and @DavidSKrueger

[Ethan Caballero @ethanCaballero]

We present the True Functional Form of the Scaling behavior of All things that involve Artificial Neural Networks, “Broken Neural Scaling Laws”:
https://t.co/BqRwxRDXLc

We’re giving $1Billion to 1st person who disproves this claim.

Details to win $1Billion in this thread. (1/N)

[Kshitij Gupta @kshitijkgupta]

What I find most exciting about this work:
-Efficiently captures long contexts without an O(T^2) complexity.
-Encourages capturing only relevant information from the past.
-Top-Down information introduces Feedback and Recurrence into Transformers helping model sequences better!

[Aniket Didolkar @Aniket_d98]

Temporal Latent Bottleneck: Synthesis of Fast and Slow Processing Mechanisms in Sequence Learning

Work with @kshitijkgupta , @anirudhg9119 , Alex Lamb, @rosemary_ke , Yoshua Bengio

@Mila_Quebec

https://t.co/wjDWoidzj6 (1/N)

[Kshitij Gupta @kshitijkgupta]

This is super exciting work by @GoogleAI! Chain of thought prompting and step-by-step reasoning can help LLMs break down complex multi-step problems and iteratively reuse their knowledge to solve each sub-problem! Solving problems beyond what was seen during pretraining stage!

[aitor @alewkowycz]

Very excited to present Minerva🦉: a language model capable of solving mathematical questions using step-by-step natural language reasoning.
Combining scale, data and others dramatically improves performance on the STEM benchmarks MATH and MMLU-STEM. https://t.co/bQJOyMSCD4

[CoLLAs 2026 @CoLLAs_Conf]

We are thrilled to release the list of invited speakers at @CoLLAs_Conf 2022: Yoshua Bengio, Rich Caruana, Claudia Clopath, Abhinav Gupta, @hugo_larochelle, @HanieSedghi, Tinne Tuytelaars. Our registrations are also now open: https://t.co/pCLkU7R2BS

[Kshitij Gupta @kshitijkgupta]

This helps improve generalization and sample efficiency in image classification, multi-task RL, offline RL, self-supervised learning, and language modeling!

Work done with wonderful collaborators - @Aniket_d98, @anirudhg9119, A. Lamb, @rosemary_ke, and Y. Bengio!

[Kshitij Gupta @kshitijkgupta]

Excited to share the Temporal Latent Bottleneck! Proposed model jointly leverages the strengths of Transformers and Recurrent Neural Networks by dividing the computation into two streams (Fast and Slow) that interact via an attentional bottleneck (Temporal Latent Bottleneck)!

[Sepp Hochreiter @HochreiterSepp]

ArXiv https://t.co/wC81XYtm6M: Combining RNNs and Transformers: 1) sequence chunks compressed by an RNN to vectors, 2) vectors processed by Transformers. RNN compression facilitates generalization and Transformer expressiveness. Improves sample efficiency and generalization.

[Kshitij Gupta @kshitijkgupta]

Excited to be here! Quick intro: Student at @Mila_Quebec, advised by @apsarathchandar and @irinarish! Passionate about building AI agents! Currently working in Sequential Decision Making, Scaling Laws, Reasoning, Memory, and Planning! Love exploring and learning new things!