Arindam Duttagupta

Principal ML Engineer @Cadence | MS in ECE | Machine Learning @GeorgiaTech | EE @iiest_s

[elie @eliebakouch]

Training LLMs end to end is hard. Very excited to share our new blog (book?) that cover the full pipeline: pre-training, post-training and infra. 200+ pages of what worked, what didn’t, and how to make it run reliably

https://t.co/iN2JtWhn23

[Ali Behrouz @behrouz_ali]

We keep scaling model parameters by increasing width and stacking more layers, but what if the truly missing axes for continual learning are compression and stacking the learning process?

Excited to share the full version of Nested Learning, a new paradigm for continual learning and machine learning in general.

Paper: https://t.co/75T93mvwKm

[Vahab Mirrokni @mirrokni]

Check out the blog post about our NeurIPS'24 paper on graphs and reasoning. We explored the theoretical and practical underpinnings of reasoning within transformer-based neural networks. More in comments: https://t.co/k2dmDQAS79

[moe @mohbibi_]

if you want to understand how computers work at the hardware level.

I've seen the first video banger so far.

[Preetum Nakkiran @PreetumNakkiran]

Our tutorial on diffusion & flows is out! We made every effort to simplify the math, while still being correct. Hope you enjoy! (Link below -- it's long but is split into 5 mostly-self-contained chapters).

lots of fun working with @ArwenBradley @oh_that_hat @advani_madhu on this

[Hyung Won Chung @hwchung27]

I gave a talk at Seoul National University.

I titled the talk “Large Language Models (in 2023)”. This was an ambitious attempt to summarize our exploding field.

Video: https://t.co/vumzAtUvBl
Slides: https://t.co/IidLe4JfrC

Trying to summarize the field forced me to think about what really matters in the field. While scaling undeniably stands out, its far-reaching implications are more nuanced. I share my thoughts on scaling from three angles:

1) Change in perspective is necessary because some abilities only emerge at a certain scale. Even if some abilities don’t work with the current generation LLMs, we should not claim that it doesn’t work. Rather, we should think it doesn’t work yet. Once larger models are available many conclusions change.

This also means that some conclusions from the past are invalidated and we need to constantly unlearn intuitions built on top of such ideas.

2) From first-principles, scaling up the Transformer amounts to efficiently doing matrix multiplications with many, many machines. I see many researchers in the field of LLM who are not familiar with how scaling is actually done. This section is targeted for technical audiences who want to understand what it means to train large models.

3) I talk about what we should think about for further scaling (think 10000x GPT-4 scale). To me scaling isn’t just doing the same thing with more machines. It entails finding the inductive bias that is the bottleneck in further scaling.

I believe that the maximum likelihood objective function is the bottleneck in achieving the scale of 10000x GPT-4 level. Learning the objective function with an expressive neural net is the next paradigm that is a lot more scalable. With the compute cost going down exponentially, scalable methods eventually win. Don’t compete with that.

In all of these sections, I strive to describe everything from first-principles. In an extremely fast moving field like LLM, no one can keep up. I believe that understanding the core ideas by deriving from first-principles is the only scalable approach.

[Lucas Beyer (bl16) @giffmana]

What makes CLIP work?
The contrast with negatives via softmax?
The more negatives, the better -> large batch-size?

We'll answer "no" to both in our ICCV oral🤓
By introducing SigLIP, a simpler CLIP that also works better and is more scalable, we can study the extremes.

Hop in🧶

[AK @_akhaliq]

Tool Documentation Enables Zero-Shot Tool-Usage with Large Language Models

paper page: https://t.co/voXsq7cLF5

Today, large language models (LLMs) are taught to use new tools by providing a few demonstrations of the tool's usage. Unfortunately, demonstrations are hard to acquire, and can result in undesirable biased usage if the wrong demonstration is chosen. Even in the rare scenario that demonstrations are readily available, there is no principled selection protocol to determine how many and which ones to provide. As tasks grow more complex, the selection search grows combinatorially and invariably becomes intractable. Our work provides an alternative to demonstrations: tool documentation. We advocate the use of tool documentation, descriptions for the individual tool usage, over demonstrations. We substantiate our claim through three main empirical findings on 6 tasks across both vision and language modalities. First, on existing benchmarks, zero-shot prompts with only tool documentation are sufficient for eliciting proper tool usage, achieving performance on par with few-shot prompts. Second, on a newly collected realistic tool-use dataset with hundreds of available tool APIs, we show that tool documentation is significantly more valuable than demonstrations, with zero-shot documentation significantly outperforming few-shot without documentation. Third, we highlight the benefits of tool documentations by tackling image generation and video tracking using just-released unseen state-of-the-art models as tools. Finally, we highlight the possibility of using tool documentation to automatically enable new applications: by using nothing more than the documentation of GroundingDino, Stable Diffusion, XMem, and SAM, LLMs can re-invent the functionalities of the just-released Grounded-SAM and Track Anything models.

[Sayan Ranu @SayanRanu]

Incredibly excited to attend and present 3 papers at #ICML2023 next week.

In our first paper, we propose a GNN framework for graphs with heterogenous feature sets.

Details: https://t.co/OynvTEQ43g

@srikanta @shubhamgupta583 @sahim1992 @icmlconf

[Gin Jiang @ZhiyingJ]

I'm surprised to see gzip has received that much attention😂 I'd like to make some clarifications in case this paper delivers any incorrect messages: 🧵1/8

[Martin Bauer @martinmbauer]

The wavelengths of the gravitational waves measured by the pulsar timing arrays are enormous

One wavelength spans trillions of miles, 6000 the size of our solar system, and takes 30 years to move past. One period ago we were using Windows 3.1 and Jurrassic park was in cinemas

[Martin Bauer @martinmbauer]

Physics threads 2022

Happy 2023 everyone!

[Martin Bauer @martinmbauer]

This was a great year for physics books. Here is my reading list (in no particular order)

Disclaimer: I haven't read all of them completely (yet), nobody paid me and I might add more later if I missed something

Feel free to add more recommendations!

https://t.co/f1zxe89Eum

[Arvind Narayanan @random_walker]

15 years ago my PhD advisor taught me One Weird Trick for editing your own writing. Edit **back to front**, paragraph by paragraph. I still use it and it still surprises me how well it works. When I get my students to do it, it often blows their minds. Try it!

[Juhani Halkomäki @JuhaniHalkomaki]

#creativecoding #p5js

Each particle tries to catch the next one. When they get close enough a line is drawn between them. The rules are very simple yet the form they generate is fascinating.

[hardmaru @hardmaru]

New blog post: Collective Intelligence for Deep Learning

Recently, @yujin_tang and I published a paper about how ideas like swarm behavior, self-organization, emergence are gaining traction in deep learning.

I wrote a blog post summarizing the key ideas:
https://t.co/S644KjM20e

[Daniel Khashabi 🕊️ @DanielKhashabi]

For my first course at @jhuclsp, I am leading a class on recent developments in "self-supervised models." Here is the list of the papers and slides we cover: https://t.co/zDjxlNXL2L Would love to hear Twitter's suggestions for additional exciting developments to discuss!🤗

[Akari Asai @AkariAsai]

This is a comprehensive list of the must-read papers on the recent progress of self supervised NLP models (or impressive capabilities of LLMs) and great summary slides!
I also love the role-playing paper-reading seminar fromat! 👩🏽‍🔬 (https://t.co/ydNANw0Smt)

[Lilian Weng @lilianweng]

🧮 I finally spent some time learning what exactly Neural Tangent Kernel (NTK) is and went through some mathematical proof. Hopefully after reading this, you will not feel all the math behind NTK is that scaring, but rather, quite intuitive.

https://t.co/iLliYzlJqY