Rajat Patel

ML Engineer
I retweet to keep notes of important stuff I find here

[Percy Liang @percyliang]

Most classes teaching the algorithms, but a lot of ML is about the empirical intuitions that develop from experience. This class systemizes this process. Assessment is next experiment prediction.

[Azalia Mirhoseini @Azaliamirh]

I had so much fun creating the Self-Improving AI Agents course with @achowdhery, and teaching it twice in one year at Stanford!

We also collaborated with Stanford Online to make the course available online:

YouTube: https://t.co/t1uZlWz5eq

The field is moving incredibly fast, but we tried to focus on the core concepts that help us build better AI systems. I hope you enjoy it!

[Turing Post (Ksenia Se) @TheTuringPost]

This is a very interesting paper

It argues that a real scientific theory of deep learning is starting to form.
Researchers call it "learning mechanics." It's like physics, but for how neural networks learn.

Now there are 5 active research areas that together look like pieces of this theory:

1. Simple systems (like linear networks) that we can fully solve. Math there works cleanly and we have intuition about how learning behaves.

2. Studying extreme limits, like what happens if a network becomes infinitely wide.
Systems become mathematically tractable in these cases.

3. Simple laws that describe large-scale behavior, like scaling laws (performance vs. data/model size) and relationships between sharpness and generalization

4. Understanding hyperparameters separately
Learning rate, batch size, weight decay and other effects can be separated to make training look like a simpler system underneath.

5. There are underlying principles shared across systems as they scale: similar training dynamics, scaling trends, internal structures

Old theory can’t explain what we see today, that's why we need a real theory upgrade. But why it should be about mechanics?
The researchers see that deep learning needs 2 parts: mechanistic interpretability is like biology that studying individual parts, and studying overall laws and behavior is like physics.

[𝗿𝗮𝗺𝗮𝗸𝗿𝘂𝘀𝗵𝗻𝗮— 𝗲/𝗮𝗰𝗰 @techwith_ram]

Stanford's latest seminar is a deep dive into the evolution of world modeling in AI.

Focuses on the shift in the world model from traditional reconstruction methods toward latent space prediction.

Covers topics like:
- Introduction to JEPA & World Models
- Causal JEPA
- LOWER Model
- Practical Applications & Planning
- Future Outlook

[Stella Li @StellaLisy]

Millions of users now have months-long conversation histories with AI assistants💬 But this data is proprietary and unavailable to the academic community for research, training, or benchmarking.

We introduce HorizonBench🌅, a benchmark and data generator for long-horizon personalization: tracking a user's current preferences across a history where life events have silently changed them.

[Andrej Karpathy @karpathy]

Wow, this tweet went very viral!

I wanted share a possibly slightly improved version of the tweet in an "idea file". The idea of the idea file is that in this era of LLM agents, there is less of a point/need of sharing the specific code/app, you just share the idea, then the other person's agent customizes & builds it for your specific needs.

So here's the idea in a gist format: https://t.co/NlAfEJjtJV

You can give this to your agent and it can build you your own LLM wiki and guide you on how to use it etc. It's intentionally kept a little bit abstract/vague because there are so many directions to take this in. And ofc, people can adjust the idea or contribute their own in the Discussion which is cool.

[Paul Graham @paulg]

How to Do Great Work: https://t.co/zzxQTx7JJ3

[Rajat Patel @patel_raj55]

@MujumdarRohit can this out - https://t.co/Z8nQPonxay if you use torch, also seen datadog provide good utilities to so by having an agent run on your pod/container

[Dr. Julie Gurner @drgurner]

If you avoid even *the potential* for embarrassment, mistakes, failure, or ridicule...you are also avoiding the potential for success.

Doing what you really want to do & getting where you really want to go, is worth it.

Don't sit on the sidelines....
Push that line today.

[Rajat Patel @patel_raj55]

Maybe I am late on this - but analogies made by
@karpathy
in his talk at Stanford are really interesting and help connect dots to understand transformer's learning and performance at scale
https://t.co/abJjjoY6ou

[Jia-Bin Huang @jbhuang0604]

How to meet with your advisors/mentors?

If you are a grad student, having effective regular meetings with advisors or mentors is absolutely crucial for your success!

Here are some tips on how to make the most of it! 🧵

[Rajat Patel @patel_raj55]

As someone early in the career, how does one keep up with the pace of the current development around LLMs, any ideas or suggestions ?
#LLMs #NLProc #Transformers #MachineLearning

[Chip Huyen @chipro]

New post: bringing LLM applications to production!

1. Challenges of LLM engineering & the solutions that I’ve seen

2. How to compose multiple tasks and incorporate tools (e.g. SQL executor, bash, web browsers, third-party APIs)

3. Promising use cases

https://t.co/XWmZfIc5mr

[Rajat Patel @patel_raj55]

@aryaman2020 I have always wondered how we can scale the learning of LSTM based model that might some day match transformers learning capacity

[Thiago Ghisi @thiagoghisi]

I have managed nearly 100 Senior Engineers at this point in my career as an Engineering Leader.

Check out my latest article, On Being a Senior Engineer: https://t.co/XMBDb3n0FM

[Kevin Meng @mengk20]

How & where do large language models (LLMs) like GPT store knowledge? Can we surgically write *new* facts into them, just like we write records into databases?

Explainer 🧵 on how interpretability & model editing go hand-in-hand, and why these emerging areas are so important 👇

[Gibson Biddle @gibsonbiddle]

1. How to define your product strategy. When I start a new job, I define a SWAG (Stupid Wild-Ass Guess) product strategy within my first two weeks. Here's my approach...

[Sachin Tendulkar @sachin_rt]

.@imVkohli, it was undoubtedly the best innings of your life. It was a treat to watch you play, the six off the back foot in the 19th over against Rauf over long on was spectacular! 😮
Keep it going. 👍

#INDvPAK #T20WorldCup

[Rajat Patel @patel_raj55]

@davefarley77 merged!

[Ofir Press @OfirPress]

We've found a new way to prompt language models that improves their ability to answer complex questions
Our Self-ask prompt first has the model ask and answer simpler subquestions. This structure makes it easy to integrate Google Search into an LM. Watch our demo with GPT-3 🧵⬇️