mike

interested in ai, bio, stats | PhD @MIT

[mike @mr_sdach]

@industriaalist Is there a world where you need task specific post-training to max out performance though?

SGD isn’t memoryless so if you know you want to make an AI lawyer just throwing more law bits in pre-training won’t be as good.

Unless you replace SGD… (realizing this realtime)

[mike @mr_sdach]

@industriaalist Yesterday Andrew Gordon Wilson said we’re gonna replace transformers soon too.

Related? 👀

[mike @mr_sdach]

@llllvvuu So RSI is just meta learning optimizers?

[mike @mr_sdach]

@kanomalie Yeah I was. I left in April.

[mike @mr_sdach]

@llllvvuu Why do you think NTP and transformers are so dominant? Why hasn’t there been another recipe that competes with them?

Software-hardware codesign has trapped everyone in transformer land with huge mercor contracts.

[mike @mr_sdach]

@industriaalist Was really into @drmichaellevin in 2022-2023. He and Karl Friston are mentioned far too infrequently.

[mike @mr_sdach]

@madprizm0 @AlexiGlad It then makes sense that diversity is increased since the XM allows you to get grad from low prob trajectories. XM up weights these relative to sampling prob!

Huge ah-ha moment for me. Thanks!

[mike @mr_sdach]

@AlexiGlad @madprizm0 Trying to understand how they prioritize diversity.

Sample diversity is fixed for model at any training step and we only see variance go down with training. How is variance for exploration model so much higher if initialization is same? (See @madprizm0 reply to me for context)

[mike @mr_sdach]

@madprizm0 @AlexiGlad How much of this extra variance is inherent to the true data distribution though?

Like does exploration objective ALWAYS result in higher variance or does it just learn the real distribution better and that has higher variance?

Exploration = more epiplexity in outputs

[mike @mr_sdach]

@AlexiGlad @madprizm0 Super super cool! XMs are fascinating. Excited for the follow up papers!

[mike @mr_sdach]

@madprizm0 @AlexiGlad Awesome! Super cool experiment!

[mike @mr_sdach]

@jayden_teoh_ Combining this with zeroth order search would be super cool. Reminds me a lot of the neural thickets paper that went viral several months ago.

[mike @mr_sdach]

@AlexiGlad Are these lines commensurable? Like don’t the training steps require k forward passes so much more compute for higher k?

Super cool work! Excited for the follow up papers.

[mike @mr_sdach]

@waterloo_intern But I certainly agree that model has more capacity. But bottleneck is better training and better data. Which you can argue is both research/eng.

2/2

[mike @mr_sdach]

@waterloo_intern Point I was trying to make is that the current representations are the result of a regularized learning process built to make the model generalize. So excess model capacity for information is potentially a red herring because you’ve found the representation that works well

1/2

[The Tennis Letter @TheTennisLetter]

Image of the day.

Roger Federer sitting by himself in the Royal Box at Wimbledon.

He’s truly there just to watch some tennis.

One of the most legendary athletes is also a true tennis fan at the root of it all. ❤️

[mike @mr_sdach]

@chamath You said the opposite in your recent YouTube video.