Scientific American has named ODBI’s Kaiyi Jiang to its inaugural list of Young American Scientists. Congrats, @idmjky!
Jiang will join ODBI next month. Read more on the ODBI site: https://t.co/nziKegQQqm
1/8) Excited to share PR²: Predictive Routing Replay for MoE-Based LLM RL! 🎉
This is also the first paper from our research group at @RutgersCS@RutgersU.
While MoE LLMs scale remarkably well, RL training exposes a hidden source of instability: routing.
Paper: https://t.co/uekygKHSON
A special shout-out to my talented PhD student, @DaizeDongCS, for leading this project and driving many of its key ideas and experiments.
@ChengleiSi Hard agree! If NanoGPT speedrun and Karpathy’s auto research has taught us anything, it’s that we still need major breakthrough(s) for agents to discover paradigm shift ideas.
@karpathy We worked on building an evolutionary agent for the NanoGPT benchmark back in October and shared our findings in the paper: https://t.co/C8CxWBIGKH
Similarly, we also observed that the agent is really good at tuning hyperparameters, designing context / lr / decay schedules!
@_ScottCondron@karpathy Shameless self-plug here: we’ve worked on the very task of self-evolution on the NanoGPT benchmark in our paper!
https://t.co/y0jMrWNyni
We even deployed PACEvolve on the Modded NanoGPT challenge. Despite the benchmark being heavily optimized by the community, PACEvolve discovered further gains in data loading, network initialization, and tuned better hyperparameters.
🚀 Thrilled to introduce PACEvolve: Enabling Long-Horizon Progress-Aware Consistent Evolution.
We show how to push LLM self-evolution beyond short, unstable improvements and into consistent, long-horizon gains. 🧵👇
This work was done during my internship at Google and would not have been possible without my mentors and collaborators across Google and DeepMind. Kudos to everyone involved!
Paper: https://t.co/yYL82IjYFx
Code drop coming soon, stay tuned!
We even deployed PACEvolve on the Modded NanoGPT challenge. Despite the benchmark being heavily optimized by the community, PACEvolve discovered further gains in data loading, network initialization, and tuned better hyperparameters.
@eliebakouch agree! we tried it in our paper and it has been an awesome experience learning about both the capabilities and the limitations of current frontier LLMs.
We even deployed PACEvolve on the Modded NanoGPT challenge. Despite the benchmark being heavily optimized by the community, PACEvolve discovered further gains in data loading, network initialization, and tuned better hyperparameters.
🚀 Thrilled to introduce PACEvolve: Enabling Long-Horizon Progress-Aware Consistent Evolution.
We show how to push LLM self-evolution beyond short, unstable improvements and into consistent, long-horizon gains. 🧵👇