As a child, Hannah Cairo learned math by taking online lessons from Khan Academy. By the time she was 14, she had taught herself the equivalent of an advanced undergraduate math degree.
https://t.co/gTOeUIa9cZ
Happy birthday to Caltech professor Katie Bouman! As a PhD student at MIT she was part of the team that helped create the first-ever image of a black hole.
I just put up a new video on the fundamentals of quantum computing, building up to a step-by-step walk-through of something known as Grover’s algorithm. Here's an excerpt of a quiz used to introduce the topic.
ProGen3: scaling protein language model data and parameters improves the quality of generations, especially further away from natural sequences.
@AadyotB@jeffruffolo@thisismadani
60 years ago this month, the Fast Fourier Transform was created, a powerful tool for image compression & data analysis.
Watch a classic MIT breakdown of FFT, perhaps the most-taught algorithm at the Institute: https://t.co/R7zdspBswx
v/@MITOCW
First draft online version of The RLHF Book is DONE. Recently I've been creating the advanced discussion chapters on everything from Constitutional AI to evaluation and character training, but I also sneak in consistent improvements to the RL specific chapter.
https://t.co/2oveIOpdCD
RLHF has a long future ahead of it and this will do a lot to make it more accessible to the next generation.
What's next: Getting a physical copy in your hands (may not be exactly 1to1, we'll see) and minor fixes at a slower cadence (thanks to many github contributors, some of you will get a copy from me).
Here are all the chapters.
1.Introduction: Overview of RLHF and what this book provides.
2.Seminal (Recent) Works: Key models and papers in the history of RLHF techniques.
3.Definitions: Mathematical definitions for RL, language modeling, and other ML techniques leveraged in this book.
4.RLHF Training Overview: How the training objective for RLHF is designed and basics of understanding it.
5.What are preferences?: Why human preference data is needed to fuel and understand RLHF.
6.Preference Data: How preference data is collected for RLHF.
7.Reward Modeling: Training reward models from preference data that act as an optimization target for RL training (or for use in data filtering).
8.Regularization: Tools to constrain these optimization tools to effective regions of the parameter space.
9.Instruction Tuning: Adapting language models to the question-answer format.
10.Rejection Sampling: A basic technique for using a reward model with instruction tuning to align models.
11.Policy Gradients: The core RL techniques used to optimize reward models (and other signals) throughout RLHF.
https://t.co/IJ0TeZ6hps Alignment Algorithms: Algorithms that optimize the RLHF objective directly from pairwise preference data rather than learning a reward model first.
13.Constitutional AI and AI Feedback: How AI feedback data and specific models designed to simulate human preference ratings work.
14.Reasoning and Reinforcement Finetuning: The role of new RL training methods for inference-time scaling with respect to post-training and RLHF.
15.Synthetic Data: The shift away from human to synthetic data and how distilling from other models is used.
16.Evaluation: The ever-evolving role of evaluation (and prompting) in language models.
17.Over-optimization: Qualitative observations of why RLHF goes wrong and why over-optimization is inevitable with a soft optimization target in reward models.
https://t.co/BoIMqWyK9h and Information: How RLHF is often underestimated in its role in improving the user experience of models due to the crucial role that style plays in information sharing.
19.Product, UX, Character: How RLHF is shifting in its applicability as major AI laboratories use it to subtly match their models to their products.
Our December 2024 free and open databases https://t.co/NB0uzLwxFG are now available for download, bringing the total number of rated games to over 6.2 billion in standard chess and 125 million across variants!