Top Tweets for #InstructGPT

Today is the 2-year birthday of InstructGPT, the mother of all modern LLMs. The AI circle has a time dilation effect - can't believe it's been 2 yrs! InstructGPT laid out the canonical recipe of pre-training -> supervised finetuning -> RLHF, a strategy that everyone is still following till this day (with a bit of variations like DPO). InstructGPT was likely OpenAI's last paper that detailed how they train the frontier models. Looking back, I think it marked the watershed moment when LLMs finally went from an academic curiosity (GPT-3) to an impactful product (ChatGPT). Some fun facts: - InstructGPT didn't invent RLHF. In fact, the blog linked to the OG RLHF work, also done by OpenAI's team in 2017. It was conceived to solve hard-to-specify tasks in simulated robotics. RLHF asked a human annotator to give 900 binary preferences, which helped a simple "hopper" robot learn backflips in sim: https://t.co/gZRPGk6CFO - InstructGPT was published at NeurIPS 2022 in New Orleans! I was presenting MineDojo at the conference and was quite surprised to see the OpenAI poster there. - The models came in 3 sizes: 1.3B, 6B, 175B. The labelers strongly preferred Instruct-1.3B to the old, prompt-engineered GPT-3-175B. Microsoft Phi-1, one of the best-known small LMs, was also 1.3B. - InstructGPT is a master class on how to present your research. The 3-step figure is crystal clear and becomes one of the most iconic visuals in AI. The intro section has no BS and gives 8 take-home messages in bold. The discussions on limitations and bias are grounded and honest. Your weekend read: https://t.co/50Ko4pTRks