Paul Jones

Data Science, Cybersecurity, AI Security, PhD, Brit.

[Andrew Ng @AndrewYNg]

It is only rarely that, after reading a research paper, I feel like giving the authors a standing ovation. But I felt that way after finishing Direct Preference Optimization (DPO) by @rm_rafailov @archit_sharma97 @ericmitchellai @StefanoErmon @chrmanning and @chelseabfinn. This beautiful paper proposes a much simpler alternative to RLHF (reinforcement learning from human feedback) for aligning language models to human preferences.

RLHF has been a key technique for training LLMs. In brief, RLHF (i) Gets humans to specify their preferences by ranking LLM outputs, (ii) Trains a reward model (used to score LLM outputs) -- typically represented using a transformer network -- to be consistent with the human rankings, (iii) Uses reinforcement learning to tune an LLM, also represented as a transformer, to maximize rewards. This requires two transformer networks, and RLHF is also finicky to the choice of hyperparameters.

DPO simplifies the whole thing. Via clever mathematical insight, the authors show that given an LLM, there is a specific reward function for which that LLM is optimal. DPO then trains the LLM directly to make the reward function (that’s now implicitly defined by the LLM) consistent with the human rankings. So you no longer need to deal with a separately represented reward function, and you can train the LLM directly to optimize the same objective as RLHF.

Although it’s still too early to be sure, I am cautiously optimistic that DPO will have a huge impact on LLMs and beyond in the next few years.

You can read the paper here: https://t.co/m14qRYszVa I also write more about this in The Batch (linked to below).
https://t.co/8h2ag2plIa

[OpenAI @OpenAI]

Our Preparedness Team will drive technical work, pushing the limits of our cutting edge models to run evaluations and closely monitor risks, including during training runs. Results will be synthesized in scorecards that track model risk.

[Paul Jones @pj201]

After waiting for nearly 6 hours, the train finally arrived and we’re on our way to #campeche

[Paul Jones @pj201]

Anxiously awaiting a #trenmaya train here at the new #Cancun station (which still needs a bit of work) #quintanaroo #mexico

[Tren Maya @TrenMayaMX]

¡El Tren Maya va! 🚆🇲🇽🎉

[Mustafa Suleyman @mustafasuleyman]

The UK AI Safety Summit is a historic moment. Great to see *everyone* - big labs, small startups, OSS, academics and govt - focusing on genuine ai safety. @bletchleypark

[Yann LeCun @ylecun]

We can design AI systems to be both super-intelligent *and* submissive to humans.
I always wonder why people just assume that intelligent entities will necessarily want to dominate.
That's just plain false, even within the human species.

[AK @_akhaliq]

Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation

extensive evaluation across 14 popular LLMs (including GPT-4 and ChatGPT) demonstrates that HUMANEVAL+ is able to catch significant amounts of previously undetected wrong code synthesized by LLMs, reducing the pass@k by 15.1% on average! For example, the pass@k of widely studied open-source models like CODEGEN-16B can drop by over 18.0%, while the performance of state-of-the-art commercial models like ChatGPT and GPT-4 can also drop by at least 13.0%, largely affect the result analysis for almost all recent work on LLM-based code generation

abs: https://t.co/aAtNVuuguQ
github: https://t.co/DVirfd5POV

[Paul Jones @pj201]

@StarbucksUK please insist that all Starbucks outlets, even franchises, accept money preloaded in the app. Not being able to use Starbucks cash in Starbucks is just crazy.

[Gary Lineker @GaryLineker]

A final thought: however difficult the last few days have been, it simply doesn’t compare to having to flee your home from persecution or war to seek refuge in a land far away. It’s heartwarming to have seen the empathy towards their plight from so many of you. 3/4

[Oliver Cameron @olivercameron]

AI is changing everything, and it's not just ChatGPT or self-driving cars.

You'd be surprised at just how broadly AI is being applied across tons of industries.

I’ve now invested in 50 AI startups, and here are some recent investments using AI in fascinating applications 👇

[Scott Mills @scott_mills]

Anyone else up with this thing that’s like covid but not? Aches, pain. Shivers, can’t sleep RUBBISH 🤒

[Paul Jones @pj201]

Wow! Well done Argentina! 🇦🇷 🎉

[Carol Vorderman @carolvorders]

NO Jeremy Clarkson. Not on any level, in any circumstance, is it ok to write this stuff about any woman & absolutely NO to "everyone who's my age thinks the same"
No no no. We absolutely do NOT think the same.
Listen to the noise Jeremy. The crowds are chanting "shame on YOU"

[Paul Jones @pj201]

Now with giant coaster…

[Paul Jones @pj201]

Something’s looking awfully familiar about the #ITV1 rebrand 🤔

[Paul Jones @pj201]

Playing football is easy. Making a stand for what’s right is really hard and takes bravery. Iran won hands down on that. #ThreeLions

[Paul Jones @pj201]

@Toby_J_Benjamin @TigerClubUK @WLACFlying Hi Toby, thanks for getting in touch and condolences re your father Benjy - I have both his Tiger Club books here (and Michael’s). He did indeed make an incredible contribution to the club. I didn’t realize he passed away on the same day as Prince Philip, quite a coincidence

[Paul Jones @pj201]

Among many other things, Prince Philip was a light aviation enthusiast, I love this picture of him flying a Rollason Turbulent at White Waltham in 1960 (from The Tiger Club: A Tribute) #avgeek @TigerClubUK @WLACFlying

[Paul Jones @pj201]

@tortillauk I joined but I’m not getting any stamps - I bought 4 burritos in the last few days, and zero stamps awarded… what do I need to do to get a stamp??!