Leonard Berrada

Senior Research Scientist @GoogleDeepMind

[Oriol Vinyals @OriolVinyalsML]

Introducing Gemini 2.5 Pro Experimental! 🎉

Our newest Gemini model has stellar performance across math and science benchmarks. It’s an incredible model for coding and complex reasoning, and it’s #1 on the @lmarena_ai leaderboard by a drastic 40 ELO margin. Only a handful of model releases have leaped ahead so strongly in ELO. 📈

ELO score differences map directly to win rate: e.g. a 400 ELO difference yields a ~91% win rate. Incredible that since 1.5, just a year ago, we jumped 200 ELO (300 since 1.0).

Here’s a fun example where Gemini 2.5 Pro writes code to create an animated swarm of colorful boids swimming in a rotating hexagon. 💫

Try the model for free today in AI Studio. It’s also available to Gemini Advanced users in @geminiapp. https://t.co/Mgdp9TGlpO

Blog: https://t.co/2E7SXYhY7a

[Demis Hassabis @demishassabis]

Gemini 2.5 Pro is an awesome state-of-the-art model, no.1 on LMArena by a whopping +39 ELO points, with significant improvements across the board in multimodal reasoning, coding & STEM. You can try it out now in AI Studio https://t.co/lLpF8ToTVJ & @GeminiApp with Gemini Advanced

[Logan Kilpatrick @OfficialLoganK]

Introducing Gemini 2.5 Pro, the world's most powerful model, with unified reasoning capabilities + all the things you love about Gemini (long context, tools, etc)

Available as experimental and for free right now in Google AI Studio + API, with pricing coming very soon!

[Soham De @sohamde_]

Releasing RecurrentGemma - one of the strongest 2B-param open models designed for fast inference on long sequences and massive throughput!

Both pre-trained and IT checkpoints available + code - try them out here!

Code: https://t.co/VskJw7eo9Z
Weights: https://t.co/MvdkGhlqDS

[Samuel L Smith @SamuelMLSmith]

Announcing RecurrentGemma!
https://t.co/nR6TGrnWQK

- A 2B model with open weights based on Griffin
- Replaces transformer with mix of gated linear recurrences and local attention
- Competitive with Gemma-2B on downstream evals
- Higher throughput when sampling long sequences

[Soham De @sohamde_]

Just got back from vacation, and super excited to finally release Griffin - a new hybrid LLM mixing RNN layers with Local Attention - scaled up to 14B params!

https://t.co/FDyBXyLzAV

My co-authors have already posted about our amazing results, so here's a 🧵on how we got there!

[AK @_akhaliq]

Google presents Griffin

Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models

Recurrent neural networks (RNNs) have fast inference and scale efficiently on long sequences, but they are difficult to train and hard to scale. We propose Hawk, an RNN

[Leonard Berrada @LeonardBerrada]

Very happy to see the paper finally out! Griffin is a really strong and fast contender in the competitive LLM space. Feeling lucky to work with this world class team.

[Aleksandar Botev @botev_mg]

We present Griffin: A hybrid model mixing a gated linear recurrence with local attention. This combination is extremely effective: it preserves all the efficient benefits of linear RNNs and the expressiveness of transformers. Scaled up to 14B!
https://t.co/uptKapilDM

[Leonard Berrada @LeonardBerrada]

@judyhshen Link to the paper: https://t.co/jAdHuItNkF, and to the code: https://t.co/SGb4DTRspM

[Leonard Berrada @LeonardBerrada]

Excellent article by @judyhshen recapping our paper on differentially private image classification w/ high accuracy and low disparity. The fact that it is possible at all to obtain such low disparities was a surprise to me, and a highlight of her great internship work last year!

[Judy Shen @judyhshen]

We need to rethink the belief that all privacy-preserving models are inherently more discriminatory.

I give a high-level overview of why - in this @mtlaiethics blog post based on my summer internship work at @GoogleDeepMind https://t.co/nORIDS92fU 1/5

[Samuel L Smith @SamuelMLSmith]

ConvNets Match Vision Transformers at Scale:
https://t.co/Taf26QO82N

We scale NFNet pre-training on JFT-4B from 0.4 to 110k TPU-v4 core hours.

After fine-tuning, our largest model achieves 90.4% ImageNet Top-1, competitive with ViTs pre-trained for similar compute budgets.

1/3

[Google DeepMind @GoogleDeepMind]

Training with differential privacy (DP) prevents models from leaking sensitive training data, but it often incurs a large drop in accuracy. In recent work, our team substantially improved the performance of image classification with DP.

Read more: https://t.co/KTx30rU1K3 1/

[Google DeepMind @GoogleDeepMind]

Excited to share #NeurIPS2020 papers on efficient and tight neural network verification, based on efficient solvers for LP and SDP relaxations.

Implementations of these in JAX are also available as part of the new jax_verify library, described here: https://t.co/wh9fgit7w7

[Leonard Berrada @LeonardBerrada]

TLDR: if the loss of your deep learning task can go to zero, you might want to give ALI-G a try. It can spare you the hassle of tuning a learning-rate schedule. Overall, we hope that this is a useful step towards easier and more reliable optimisation algorithms for deep learning.

[Leonard Berrada @LeonardBerrada]

This week at #ICML2020 , I'll be presenting Adaptive Learning-rates for Interpolation with Gradients, aka ALI-G. ALI-G is designed to automatically adapt the learning-rate of SGD for deep learning when the loss can go to zero.

[Leonard Berrada @LeonardBerrada]

We provide experiments on a variety of architectures (Differentiable Neural Computer, ResNets, bi-LSTMs) and datasets (SNLI, SVHN, CIFAR-10/100, ImageNet). More details in the paper: https://t.co/j16wbAaRi8 (arxiv version to be updated soon as well).

[Leonard Berrada @LeonardBerrada]

@ID_AA_Carmack This can be seen as an example of a more general trade-off in optimization: how "good" the descent direction vs how easy it is to compute

[Leonard Berrada @LeonardBerrada]

@ID_AA_Carmack Then one can locally model the problem up to second order, which takes into account some level of interaction between layers. This yields Newton-type methods, but these are more computationally expensive