Nikhil Mehta

Staff Research Scientist @GoogleDeepMind

[Dr. Datta M.D. (Radiology) building CRASHLab.in @DrDatta_AIIMS]

🔥 Gemini 3.0 vs Radiologists: RadLE Benchmark Results Are OUT!

☠️ Is it game over for Radiology? Let us find out! ⬇️

🫨 Since yesterday, Gemini 3.0 has been everywhere for crushing benchmarks. My inbox exploded asking: “But how did it do on the hardest visual reasoning benchmark in healthcare?”

So we ran it!
And here you go. 👇

➡️ Gemini 3.0 Pro on RadLE v1:

✅ 51% accuracy; first time a general-purpose model has beaten radiology residents
✅ Radiology residents: 45%
✅ Board-certified radiologists: ~83%
✅ Shows clean step-by-step reasoning in some tough cases (appendix localization, mimics ruled out, etc.)

🚀 This is the first time ever that a generalist model has crossed the trainee bar on RadLE v1!

Congratulations to @GoogleDeepMind and @Google team including @vivnat, @alan_karthi and all others for cooking this time!

Full breakdown here:

🔗 Link in comments / bio

🔥 Huge shoutout to Lakshmi, Divya, Upasana, Hakikat, Kautik & the entire #CRASHLab team at @KCDH_A for turning around in under a day.

🙌 If you are a medical AI lab and want to improve your performances and want our expert insights, reach out!

[Sundar Pichai @sundarpichai]

Introducing Gemini 3 ✨

It’s the best model in the world for multimodal understanding, and our most powerful agentic + vibe coding model yet. Gemini 3 can bring any idea to life, quickly grasping context and intent so you can get what you need with less prompting. 

Find Gemini 3 Pro rolling out today in the @Geminiapp and AI Mode in Search. For developers, build with it now in @GoogleAIStudio and Vertex AI. 

Excited for you to try it!

[Jeff Dean @JeffDean]

Introducing Gemini 2.0 Flash Thinking, an experimental model that explicitly shows its thoughts.

Built on 2.0 Flash’s speed and performance, this model is trained to use thoughts to strengthen its reasoning.

And we see promising results when we increase inference time computation!

[Jeff Dean @JeffDean]

Which model is best for turning natural language into SQL? Querying minds want to know...

Gemini models in the top 4 positions.

[Jeff Dean @JeffDean]

No half measures in Veo 2. It stands to reason and bears looking at.

[AshutoshShrivastava @ai_for_success]

Google Veo-2 vs OpenAI Sora.
Google is getting better of OpenAI this December 😀
Credit :
Veo-2 - @agrimgupta92
Sora - @AntDX316

[Mahesh Sathiamoorthy @madiator]

Excited to offer a sneak peek at what we have been working on.

Check out the LLM-AggreFact leaderboard [1] for factuality and hallucination detection, and the demo of our model that tops the leaderboard [2].

[1] https://t.co/3OxfpuBYUG
[2] https://t.co/dwEDeYvzQd

More info to come later!

[Oriol Vinyals @OriolVinyalsML]

Today we have published our updated Gemini 1.5 Model Technical Report. As @JeffDean highlights, we have made significant progress in Gemini 1.5 Pro across all key benchmarks; TL;DR: 1.5 Pro > 1.0 Ultra, 1.5 Flash (our fastest model) ~= 1.0 Ultra.

As a math undergrad, our drastic results in mathematics are particularly exciting to me!

In section 7 of the tech report, we present new results on a math-specialised variant of Gemini 1.5 Pro which performs strongly on competition-level math problems, including a breakthrough performance of 91.1% on Hendryck’s MATH benchmark without tool-use (examples below 🧵).

Gemini 1.5 is widely available, try it out for free here https://t.co/GJXW8lduNk & read the full tech report here: https://t.co/Pltp92WcNo

[Gowthami @gowthami_s]

@OriolVinyalsML @JeffDean Gemini 1.5 Pro did pretty well on our recently introduced long video understanding benchmark too! Did better than GPT-4o on the hard split. Congrats on a great model. 🎉

[Jeff Dean @JeffDean]

Gemini 1.5 Flash has really great qualities. A really good capable, natively multimodal, 1M token context window (with signup available to get access to a 2M token variant), and super lower latencies and fast response generation.

[Demis Hassabis @demishassabis]

We think of @GoogleDeepMind as the engine room of @Google in the AI era. Thrilled to share our vision at #GoogleIO incl the latest Gemini model 1.5 Flash, Project Astra our universal AI agent effort, our new gen video model Veo, Imagen 3 & lots more! https://t.co/lcsgr7ptJK

[Google DeepMind @GoogleDeepMind]

We watched #GoogleIO with Project Astra. 👀

[Nikhil Mehta @_nikhilmehta]

@YiTayML Congratulations Yi for this achievement!! Amazing vibes!

[Jeff Dean @JeffDean]

Gemini 1.5 Pro - A highly capable multimodal model with a 10M token context length

Today we are releasing the first demonstrations of the capabilities of the Gemini 1.5 series, with the Gemini 1.5 Pro model. One of the key differentiators of this model is its incredibly long context capabilities, supporting millions of tokens of multimodal input. The multimodal capabilities of the model means you can interact in sophisticated ways with entire books, very long document collections, codebases of hundreds of thousands of lines across hundreds of files, full movies, entire podcast series, and more.

Gemini 1.5 was built by an amazing team of people from @GoogleDeepMind, @GoogleResearch, and elsewhere at @Google. @OriolVinyals (my co-technical lead for the project) and I are incredibly proud of the whole team, and we’re so excited to be sharing this work and what long context and in-context learning can mean for you today!

There’s lots of material about this, some of which are linked to below.

Main blog post:
https://t.co/QAsDKXBdao

Technical report:
“Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context”
https://t.co/CTzTHNDCdo

Videos of interactions with the model that highlight its long context abilities:
Understanding the three.js codebase: https://t.co/yq7d6OSD6c
Analyzing a 45 minute Buster Keaton movie: https://t.co/adyMgDYHoK
Apollo 11 transcript interaction: https://t.co/Pqvq3Eac1R

Starting today, we’re offering a limited preview of 1.5 Pro to developers and enterprise customers via AI Studio and Vertex AI. Read more about this on these blogs:
Google for Developers blog:
https://t.co/x73Vun0kVS
Google Cloud blog:
https://t.co/OlaTW6PYGn

We’ll also introduce 1.5 Pro with a standard 128,000 token context window when the model is ready for a wider release. Coming soon, we plan to introduce pricing tiers that start at the standard 128,000 context window and scale up to 1 million tokens, as we improve the model.

Early testers can try the 1 million token context window at no cost during the testing period. We’re excited to see what developer’s creativity unlocks with a very long context window.

Let me walk you through the capabilities of the model and what I’m excited about!

[Google @Google]

Bard is becoming Gemini, and we’re launching two new experiences:

1️⃣ Gemini Advanced, which gives you access to Ultra 1.0, our most capable AI model
2️⃣ A new mobile app for easier collaboration on the go

Learn more ↓
https://t.co/40maLyUXc0

[Mahesh Sathiamoorthy @madiator]

Our work "Recommender Systems with Generative Retrieval" got accepted to NeurIPS 😊🎉

Congrats again to my co-authors @shashank_r12, @_nikhilmehta, @vqctran, @YiTayML, @jonahsamost, @Maciej_Kula, @edchi

Latest version at https://t.co/qOT30po0RX

[Demis Hassabis @demishassabis]

The phenomenal teams from Google Research’s Brain and @DeepMind have made many of the seminal research advances that underpin modern AI, from Deep RL to Transformers. Now we’re joining forces as a single unit, Google DeepMind, which I’m thrilled to lead! https://t.co/n2cpn91AOl

[Mahesh Sathiamoorthy @madiator]

Happy to share our recent work "Recommender Systems with Generative Retrieval"!

Joint work with @shashank_r12, @_nikhilmehta, @YiTayML, @vqctran and other awesome colleagues at Google Brain, Research, and YouTube.

Preprint: https://t.co/neD3SpxGJ5

#GenerativeAI
🧵 (1/n)

[Jeff Dean @JeffDean]

Bard is now available in the US and UK, w/more countries to come. It’s great to see early @GoogleAI work reflected in it—advances in sequence learning, large neural nets, Transformers, responsible AI techniques, dialog systems & more.

You can try it at https://t.co/m9D7JYTHvU

[Yann LeCun @ylecun]

LLMs are still making sh*t up.
That's fine if you use them as writing assistants.
Not good as question answerers, search engines, etc.
RLHF merely mitigates the most frequent mistakes without actually fixing the problem.