Vincent Perot

gemini long context @GoogleDeepMind

[Manos Koukoumidis @Koukoumidis]

Enterprise AI is in a wildly paradoxical state 🤔, and we’re reaching the inflection point that will resolve it. 💥

Enterprises want to differentiate with AI, yet rent the same intelligence as their competitors.

Their workflows and expertise are highly specialized, yet they rely on generic models built to be good at everything.

They worry about AI costs, yet pay premium prices for massive models where only a fraction (1%) of the intelligence is relevant to their task. 💸

And they demand control and sovereignty, yet rent the intelligence becoming core to their business.

This is not a sustainable equilibrium.

The next era of enterprise AI is specialized intelligence companies build, own, and compound. And we are at the inflection point of this transition.

That’s the bet we made when we started @oumi_ai two years ago.

Today we’re closing the loop: Oumi can now not only automatically build your specialized AI models, but also deploy them into production, learn from their production experience, and continuously improve them.

The intelligence that your business runs on, becomes your differentiator. Your compounding advantage.

The winners of the next AI era will turn their own data, expertise, and experience into specialized intelligence that nobody else can rent.

Don’t rent your AI. Build it. Own it. Compound it.

[Sundar Pichai @sundarpichai]

🤔🤔

[Kiran Vodrahalli @kiranvodrahalli]

I'm happy to announce that we are open sourcing the MRCR v2 datasets we report on Gemini releases!

https://t.co/Bvi5PxL2QY

[Dillon Uzar @DillonUzar]

Context Arena Update: Added @GoogleDeepMind's Gemini 3 Flash Preview [12-17] to the OAI-MRCR leaderboards (2-, 4-, 8-needle)!

This sets a new bar for the efficiency tier. With reasoning set to High, Gemini 3 Flash is effectively matching and at ultra long context even beating its big brother Gemini 3 Pro!

On the harder 8-needle test, Flash hits 71.6% AUC @ 128k (Rank #3), near tying with Pro (73.0%). But at the 1M mark? Flash actually pulls ahead to take the top spot, scoring 49.4% AUC (Rank #1) while Pro drops to 39.0%.

I will be replying with the Minimal reasoning tests very soon (to be the "no-reasoning" version) to see the baseline capability. I've also been testing Medium/Low, and might release if there is a clear difference, stay tuned.

gemini-3-flash-preview:thinking results:

4-Needle Performance (@ 128k / @ 1M):
AUC: 85.5% (Rank #5, vs Gem 3 Pro: 85.8%) / 68.0% (Rank #1, vs Gem 3 Pro: 57.3%)
Pointwise: 83.1% (vs Gem 3 Pro: 80.8%) / 51.5% (vs Gem 3 Pro: 34.3%)

8-Needle Performance (@ 128k / @ 1M):
AUC: 71.6% (Rank #3, vs Gem 3 Pro: 73.0%) / 49.4% (Rank #1, vs Gem 3 Pro: 39.0%)
Pointwise: 51.5% (vs Gem 3 Pro: 54.2%) / 32.6% (vs Gem 3 Pro: 24.5%)

Full data: https://t.co/gLEWzxpvMe

Congrats to @GoogleDeepMind! Enjoy.

NOTE: I have some updated Gemini 3 Pro scores, fixing some API error results, however, this doesn't change this post.

@GoogleDeepMind @googleaidevs
@OpenAI @OpenAIDevs
@Kimi_Moonshot
@xai

[Kiran Vodrahalli @kiranvodrahalli]

Can 3 Flash catch up to 3 Pro w/a headstart in Crystal? Come through!! https://t.co/KwUpAkBxKS

[Google DeepMind @GoogleDeepMind]

Gemini 3 Flash gives you frontier intelligence at a fraction of the cost. ⚡

Here’s how it’s built for speed and scale 🧵

[Dillon Uzar @DillonUzar]

Context Arena Update: Added Gemini 3.0 Pro Preview (Thinking, 11-18) to the MRCR leaderboards. It establishes a new state-of-the-art in context performance, taking the #1 spot on all our AUC leaderboards and for nearly all pointwise scores.

All results at: https://t.co/gLEWzxoXWG

The 2-needle results are incredible, maintaining a 99%+ pointwise score at <=128k tokens and a strong 72% at 1M. Even on the difficult 8-needle test, it achieves an impressive 54% pointwise score at 128k.

The performance curve is interesting: on 2-needle, it's a nearly flat line of near-perfect recall up to 128k. On harder tests, the degradation slope steepens past 128k, with a clear performance shift in the 128k-256k range (likely around the 200k mark seen in prior Gemini models).

It dethrones the previous champions: `openai/gpt-5:thinking` at 128k and the top `google/gemini-2.5` models at 1M.

2-Needle Performance (@ 128k / @ 1M):
- AUC: 99.4% (vs 96.7%) / 81.2% (vs 78.3%)
- Pointwise: 99.0% (vs 95.0%) / 72.2% (vs 68.1%)
(going to have to retire 2-needle soon)

4-Needle Performance (@ 128k / @ 1M):
- AUC: 84.7% (vs 74.1%) / 49.9% (vs 49.5%)
- Pointwise: 80.9% (vs 70.6%) / 34.3% (#2, behind Gem 2.5 Flash Thinking)

8-Needle Performance (@ 128k / @ 1M):
- AUC: 67.8% (vs 50.3%) / 34.5% (vs 28.0%)
- Pointwise: 54.2% (vs 40.0%) / 24.5% (#2, behind Gem 2.5 Flash)

A significant leap over all prior models, establishing clear dominance in AUC performance across all context lengths and difficulties. Congrats to the @GoogleDeepMind team!

Enjoy.

@GoogleDeepMind @googleaidevs @sundarpichai @demishassabis
@OpenAI @OpenAIDevs

[Sundar Pichai @sundarpichai]

Incredible milestone: an advanced version of Gemini 2.5 Deep Think achieved gold-medal performance at the ICPC World Finals, a top global programming competition, solving an impressive 10/12 problems. Such a profound leap in abstract problem-solving - congrats to @googledeepmind!

[Vincent Perot @vincentperot]

@HaritzPuerto Congrats Haritz!

[Jeff Dean @JeffDean]

A powerful features of our Gemini models since Gemini 1.5 (including 2.0 and 2.5 models) is their powerful long context capability (2M tokens serving in production). You can process ~1000 pages of text, hrs of video, 10+ hrs ofaudio. Come see @SavinovNikolay discuss this!

[Google Gemini @GeminiApp]

📣 Deep Research is now powered by Gemini 2.5 Pro, our most intelligent AI model. ✨

This upgraded Deep Research is now even better at:

🔍 Finding & synthesizing information
📊 Providing more insightful reports
🧠 Analytical reasoning

Gemini Advanced users can access the new Deep Research across the web, Android, and iOS devices to generate detailed, easy-to-read reports on just about any research topic: https://t.co/zQz2CmIzih

[Jeff Dean @JeffDean]

🥁Introducing Gemini 2.5, our most intelligent model with impressive capabilities in advanced reasoning and coding.

Now integrating thinking capabilities, 2.5 Pro Experimental is our most performant Gemini model yet. It’s #1 on @lmarena_ai leaderboard. 🥇

[Justin Chih-Yao Chen @cyjustinchen]

🚨 Reverse Thinking Makes LLMs Stronger Reasoners

We can often reason from a problem to a solution and also in reverse to enhance our overall reasoning. RevThink shows that LLMs can also benefit from reverse thinking 👉 13.53% gains + sample efficiency + strong generalization!

-- We train a student LLM to generate forward reasoning and backward question from a question, and backward reasoning from backward question using a joint objective.

-- Across 12 datasets on commonsense, math, logical reasoning and NLI, RevThink shows an average 13.53% improvement over the student LLM’s zero-shot performance and a 6.84% improvement over the strongest knowledge distillation baselines.

-- Using only 10% of the correct forward reasoning from the training data, RevThink outperforms a standard fine-tuning method trained on 10x more forward reasoning.

-- RevThink also exhibits strong generalization to 4 out-of-distribution held-out datasets.

🧵

[Vincent Perot @vincentperot]

@HaritzPuerto Thank you Haritz!

[Vincent Perot @vincentperot]

Very happy to have contributed to this work with an amazing team! Check it out!

[Jinhyuk Lee @leejnhk]

Can long-context language models (LCLMs) subsume retrieval, RAG, SQL, and more?

Introducing LOFT: a benchmark stress-testing LCLMs on million-token tasks like retrieval, RAG, and SQL. Surprisingly, LCLMs rival specialized models trained for these tasks!

https://t.co/FFyI0rMCqw

[Sebastian Riedel (@[email protected]) @riedelcastro]

"just put the corpus into the context"!

Long context models can already match or beat various bespoke pipelines and infra in accuracy on non-trivial tasks! Hadn't expected this so soon, and honestly was hoping to milk RAG impact for a little longer 🤪

[Zifeng Wang @ZifengWang315]

Looking to tailor instruction tuning data to customize your own LLM? Check out CodecLM @NAACL'24 - an end-to-end data synthesis framework designed to generate high-quality data you need to align LLMs for different downstream tasks.

Paper: https://t.co/DgYCWdZBfg

[elvis @omarsar0]

Prompting LLMs for Table Understanding

A new framework to improve understanding of tabular information with LLMs.

Inspired by Chain-of-Thought prompting, it instructs LLMs to dynamically plan a chain of operations that transforms a complex table to reliably answer the input question.

An LLM is used to iteratively generate operations, step-by-step, that will perform necessary transformations to the table (e.g., adding columns or deleting info).

Looks like a nice approach to improve the reasoning, accuracy, and reliability of LLMs when dealing with more structured information like tables. Looks very promising.

I wonder how this would look for graph structures? Or there is probably a paper out that I missed.

[Quoc Le @quocleix]

XLNet: a new pretraining method for NLP that significantly improves upon BERT on 20 tasks (e.g., SQuAD, GLUE, RACE)

arxiv: https://t.co/C1tFMwZvyW

github (code + pretrained models): https://t.co/kI4jsVzT1u

with Zhilin Yang, @ZihangDai, Yiming Yang, Jaime Carbonell, @rsalakhu

[Alicia Jessop @RulingSports]

This is why we can’t have nice things. I was trying to take a picture of the lobster roll I ordered in Maine and well, this happened 😂🤦🏻‍♀️