SamT

Making AI stuff

[Ahmad @TheAhmadOsman]

https://t.co/kvDM29hgTL

[Shane Gu @shaneguML]

I had a chance to chat with Andrej when he visited Tokyo in 2022, right after he wrapped up his 5-year work with Elon (he told me he was "recovering"). It was clear that his true passions are education and empowerment. You can see this reflected on his X feed—there is no hidden agenda, just a genuine desire to share what excites him. In a way, Andrej represents the ultimate nightmare for organizations like OpenAI. His goal is to empower developers outside of closed labs to train their own models, making AI knowledge accessible far beyond a small, elite group. If you empower enough people, closed labs will eventually run out of the capital needed to acquire all that distributed talent, and the monopoly on AI power will dissolve. This is just my speculation, but looking at it through this lens, it makes complete sense why he left OpenAI after just a year.

[Alan Karthikesalingam @alan_karthi]

Delighted to share: Prospective Study of Conversational Diagnostic AI in Ambulatory Primary Care @GoogleDeepMind @GoogleResearch @BIDMChealth – https://t.co/WIBiU8cz6M

https://t.co/hXY9ybXzpm

Like real medical training, we @GoogleDeepMind took a series of milestones on the way to performing these real-world clinical encounters: from passing medical licensing-style exams (https://t.co/gEuFWavD9A) to mirroring PCP skills in simulated consultation OSCEs (https://t.co/XPGYIx14M0).

Now, in partnership with @BIDMChealth, 100 patients engaged in a text-chat with our AI (AMIE) up to 5 days before their in-person PCP visit. AMIE took a comprehensive history and drafted a summary & differential diagnosis (DDx) for the PCP. It was carefully monitored by a physician ready to intervene based on strict safety criteria.

[SamT @st_bomba]

@neural_avb Would you recommend any specific frameworks or do you code your agents from scratch?

[Oriol Vinyals @OriolVinyalsML]

The secret behind Gemini 3?

Simple: Improving pre-training & post-training 🤯

Pre-training: Contra the popular belief that scaling is over—which we discussed in our NeurIPS '25 talk with @ilyasut and @quocleix—the team delivered a drastic jump. The delta between 2.5 and 3.0 is as big as we've ever seen. No walls in sight!

Post-training: Still a total greenfield. There's lots of room for algorithmic progress and improvement, and 3.0 hasn't been an exception, thanks to our stellar team.

Congratulations to the whole team 💙💙💙

[Google DeepMind @GoogleDeepMind]

Our first release is Gemini 3 Pro, which is rolling out globally starting today.

It significantly outperforms 2.5 Pro across the board:
🥇 Tops LMArena and WebDev @arena leaderboards
🧠 PhD-level reasoning on Humanity’s Last Exam
📋 Leads long-horizon planning on Vending-Bench 2

[Artificial Analysis @ArtificialAnlys]

Gemini 3 Pro is the new leader in AI. Google has the leading language model for the first time, with Gemini 3 Pro debuting +3 points above GPT-5.1 in our Artificial Analysis Intelligence Index

@GoogleDeepMind gave us pre-release access to Gemini 3 Pro Preview. The model outperforms all other models in Artificial Analysis Intelligence Index. It demonstrates strength across the board, coming in first in 5 of the 10 evaluations that make up Intelligence Index. Despite these intelligence gains, Gemini 3 Pro Preview shows improved token efficiency from Gemini 2.5 Pro, using significantly fewer tokens on the Intelligence Index than other leading models such as Kimi K2 Thinking and Grok 4. However, given its premium pricing ($2/$12 per million input/output tokens for <200K context), Gemini 3 Pro is among the most expensive models to run our Intelligence Index evaluations.

Key takeaways:

📖 Leading intelligence: Gemini 3 Pro Preview is the leading model in 5 of 10 evals in the Artificial Analysis Intelligence Index, including GPQA Diamond, MMLU-Pro, HLE, LiveCodeBench and SciCode. Its score of 37% on Humanity’s Last Exam is particularly impressive, improving on the previous best model by more than 10 percentage points. It also is leading in AA-Omniscience, Artificial Analysis’ new knowledge and hallucination evaluation, coming first in both Omniscience Index (our lead metric that takes off points for incorrect answers) and Omniscience Accuracy (percentage correct). Given that factual recall correlates closely with model size, this may point to Gemini 3 Pro being a much larger model than its competitors

💻 Advanced coding and agentic capabilities: Gemini 3 Pro Preview leads two of the three coding evaluations in the Artificial Analysis Intelligence Index, including an impressive 56% in SciCode, an improvement of over 10 percentage points from the previous highest score. It is also strong in agentic contexts, achieving the second highest score in Terminal-Bench Hard and Tau2-Bench Telecom

🖼️ Multimodal capabilities: Gemini 3 Pro Preview is a multi-modal model, with the ability to take text, images, video and audio as input. It scores the highest of any model on MMMU-Pro, a benchmark that tests reasoning abilities with image inputs. Google now occupies the first, third and fourth position in our MMMU-Pro leaderboard (with GPT-5.1 taking out second place just last week)

💲Premium Pricing: To measure cost, we report Cost to Run the Artificial Analysis Intelligence Index, which combines input and output token prices with token efficiency to reflect true usage cost. Despite the improvement in token efficiency from Gemini 2.5 Pro, Gemini 3 Pro Preview costs more to run. Its higher token pricing of $2/$12 USD per million input/output tokens (≤200k token context) results in a 12% increase in the cost to run the Artificial Analysis Intelligence Index compared to its predecessor, and the model is among the most expensive to run on our Intelligence Index. Google also continues to price long context workloads higher than lower context workloads, charging $4/$18 per million input/output tokens for ≥200k token context.

⚡ Speed: Gemini 3 Pro Preview has comparable speeds to Gemini 2.5 Pro, with 128 output tokens per second. This places it ahead of other frontier models including GPT-5.1 (high), Kimi K2 Thinking and Grok 4. This is potentially supported by Google’s first-party TPU accelerators

Other details: Gemini 3 Pro Preview has a 1 million token context window, and includes support for tool calling, structured outputs, and JSON mode

See below for further analysis

[SamT @st_bomba]

@akshay_pachaar Do you have a good example or repo for summary indexing?

[Crystal @crystalsssup]

I'm so proud!! The open-source trillion parameters reasoning model &lt;3
&gt; SOTA on HLE (44.9%) and BrowseComp (60.2%)

[Shubham Saboo @Saboo_Shubham_]

Vibe coding with AI Studio is so addictive!

I built an AI Hair Stylist Agent that:
&gt; find the best haircut by looking my face
&gt; generates the after images using nano banana
&gt; lets me edit it live with the live API
&gt; find nearby salons using google maps

Built in just 5 mins!

[SamT @st_bomba]

@NVIDIAGeForce GeForce Day

[Ross Taylor @rosstaylor90]

RL is not enough. It only reaches its potential when combined with other ideas.

The most famous example is AlphaZero. RL was combined with self-play which created an implicit task curriculum that evolved through training. This is very different from many RL datasets for LLMs which have a fixed set of tasks.

Even where the task set is fixed, RL still needs to be combined with other ideas to show signs of life. Thinking models only come to life through RL when we remove length penalisation and have enough prior knowledge in the training data.

Looking ahead, long horizon tasks will require much more exploration and “going off-piste”. But current RL methods induce policy entropy collapse. Diverse mid-training before RL could help, but fundamentally current RL objectives don’t reward “interestingness” and deviating from the “current (high reward) thing”.

And yet, discovery is all about deviating from the current thing - and the best ideas come from the wilderness. Deep learning is no exception. It was interesting long before it reached its true potential. We didn’t wait to introduce backprop and SGD until compute and data came online :).

This is a long way of saying: crude RL maximalism is overhyped. The magic comes from the interplay of RL with other things, and the angels are in the details.

[SamT @st_bomba]

@akshay_pachaar Good stuff! What would you recommend for multi agent fine tuning?

[Nathan Lambert @natolambert]

Thinking, Searching, and Acting
A reflection on reasoning models.

It's easy to fixate on the "thinking" that gave reasoning models their name, but just over a year out from o1-preview's release by OpenAI, the core primitives that make up models today has expanded. Searching and executing tools make up for their deficiencies as probabilistic tools with outdated information in their parameters.

Together, these three actions will act as the foundation of the systems we use for years, and the engineer aspects of them matter just as much as getting precisely the right model weights.

[SamT @st_bomba]

@Shruti_0810 MIT

[Shruti Codes @Shruti_0810]

"Mathematics for Computer Science" — MIT

This book of 1048 pages is now FREE.

A MUST for all Beginners.

To Get it:

1. Follow me (so that i can DM you
2. Repost
3. Comment "MIT"

[Lakshya A Agrawal @LakshyAAAgrawal]

In this context, GEPA works as a prompt optimizer, so the end result is a prompt (or multiple prompts for a multi-agent system, one for each component).

However, one aspect that does not get highlighted enough is that GEPA is a text evolution engine: Given a target metric, GEPA can efficiently search/evolve the right text to improve that metric. What the text represents is upto the user.

For example, In this notebook (https://t.co/8Qt5hveIAM), we use text to represent a full agent code, and GEPA ends up discovering a very sophisticated agent (that can perform self-reflection and iterative refinement on code) for ARC-AGI, improving Gemini-2.5-Pro's score by +5.5% on ARC-AGI-1.

In the paper, we also explore generating fast code kernels for hardware like GPUs/NPUs.

[SamT @st_bomba]

@akshay_pachaar 👍🏻👍🏻👍🏻

[Shubham Saboo @Saboo_Shubham_]

China's Alibaba just dropped an opensource 30B agentic LLM that outperforms Claude 4 Sonnet, DeepSeek v3.1, Kimi k2 on a range of agentic search benchmarks.

Only ~3B parameters are activated per token.

100% Opensource.

[David Sinclair @davidasinclair]

Announcing “K-Dense”, a multi-agent AI scientist that has already made a new discovery in aging research 🧵 @ashwingop &amp; @BioStateAI
https://t.co/PdUsPDFEp9