Tulsee Doshi

Sr. Director & Product Lead, Gemini Models @GoogleDeepmind. @Stanford CS. Opinions my own.

[Tulsee Doshi @tulseedoshi]

Had a great time chatting with @lennysan about finding magic in modelling, and where the PM field is going.

There is something amazing about an interviewer who helps you connect the threads of your own background in fun and interesting ways. Have a listen!

https://t.co/Gl48bATDzf

[Logan Kilpatrick @OfficialLoganK]

Introducing Gemini Decision 1, our new frontier decision model in the Gemini API.

It is $0.10 per 1M input tokens with no additional cost. With support for image, audio, video, and PDF natively, 256K context window, & 1K questions.

You can try it for free in the API!

[Tulsee Doshi @tulseedoshi]

A week into my last stretch of maternity leave after shipping our first Gemini 4 models, and finally getting a chance to agent-ify our household 🍼 What agents or automations are making your home setups meaningfully easier?

[Tulsee Doshi @tulseedoshi]

Excited to introduce the Multimodal Knowledge Agent eval — a benchmark we have been building and leveraging to test for the inherently multimodal way we do real work.

MKA evaluates across a range of visual knowledge work tasks like whether models can read a schematic or a scientific chart and turn it into a real deck, doc, or spreadsheet

Thanks to @ArtificialAnlys for the partnership and shoutout to @RohanLikesAI and @bcaine for the leadership on this

[Artificial Analysis @ArtificialAnlys]

We’ve partnered with Google DeepMind to launch their Multimodal Knowledge Agent (MKA) evaluation, a new benchmark for agentic knowledge work over complex visual files

@GoogleDeepMind’s MKA is a new benchmark requiring agents to review visually complex project files such as schematics and scans, and produce deliverables like slide decks, spreadsheets, reports, and marked-up images

Multimodal Knowledge Agent (MKA) overview:

➤ 200 private real-world visual tasks across 10 industries: Agents work from dense visual project files including architectural drawings, engineering schematics, and clinical scans to produce deliverables including slide decks, spreadsheets, PDFs, Word documents, and marked-up images. Unlike many agentic evaluations, these inputs emphasize realistic visual complexity and domain-specificity, and are genuinely required to complete the task rather than just providing additional context

➤ Climb vs. Regression rubric scoring: Each task is graded against expert rubrics, averaging ~70 weighted criteria per task. Criteria are split into ‘Regression’ (baseline checks models should pass) and ‘Climb’ criteria (harder checks designed to separate strong models). The headline score is Climb WSR (Weighted Satisfaction Rate) - the weighted proportion Climb rubric criteria successfully completed

➤ Evaluated in Stirrup with an agentic judge: We run every task in Stirrup, our open source agent harness, with code execution, web search, web fetch, and image-viewing tools (up to 250 turns), and grade submitted deliverables using GPT-6 Luna (medium) as an agentic judge

Key MKA results:

➤ Claude Fable 5.2 leads on multimodal knowledge work: Anthropic’s recently released Claude Fable 5.2 (max) leads the MKA leaderboard with a Climb WSR of 57%, followed by Gemini 4 Carbon (high) at 54% and Claude Opus 5.5 (max) at 53%

➤ Cost per Task spans nearly 30× within 10 points of the top score: Claude Fable 5.2 averages $31.97 per task, over 5× Gemini 4 Carbon ($6.29) for ~2 extra points, and ~29× GPT-6.1 Sol ($1.11). Below Carbon, all four OpenAI models tested sit on the cost-performance Pareto frontier (GPT-6 Astra, GPT-6.1 Sol, GPT-6.1 Terra, and GPT-6 Luna), with Claude Haiku 5.5 and MiMo-V2.6-Pro making up the rest of the frontier

➤ No single model leads across all 10 industry verticals: Claude Fable 5.2 leads five verticals (Software & Data, Engineering, Logistics & Mapping, Business, and Finance alongside Claude Sonnet 5.5), Claude Opus 5.5 (max) leads Education and Construction, Gemini 4 Argon (high) leads Science, Gemini 4 Carbon (high) leads Healthcare, and Claude Sonnet 5.5 (max) narrowly leads Law & Government

Thank you to the @GoogleDeepMind team for their work building this benchmark and collaborating with us to bring it exclusively to Artificial Analysis!

[Tulsee Doshi @tulseedoshi]

Gemini 4 Carbon is now rolling out to developers, enterprises, scientists, and consumers!

We built it to be a true partner for your hardest coding, research, and engineering workflows.

Here's a fun example where Carbon read research papers on how songbirds sing and built an interactive model of a bird’s vocal organ…tuning the live audio physics against real bird recordings so you can hear 6 species sing or play it like a music synthesizer.

[Sundar Pichai @sundarpichai]

Our teams put Gemini 4 Carbon to work on a major security challenge: mathematically proving the code underlying the Signal protocol.

Using AlphaProof Code, @GoogleDeepMind researchers guided agents to verify the protocol down to compiled device code, closing the gap where real bugs hide. The result:

- Gemini 4 Carbon generated over 6M lines of machine-checked Lean proofs
- years of expert manual verification compressed to just weeks
~1% of the codebase required human review (Lean automatically checked the rest)

This is helping unlock scalable formal verification for building secure software, laying the groundwork to fundamentally shift the advantage to defenders. We’re open sourcing the results for the security community.

[Google Antigravity @antigravity]

Today, we are launching a new Knowledge Work interface layout tuned for research, writing, and artifacts: same agent harness, plugins, and MCP tools, without the terminal and git clutter. Select your layout in the General tab of Settings.

Our goal with Antigravity is to help the world deliver value through products and research. Doing that always starts with building the logic, which is why software development is at the core of the product. But turning that work into real outcomes takes every role working together with a shared language and understanding.

[Tulsee Doshi @tulseedoshi]

Key takeaways from my convo with @lennysan!

When you can build anything, I think the skill becomes knowing which of the thousand possible things is actually worth building, and getting everyone pointed at it

[Lenny Rachitsky @lennysan]

My biggest takeaways from Head of Product for @Gemini and Google's other frontier AI models—Tulsee Doshi (@tulseedoshi):

1. Great PMs are more in demand than ever. “When you can build anything, building something great actually is now where most of the value is,” Tulsee said. The scarce skill is now identifying which of the thousands of possible things to build is the one worth building. Alongside that is putting that judgment into practice: aligning people and converting speed-induced chaos into a coherent direction.

2. The PM role has shifted from writing requirements to creating something with “heft.” The PM who hands a requirements doc to an engineer and calls it done is not going to do well in this new environment. The new baseline is going one step further: actually producing something of substance, be it a prototype, a thorough data analysis, or even a ready-to-ship MVP.

3. Tulsee hires PMs who have opinions about models. She wants PMs who play with models every day, have their own use cases for judging whether a model is good, and stress-test capabilities themselves. They’re also technical enough to roll up their sleeves, dig into data, and build an eval. And they’re adaptive: “The space is changing every two weeks. A roadmap you wrote two months ago, you’re gonna be looking at it again and rewriting it.”

4. To understand a new model, test it on work you care about. When a new model comes out, some people try 3D animations, others creative writing or hard financial analysis. Tulsee’s advice for evaluating new models is to pick tasks where models “are already showing signs of life” but aren’t perfect by your own definition. That leaves room to see different models handle the problem differently, and it’s where you’ll notice how the models improve.

5. Model development is less like engineering and more like alchemy. “Model training is kind of like Snape’s potions class,” she says. “You can put all the ingredients in and be thoughtful about what you expect, but you also have to expect that there will be a little bit of magic.” Her team’s job is to identify where the model shines (often in ways no one planned) and to turn that into something great for users.

6. Tulsee’s 60-20-20 rule is a practical way to build ahead of the models. Your AI product roadmap should look like this: roughly 60% of what you build should work today, another 20% should be close—“not perfect, but you can see how the models might get here in the very near future”—and the final 20% should show glimmers of something magical that the model can’t quite do yet. The exact numbers are flexible, but the idea is to be ready to double down on something that just became possible.

7. Your first eval can be five prompts. When tackling evals, the most common mistake people make is waiting until you can build something large-scale before even starting. “Start with even just five prompts,” she says. “Start simple. Start with even just yourself as the rater.” A set of even 20 prompts that creates a signal between model A and model B is “so powerful.” Getting to a small, honest set of examples forces you to articulate what “good” looks like. Get clear on a set of good examples first, then scale up.

8. Burnout comes from chaos and grunt work more than from long hours. Tulsee admits everyone on her team is “putting in all of the hours.” But in her experience, burnout comes from churn and chaos, and work that feels like a waste of time. Work you’re excited about is exhausting but brings more joy. That’s why she hopes AI tools can reduce burnout by removing the grunt work, so people spend more time thinking and being creative. My tech workforce survey found the same thing: more hours doesn’t necessarily equal burnout.

9. Google’s full-stack AI strategy gives it an advantage few competitors can match. From custom TPU hardware to products used by over a billion people each, Google operates across the entire value chain. Tulsee describes Google’s goal as building models across the “Pareto curve of cost to quality”—not just the frontier LLM but models that serve a search user, a software engineer, and a cyber defender. Frontier competition matters to Google, and Tulsee is direct about Google’s intention to win the SOTA model race.

[Google Workspace @GoogleWorkspace]

Say goodbye to language barriers. 🚀

Powered by Gemini 3.5 Live Translate, Google Meet's speech translation now supports 63 languages, translated across any combination, in real time.

With automatic speaker language detection, you just select your preferred language and speech translation does the rest. Watch here 👇 https://t.co/CFXtOk6BTY

[Josh Woodward @joshwoodward]

HBD SRK!

[Tulsee Doshi @tulseedoshi]

Just two weeks after we released Gemini 4 Argon, introducing Carbon!

Built on your feedback on Argon, Gemini 4 Carbon is SOTA for cyber capabilities, improves on coding usability & output quality, and ranks #1 on Artificial Analysis V5. We've also continued to invest in safety & security, with improved alignment and transparency.

We're starting to roll out Carbon to a subset of cyber defenders through our Fairwind Program, and we’ll continue to iterate before expanding access!

[Eli Collins @elicollins]

Genie 4 is incredible for real time world simulation. It's built on Gemini Omni and uses Gemini in the loop to maintain coherence and continuity. More coming soon! Congrats @RaiaHadsell @tejasdkulkarni @drivascos and team

[Melvin Johnson @melvinjohnsonp]

Great to see this performance for Argon and also the jump from switching to the Antigravity harness!

[koray kavukcuoglu @koraykv]

We’ve seen an incredible response to Gemini 4 Argon from users, developers & enterprises: nearly 25 trillion tokens processed in the first 72 hours after launch - 3x what Gemini 3 did.

Hear more about Argon and our commitment to the AI frontier from @sundarpichai ‘s conversation with @kantrowitz

[Logan Kilpatrick @OfficialLoganK]

In just the first 72 hours since Gemini 4 Argon launched, we processed ~25 trillion tokens, the demand for intelligence is incredible and still under appreciated!

[Varun Mohan @_mohansolo]

Gemini 4 Argon is live in Antigravity for paid Pro subscribers. As part of Pro, you get a separate weekly quota so it doesn’t eat up your Flash quota. Excited to see Antigravity push the limits of our best model yet. Love to see what you build!

[Carolina Parada @carolina_parada]

We asked #Gemini 4 Argon to explain how it figured out how to play cornhole with @Apptronik's Apollo - here it is in its own words 🤯. This is transformative for Physical AI 🦾.

Try it yourself at : https://t.co/8q0nD2IhjL
Learn more: https://t.co/a8jTjNh9z8

@GoogleDeepMind #Robotics