Chengrun Yang

Research Scientist at @GoogleDeepMind

[Chengrun Yang @chengrun_yang]

New preprint: Large Language Models as Optimizers (https://t.co/Hj122cGdSz)

(1/5)

[Chengrun Yang @chengrun_yang]

@quocleix @JeffDean @Sanjay_Ghemawat @OriolVinyalsML Congrats Quoc!

[Chengrun Yang @chengrun_yang]

@archanfel_anoth Good luck and wish you all the best, Juntang!

[Chengrun Yang @chengrun_yang]

@Guodzh Wishing you all the best in your next chapter!

[Chengrun Yang @chengrun_yang]

Excited be part of the team that used LLM agents as AI researchers to find better recipes for Gemini DeepThink! Thrilled to see the intelligence level it has achieved.

[Sundar Pichai @sundarpichai]

Gemini 3 Deep Think is getting a significant upgrade. We’ve refined Deep Think in close partnership with scientists and researchers to tackle tough, real-world challenges.

And it’s pushing the frontier across the most challenging benchmarks, achieving an unprecedented 84.6% on ARC-AGI-2. It also sets a new standard on Humanity’s Last Exam - 48.4% without tools.

[Jeff Dean @JeffDean]

We’ve updated Gemini 3 Deep Think to better tackle the complexity of real-world research, science, and engineering. ♊ 🚀

It achieves gold-medal standards on the written portions of the Physics and Chemistry Olympiads, building on gold-level performance at IMO and ICPC and has SOTA results on ARC-AGI-2, MMMU-Pro, and HLE. 🏅

From optimizing code to validating scientific data, we’re seeing it become a genuine partner in the discovery process.

Read more here: https://t.co/phbQ0jAkYd

[Chengrun Yang @chengrun_yang]

@yubai01 @Song__Mei @yaodong_yu @Yuf_Zh @ofirnachum @OpenAI Congratulations!

[Google DeepMind @GoogleDeepMind]

This is Gemini 3: our most intelligent model that helps you learn, build and plan anything.

It comes with state-of-the-art reasoning capabilities, world-leading multimodal understanding, and enables new agentic coding experiences. 🧵

[Google DeepMind @GoogleDeepMind]

Veo is getting a major upgrade. 🚀

We’re rolling out Veo 3.1, our updated video generation model, alongside improved creative controls for filmmakers, storytellers, and developers - many of them with audio. 🧵

[Ethan Mollick @emollick]

AIs have a bad reputation for truth, so three important findings in this paper:
1) "LLM agents can achieve superhuman rating performance" on fact checking when given access to Google!
2) Bigger models are more factual
3) LLMs are 20x cheaper than humans
https://t.co/lSSMAjoOnF

[Chengrun Yang @chengrun_yang]

We focus on long-form factuality in open domain, and so we show an entire evaluation pipeline with dataset + autorater + metric. The dataset was generated with LLMs and the autorater is an LLM agent with Google Search, demonstrating LLMs can rate themselves better than humans!

[Jerry Wei @JerryWeiAI]

New @GoogleDeepMind+@Stanford paper! 📜

How can we benchmark long-form factuality in language models?

We show that LLMs can generate a large dataset and are better annotators than humans, and we use this to rank Gemini, GPT, Claude, and PaLM-2 models.

https://t.co/A3vgEjbqTV

[anton @abacaj]

> superhuman performance
> more than 20x cheaper than a human

This is the state of AI, and it’s just getting started

[AK @_akhaliq]

Google announces Long-form factuality in large language models

Large language models (LLMs) often generate content that contains factual errors when responding to fact-seeking prompts on open-ended topics. To benchmark a model's long-form factuality in open domains, we first

[Aran Komatsuzaki @arankomatsuzaki]

Google presents Long-form factuality in large language models

- Proposes that LLM agents can be used as automated evaluators for longform factuality
- Shows that LLM agents can achieve superhuman rating performance

repo: https://t.co/rlAIFSqfTU
abs: https://t.co/L3CpeLaFpQ

[Chengrun Yang @chengrun_yang]

And we've released our code for Large Language Models as Optimizers (OPRO)! Check out https://t.co/oiSp2VkDIf.

[Chengrun Yang @chengrun_yang]

New preprint: Large Language Models as Optimizers (https://t.co/Hj122cGdSz)

(1/5)

[Xinyun Chen @xinyun_chen_]

In our work https://t.co/ZZ54i5yhtY, besides prompt optimization as our primary application, we also investigate the potential of LLMs for broader optimization problems. Interestingly, LLMs can find good solutions to some small-scale classic optimization problems; e.g., TSP. This is done with a simple meta-prompt that only contains a problem description and past optimization trajectory, w/o additional training or human-designed heuristics in the prompt!

[Chengrun Yang @chengrun_yang]

@andrew_nyu I guess so. It would be interesting to study the relationship between the best prompt and the pre-training / fine-tuning corpus.

[Chengrun Yang @chengrun_yang]

New preprint: Large Language Models as Optimizers (https://t.co/Hj122cGdSz)

(1/5)

[Xinyun Chen @xinyun_chen_]

Thanks for sharing our work (https://t.co/U0NPcuNIXZ)! Besides the huge improvement with prompts optimized by LLMs, we are also amazed by the creativity of LLMs, which continually surprise us with interesting prompts tailored to the LLM in the optimization loop!

[Chengrun Yang @chengrun_yang]

That said, there is a case you may need to worry more about overfitting: the generated instructions become even more specific to the task, or the sentence just get much longer, so that it may overfit more severely. In these cases it may still be necessary to check overfitting once in a while.

(3/3)

[Chengrun Yang @chengrun_yang]

But indeed, the found instructions on BBH overfit more than "Let's think step by step." or "Let’s work this out in a step by step way to be sure we have the right answer.": in Table 7, the "Our Acc" column has a larger training and test difference than the other columns (although our test accuracies are still often the highest). So the takeaway might be: we still overfit, but the extent of overfitting doesn't drastically increase in most of our searches.

(2/3)