Emmy Probasco

National security, tech, and the Oxford comma. RT isn't endorsement.

[Emmy Probasco @EmmyProbasco]

Thanks for the nuanced breakdown of the issues with distillation @KyauMill21! Looking forward to seeing the full paper!

[Kyle Miller @KyauMill21]

On the input side of things, researchers typically used the training sets of popular benchmarks as “seeds” to elicit knowledge from the teacher (e.g. they tell the teacher to expand on or enrich the data they provide it). It is unclear if Chinese labs are using this method for distillation, and my hunch is that they are doing much more than inputting benchmark questions as seed knowledge.

3/ Distillation vs. other training methods. Another issue is that some literature does more than just distillation. For example, the Phi-1.5 paper distills using billions of teacher-generated tokens, but it also trains the student on 6B tokens of “textbook-quality” data from the web, which was not generated by the teacher. So it's harder to know how much the distillation of teacher-generated data influenced the student’s performance.

4/ Benchmark issues. Using benchmarks to evaluate the effectiveness of distillation can be problematic. While distillation provides uplift for student models on benchmarks, it may not truly reflect the degree to which the teacher’s general knowledge and capabilities are transferred to the student. Moreover, some of the distillation literature runs the risk of benchmaxxing, as the distillation methods they employ focus very heavily on enriching and fine-tuning on benchmark data.

5/ Distillation in context. My last point, and perhaps one of the most important, is that Chinese labs’ distillation efforts are one piece of a much broader post-training pipeline. It’s hard to know how much distillation provides uplift relative to all of the other methods they employ to optimize model performance. This is another reason why it’s so easy to overstate or understate the role distillation plays in China’s overall competitiveness.

6/ What to do. None of this is to suggest that we cannot know the effectiveness of Chinese distillation, rather it's that the literature only paints half of the picture. There’s likely some uplift, but we cannot yet quantify it reliably.

We need more research here. We need to get a better understanding of how distillation scales, and the degree to which it provides uplift for strong student models on the most challenging benchmarks. Without this, we really can’t know how much distillation can help Chinese labs

The level of urgency here really does, in my view, boil down to this core question of ‘how much knowledge and capability can you effectively distill from frontier proprietary models.’ If it's a minor uplift, then maybe we can accept it as a natural byproduct of API access, or apply defenses in ways that actually match the threat. If it's a major uplift that really helps the Chinese labs compete, then maybe more aggressive defenses and policies need to be pursued.

[Sam Bresnick @SamBresnick]

.@colemcfaul and I have documented the Chinese military's interest in @nvidia chips; it's clear that H200s will contribute to the PLA’s modernization, either through direct purchases or through the use of LLMs trained on them. Link below.
@CSETGeorgetown @emergingtechobs

[Cole McFaul @colemcfaul]

NEW @CSETGeorgetown + @emergingtechobs piece!

Does China's access to US semiconductor technology help the PLA develop and deploy military AI?

After 3 years reading thousands of PLA procurement docs, @sambresnick and I say yes.

Here’s how, and why it matters:

🧵/13

[Emmy Probasco @EmmyProbasco]

This will be the thing that thaws the relationship with DOD/DOW: a better way to do PowerPoint with Claude 😂 https://t.co/kbSWwMilDL

[Emmy Probasco @EmmyProbasco]

Now we have to find a way to hang on to the innovation as MSS transitions to a program of record...
https://t.co/qcPswVJoGX

[Emmy Probasco @EmmyProbasco]

Many great points in @HerbLinCyber new piece: "On Optimism About New Military Technologies"

His recommendation at the end about user-driven innovation is especially compelling. I've seen the user-driven innovation in Maven Smart System and it is really impressive.

[Helen Toner @hlntnr]

Amazing role for anyone interested in helping policymakers make sense of the fast-moving, confusing world of frontier AI—what's real, what's overblown, what do we need to be prepared for, and how do we prepare?

Come lead & grow a new team at CSET!

Reposts appreciated 🙇

[Foreign Affairs @ForeignAffairs]

“China is intent on developing AI to gain military advantage over the United States,” write @SamBresnick, @EmmyProbasco, and @colemcfaul. To stay ahead of Beijing, Washington must “scale the AI systems that give it a battlefield advantage.”
https://t.co/lMD3CYqyAk

[CSET @CSETGeorgetown]

🚨 We're Hiring! 🚨

CSET is looking for the right person to build and lead our Frontier AI team! Ideal candidates bring deep expertise in frontier AI, large-scale model development, compute infrastructure, or China's AI policy ecosystem.

Apply below!
https://t.co/Oi6nBMjxKl

[Foreign Affairs @ForeignAffairs]

“China is intent on developing AI to gain military advantage over the United States,” write @SamBresnick, @EmmyProbasco, and @colemcfaul. To stay ahead of Beijing, Washington must “scale the AI systems that give it a battlefield advantage.”
https://t.co/lMD3CYqyAk

[Andrew Curran @AndrewCurran_]

'Congress should ensure that the appropriate agencies within the national security enterprise possess sufficient technical capacity to understand frontier Al model capabilities and any associated national security considerations and establish plans to mitigate potential concerns, including through consultation with frontier Al model developers.'

[CSET @CSETGeorgetown]

CSET's @Lauren_A_Kahn on @NPR: "The US is clearly internalizing some of the lessons that we've seen [from the war in Ukraine], that being the first real drone war, the first real AI war that we've seen."

Listen to the full interview: https://t.co/sCNkgMZxYv

[New York Magazine @NYMag]

Can the military prevent Claude, OpenAI, or another company from going full Terminator? Emelia Probasco, an expert on artificial intelligence in warfare, takes a less apocalyptic view. https://t.co/qeBjFoJi5V

[Foreign Affairs @ForeignAffairs]

“The PLA is fostering an ecosystem for rapid AI development that connects novel research with frontline operations,” write @SamBresnick, @EmmyProbasco, and @colemcfaul. “The United States, meanwhile, has declared the AI company Anthropic a supply chain risk.”
https://t.co/yJ11sXQEDZ

[Emmy Probasco @EmmyProbasco]

My old French teacher would be so disappointed in me, but I still found it neat to speak with France TV2 (in English).

[France TV Washington @F2Washington]

Analyse des renseignements, choix des cibles... L'intelligence artificielle joue un rôle clé dans la guerre en Iran. Son utilisation par le gouvernement américain suscite de nombreuses questions.

📺 @DianeSchlienger @Zeineb_boughzou @Marinezamb @Johanna_Beeckm

[U.S. Central Command @CENTCOM]

Update from CENTCOM Commander on Operation Epic Fury:

[Dean W. Ball @deanwball]

I am happy with how this podcast discussion with @ezraklein turned out, a happy medium between policy analysis and profound agi-pilledness. Ezra is a spectacular interviewer; I gained respect for him after this discussion, and I already respected him.

[Foreign Affairs @ForeignAffairs]

“Leaders across academia, industry, and government should prepare for a future of widespread AI-enabled deception that threatens civilians and militaries alike,” write @SamBresnick, @EmmyProbasco, and @colemcfaul.
https://t.co/lMD3CYqyAk

[Foreign Affairs @ForeignAffairs]

Ensuring U.S. national security requires bolstering partnerships with the world’s leading experts on AI technology, write @SamBresnick, @EmmyProbasco, and @colemcfaul. “This is, at least in part, why the failure of negotiations with Anthropic is so concerning.” https://t.co/qgVI2eS0El

[Foreign Affairs @ForeignAffairs]

Read @SamBresnick, @EmmyProbasco, and @colemcfaul on China’s efforts to integrate artificial intelligence into its military—and how the United States should respond:
https://t.co/lMD3CYqyAk