EvalEval Coalition

We are a researcher community developing scientifically grounded research outputs and robust deployment infrastructure for broader impact evaluations.

[EvalEval Coalition @evaluatingevals]

@3rdcultureme @evijit @IreneSolaiman @BlancheMinerva @UsmanGohar @batznerjan @prpaskov @AnkaReuel @kevin_klyman Super special thanks to @SachinPendse for making this all possible and hosting us at @UCSF 🤩

[EvalEval Coalition @evaluatingevals]

🚀 Join Evaluating Evaluations this Saturday, Oct 24, for Embedding Evaluators in Practice, a flash micro conference on embedded AI evaluators! 🎤 Come in person at UCSF Mission Bay 🌉, join our London watch party 🇬🇧, or tune in online 💻 🧵👇

[EvalEval Coalition @evaluatingevals]

Organizing team and friends: @3rdcultureme, Jennifer Mickel, @evijit, @IreneSolaiman, @BlancheMinerva, @UsmanGohar, @batznerjan, Matthew Kennedy, Aspen Hopkins, @prpaskov, @AnkaReuel, @kevin_klyman

[EvalEval Coalition @evaluatingevals]

Official Apertus blog: https://t.co/pSYm7x7auv

[EvalEval Coalition @evaluatingevals]

Apertus 1.5 Tech report is out! It is the first model release with every eval on Evaluation Cards and HF Community Evals, reproducible down to the generation settings.

48 benchmarks for the 8B, 47 for the 70B. Rerun them yourself.

Congrats Swiss AI ❤️

https://t.co/TpwCw524QT

[EvalEval Coalition @evaluatingevals]

We're bringing together third-party evaluators, model developers, policymakers, and civil society to set a shared agenda. Held under the Chatham House Rule.

🌐 Event details: https://t.co/21ul0nB85P

[EvalEval Coalition @evaluatingevals]

🚨EvalEval is excited to announce that we are hosting a microconference (hybrid) on Embedding Evaluators in Practice in SF!

More details on the agenda and outcomes in thread 🧵

Invite-only, sign up 👇
https://t.co/CjSfQ7LjZ1

[EvalEval Coalition @evaluatingevals]

That raises hard questions.

❓ What access do evaluators actually need?
❓ Who funds, selects, and validates them?
❓ What protects their ability to report unfavorable findings?
❓ When they find serious risks, who has to act?

[EvalEval Coalition @evaluatingevals]

Read the announcement: https://t.co/qicSk0iAgT

[EvalEval Coalition @evaluatingevals]

The EvalEval Coalition started almost two years ago as a small group focused on the science of AI evaluation. Today we're 800+ researchers and practitioners!

To support this growth & the community, we've adopted new bylaws and leadership structure
https://t.co/fphHfAiGlz

[EvalEval Coalition @evaluatingevals]

We're excited about what this makes possible, and look forward to supporting the growing community!

Want to get involved? [email protected]

[EvalEval Coalition @evaluatingevals]

Read the full analysis and explore the crosswalk:

https://t.co/PQNfM2s00W

[EvalEval Coalition @evaluatingevals]

New research: Mapping Evaluation Cards to Emerging AI Governance Requirements in California, the EU, and the UK.

We mapped 40 governance requirements into 281 elements and identified 60 that belong in eval reporting, showing that Eval Cards can already represent 56 of these!

[EvalEval Coalition @evaluatingevals]

We also show how Evaluation Cards fit alongside complementary standards for evaluator independence, model documentation, and incident reporting, and how this infrastructure is already being put into practice with partners such as the UKAISI, with more in the works!