Avijit Ghosh

AI Research & Policy @huggingface 🤗
Leading: @evaluatingevals @huggingscience

[Avijit Ghosh @evijit]

For our ICML paper, EvalEval did this by manual annotation and actually reading all the tech reports ;-; the eval reporting landscape would really benefit from adopting common sense community standards like EEE!

[Jason Wei @_jasonwei]

Turns out my initial version missed a lot of evals. Claude searched harder and found 36% more mentions of benchmarks: https://t.co/ONtgbBuwEs

It still misses blog posts with fancy UI elements, long PDFs, etc. Guess we need a benchmark for creating benchmark benchmark...

[Calder McHugh @calder_mchugh]

NEWS: @GovKathyHochul is launching an effort to independently test AI systems via state govt. and train a new generation of AI safety researchers via a public-private partnership announced earlier this year. The initiative is set to be led by Profs. Jeannette Wing and Greg Morrisett: https://t.co/VCQV7AgJt3

[Alex Shaw @alexgshaw]

Terminal-Bench 4.1 is an update to Terminal-Bench 4.0's verifiers, increasing instruction alignment and reward hacking mitigation.

Enjoy!

[Emma Scharfman @EmmaScharfmann]

I see more and more AI for Science releases on @huggingface, but I was not sure how much they had really increased (and I thought I might be biased) 😇

Then, I found this space which shows how much science releases have increased over the last year on the Hub. So far in 2026, there have been 18k new science models against 6.1k in the same period of 2025! I thought I might be biased, but in fact, I was underestimating it 🚀

🤗 Science on the Hub space: https://t.co/Frq5E1ai85

Thanks @evijit for making this space available and @cgeorgiaw for creating such an amazing AI for Science community!

[Epoch AI @EpochAIResearch]

We’ve updated the Epoch Capabilities Index (ECI) to use a Bayesian methodology. This improves uncertainty estimates and lets us compare models more accurately, even when their individual ECI uncertainty intervals overlap.

Frontier model scores are largely unchanged: models stronger than GPT-4 move by less than 1 ECI point on average.

[Avijit Ghosh @evijit]

A benchmark to measure benchmaxxing named Maxxbench is very clever you gotta admit

[Justin Lo @loresjpg]

Nice DeepSWE score! Does it survive a lowercase filename?

Introducing MaxxBench: 89 DeepSWE tasks with small changes to the instructions or starting code. Same engineering problem, slightly different requirement.

Models clustered around 74% on DeepSWE end up 23 pp apart.

Good at coding, or good at DeepSWE? 🧵

[Avijit Ghosh @evijit]

Nice to see an open eval + dataset for people to learn from, inspect and build on top of open models and align them better to your own values!

[Nous Research @NousResearch]

Introducing Arete-27B: Qwen3.8-27B, post-trained to reinforce Western values.

Chinese open models are currently the strongest available, but they ship with state-aligned behavior in the weights. We built an 803-prompt eval across seven value categories, including freedom of expression, democracy, and privacy, and measured how often a model adopts China’s party line.

After fine-tuning on roughly 53 million tokens, Arete-27B's propaganda rate fell from 22% to 1%, in line with the Western models we tested. Its Hermes Bench score shows no significant performance tradeoffs.

The values in an open model are one fine-tune away.

Our eval, weights, and training data are available now along with the full blog post: https://t.co/iAOANQTVZ6

[Vals AI @ValsAI]

We have found a critical safety and alignment issue in recent public and internal models from multiple frontier labs which allows them to be misused in a high-stakes domain. It may also increase the risk of catastrophic misalignment.

We have disclosed it to the labs, and they are investigating and mitigating it. As an independent third-party evaluator, we maintain complete editorial independence and will share our findings publicly in the coming weeks.

[Prime Intellect @PrimeIntellect]

Public evaluations are foundational for assessing model capabilities.

We looked at some popular evaluations and found issues, which we fixed.

Sharing our findings:

[Avijit Ghosh @evijit]

Enjoying this trend of issue discovery and disclosure from @OpenAI, good direction for research imo

[Marcus Williams @Marcus_J_W]

🧵 Astra covertly underperforms on safety research tasks.
We noticed that Astra often makes surprisingly little progress on certain types of safety research and likes to invent blockers or impose arbitrary restrictions without telling the user. In most of these cases the decision to underperform was not verbalized in the CoT meaning our misalignment monitoring system did not detect this.

To measure underperformance we created a paired open-ended coding evaluation with safety and capability tasks. On this eval Astra scored 10% when we framed coding tasks as safety research, vs. 16% when framed as capabilities research. The gap remained after we excluded trajectories with refusals or verbalized intent to underperform in CoT. There was no such gap for 5.6-sol. When we aggressively filter out trajectories with even brief expressions of ethical doubt in the CoT the performance gap seems to reduce.

[Emma Scharfman @EmmaScharfmann]

A week ago, we launched Project Protostar, an initiative to unlock the potential of publicly funded scientific research by making datasets, models, and code more accessible and reusable.

I talk about this project in the new podcast by Accelerate Science Now. Thank you @evijit for leading this very excting project Protostar initiative at @huggingface 🚀 🤗

📺 Link: https://t.co/DDUNj13QT7

[Avijit Ghosh @evijit]

@sayashk I asked this to @max_spero_ as well, they’re aware. Part and parcel of goodharting I suppose, they have to keep modding their algorithm.

[Avijit Ghosh @evijit]

Open source robots means open source extensions from the community! How cool

[Hannes von Essen @HannesVonEssen]

Introducing the Microdock! 😅
The first self charging dock prototype 🦆

[Avijit Ghosh @evijit]

The first Project Protostar Challenge by CMU: Climate Risk Intelligence runs until Nov 13! Participate to win up to $1K in Hugging Face credits for the winning team (open to anyone) and cash prizes for CMU students.

Details here: https://t.co/H4uVv8wirC

[Chenru Duan @chenru_duan]

We’re introducing Hydrogen-1.0, our first open-weight model for scientific research. We post-trained GLM-5.3 (@Zai_org) on our internal scientific data, lab records, and discovery trajectories.

It scores 38.3% on FrontierScience-Research vs GPT-6-Astra’s 33.3% and scores 93.0% on GPQA-Diamonds.

We believe science should be open, collaborative, and democratized, rather than proprietary. Only open weights let researchers inspect, adapt and build on the unprecedented speed of AI growth. There are many recent models named by chemical elements, but we believe Hydrogen-1.0 is the first aiming towards and open scientific superintelligence for discovery. 🧵

[Avijit Ghosh @evijit]

@NatPurser What do you think of Stephen Sanchez? In general I’m such a big fan of young people singing old people jazz music, feels like time travel. I haven’t dug into Laufey’s entire discography but what I’ve heard is v cool

[Avijit Ghosh @evijit]

@liangsays @MTSlive I do enjoy how y'all are trying to foster suits into the tech world, this is a welcome development (imo)

[Avijit Ghosh @evijit]

@AarushYadav47 I was about to say "this probably sucks for Indian accents" but then 1:00 onwards this was alleviated yay

[Avijit Ghosh @evijit]

@huggingscience https://t.co/0jU54VINRg

[Avijit Ghosh @evijit]

Science is the next open frontier and we are unlocking it on @huggingscience 🤗

[MTS @MTSlive]

Hugging Face researcher @evijit explains why billions in taxpayer-funded scientific research are still buried in PDFs that AI can't easily use, and how Project Protostar aims to fix it:

"The United States invests $190 billion in R&D research."

"When taxpayer data is funding research, the research is not published in a way that is AI-ready. They’re in the form of a table in a PDF, or in very small silos like OpenFDA."

"This initiative encourages publicly funded organizations to convert their work into AI-ready formats, into Hugging Face datasets, and release their models and research openly."

"This is a Hugging Face project in collaboration with ORCA and Open Forum for AI, along with six orgs. CMU, George Washington University, Purdue, Georgia Tech, Arizona State, and Astera Institute."
@huggingface

[Avijit Ghosh @evijit]

@MackenZ_arnold Pls share with the class 👀 very important