Hamidreza Ghader

Co-founder at Vinder, PhD in Computer Science

[Marzieh Fadaee @mziizm]

Very proud to share today we’re releasing Tiny Aya✨🤏: small enough to run on your phone, strong enough to support 70+ languages.

Proof that multilingual progress comes from intentional design --- and evaluation that measures balance, not just peaks 🕯️

[Marzieh Fadaee @mziizm]

I'm excited to share that I'll be stepping into the role of Head of Cohere Labs. It's an honor and a responsibility to lead such an extraordinary group of researchers pushing the boundaries of AI research.

[Paul Graham @paulg]

"In the hospital there are kids maimed by airstrikes: missing arms, missing legs, third degree burns. Often there’s not enough pain medication. But the children are not screaming about the pain, they’re screaming: 'I'm hungry! I’m hungry!'"

[David Kurten @davidkurten]

2003: "Iraq has WMDs"
2025: "Iran has WMDs"
Same propaganda. Same program. Same warmongers.

[Bernie Sanders @BernieSanders]

Netanyahu started this war by attacking Iran.

He assassinated Ali Shamkhani, Iran’s lead nuclear negotiator, deliberately sabotaging US-Iran nuclear negotiations.

The US must not be dragged into another illegal Netanyahu war – either militarily or financially.

[Assal Rad @AssalRad]

“Genocide is self-defense and starting a bunch of wars is regional stability.”

-European proverb

[Philippe Lemoine @phl43]

We have spent the past 3 years lecturing Russia about international law, but Israel launches a war against Iran, assassinates scientists and bombs civilian infrastructure, but she doesn't say a word about it and only talks about Iran. We are a fucking joke, it's truly pathetic.

[Zarah Sultana MP @zarahsultana]

The Iraq playbook is back & the media won’t question it.

Iran has no nuclear weapons. Israel — on trial for genocide — does. Iran signed the NPT & allows IAEA inspections. Israel hasn’t — & its leaders face ICC arrest warrants.

Beware of politicians beating the drums of war.

[Claudia Webbe @ClaudiaWebbe]

Israel is a rogue state. It’s a barbaric state.

Gaza, Lebanon, Syria, Iran. Israel is out of control

Israel is the single biggest threat to world peace. Arrest Netanyahu

[Marzieh Fadaee @mziizm]

Following our recent research publication, we engaged extensively with the Chatbot Arena team about their methodological observations.

Our detailed responses were not reflected in their blog post. See our full context at:
https://t.co/vOHxM9um3P

[Sara Hooker @sarahookr]

Following release of our recent work, we have spent considerable time engaging with @lmarena_ai over last week.

The organizers had concerns about the correctness of our work on the reliability of chatbot arena rankings.

[Aidan Gomez @aidangomez]

The best way forward for LmSys is to acknowledge the failures their prior system allowed (rather than denying them) and restructure processes to protect against them.

Attacking Sara is going to dig a much deeper hole, and burn a lot more trust. Hopefully they understand that.

[Sara Hooker @sarahookr]

We recently released a paper where I took on a more visible role than typical.

This was a deliberate choice to protect more junior leads given we anticipated more scrutiny than typical.

However, this isn’t an invitation to channel frustrations towards me as a person.

[Andrej Karpathy @karpathy]

There's a new paper circulating looking in detail at LMArena leaderboard: "The Leaderboard Illusion"
https://t.co/tVMrx68zwa

I first became a bit suspicious when at one point a while back, a Gemini model scored #1 way above the second best, but when I tried to switch for a few days it was worse than what I was used to. Conversely as an example, around the same time Claude 3.5 was a top tier model in my personal use but it ranked very low on the arena. I heard similar sentiments both online and in person. And there were a number of other relatively random models, often suspiciously small, with little to no real-world knowledge as far as I know, yet they ranked quite high too.

"When the data and the anecdotes disagree, the anecdotes are usually right." (Jeff Bezos on a recent pod, though I share the same experience personally). I think these teams have placed different amount of internal focus and decision making around LM Arena scores specifically. And unfortunately they are not getting better models overall but better LM Arena models, whatever that is. Possibly something with a lot of nested lists, bullet points and emoji.

It's quite likely that LM Arena (and LLM providers) can continue to iterate and improve within this paradigm, but in addition I also have a new candidate in mind to potentially join the ranks of "top tier eval". It is the @openrouter LLM rankings:
https://t.co/N1NCZyVCv3
Basically, OpenRouter allows people/companies to quickly switch APIs between LLM providers. All of them have real use cases (not toy problems or puzzles), they have their own private evals, and all of them have an incentive to get their choices right, so by choosing one LLM over another they are directly voting for some combo of capability+cost. I don't think OpenRouter is there just yet in both the quantity and diversity of use, but something of this kind I think has great potential to grow into a very nice, very difficult to game eval.

[Marzieh Fadaee @mziizm]

1/ Science is only as strong as the benchmarks it relies on.

So how fair—and scientifically rigorous—is today’s most widely used evaluation benchmark?

We took a deep dive into Chatbot Arena to find out. 🧵

[Hamidreza Ghader @hrghader]

- Filtered qrel-based collections
- OR-QuAC (retrieval-focused QuAC)

Would love to hear my network suggestions! Drop your recommendations below.

[Hamidreza Ghader @hrghader]

I find the TREC CAsT and iKAT doc/passage collections too large for automatic evaluation of conversational search systems, especially with limited hardware.

Are there high-quality, lightweight datasets made for conversational retrieval evaluation?

So far, I’ve considered:

[Marzieh Fadaee @mziizm]

You can play with Aya Vision everywhere!🪻

WhatsApp: https://t.co/dBTSkGgNuW
HF space: https://t.co/oLgXkmIdX7
Cohere playground: https://t.co/AUPPcCVBx7
Poe: https://t.co/BiUXIQkI2W

[Marzieh Fadaee @mziizm]

✨👓Aya Vision is here👓✨

A multilingual, multimodal model designed to understand across languages and modalities (text, images, etc) to bridge the language gap and empower global users!

[Marzieh Fadaee @mziizm]

Look at the amazing @sarahookr on this cover 🤩 I am so proud to be working with her. She inspires us every day!