Arash Alidoust

[Arash Alidoust @ArashAlidoust]

@nalidoust Where do you think the models are in regards to this in your field and more specifically in your eval?

[Arash Alidoust @ArashAlidoust]

@Jerr_Wu Very underrated point imho

[Arash Alidoust @ArashAlidoust]

@GuangyuRobert Definitely. If you have anything publicly published before Oct 8 (the date we published) that mentions you are working on this project and we missed it send it my way and we will certainly reference it. Otherwise, we will reference this work in our future related blogs for sure.

[Aiden M Sababi @AidenSababi]

Most benchmarks grade against what's already written down. In biology, the real answer key is what the cells did. 🧬

TahoeMysts from @tahoe_ai scores models against our proprietary experimental data.

📈 Mythos 5.2: 43% ➡️ 49% was an interesting observation today, a real jump, though with half the tasks still unsolved. 🧩

[Arash Alidoust @ArashAlidoust]

@nalidoust @Alidoust @tahoe_ai This is needed in every field right now

[Nima Alidoust @nalidoust]

What makes a biological reasoning task verifiable? Our team @tahoe_ai has spent the last few months on that question and landed on one conviction: for human-difficult tasks with verifiable rewards, the answer key has to come from ground-truth experimental data.

Along the way we built thousands of such environments, which we call TahoeMysts, and we use them to evaluate frontier models (among other things). Every so often a new model produces a big jump, and it is always exciting. Today is one of those days: @claudeai Mythos 5.2!

We will say more about this work in the future. In the meantime, take a look at our results today.

[Arash Alidoust @ArashAlidoust]

@GuangyuRobert Thank you @GuangyuRobert

[Samaya AI @samaya_AI]

Samaya AI is now listed in @Microsoft Marketplace.

We're bringing AI-powered investment research to Microsoft's expanding Copilot ecosystem for finance.

Explore Samaya ↓

[Nima Alidoust @nalidoust]

As promised, we are sharing another dataset through our Tara plug-in: Emerald Bay.

Emerald Bay measures how different cancer models respond to combination of drugs in 5 days (vs. 1 day, which was the case for Tahoe-100M). 5-day time course is designed to capture cell death in response to the treatment.

@tahoe_ai had previously open-sourced this dataset as part of our Rhaister release. And now you or your agents can access it through The Tara plug-in. We will release more and more datasets in the next few months.

Tara access: https://t.co/1AuW3uko4e
More about the dataset in the Rhaister paper: https://t.co/KJ0L0TvENw

[Arash Alidoust @ArashAlidoust]

@nalidoust @tahoe_ai 👏👏👏

[Samaya AI @samaya_AI]

When a model says it's 70% sure, is it right 70% of the time?

We compared Jev with two alternative decision models. Jev worked out of the box, and when it was unsure, pointed to the queries worth double-checking.

🔗 Full write-up in comments.

[Arash Alidoust @ArashAlidoust]

@iamjohnnyyu @tahoe_ai 👏👏👏

[Arash Alidoust @ArashAlidoust]

@nalidoust @Alidoust @tahoe_ai 👏👏👏

[Arash Alidoust @ArashAlidoust]

@maithra_raghu @aiDotEngineer @AshwinParan Excited for it!

[Ashwin Paranjape @AshwinParan]

We tested the most capable models through our specialized harness on the task of predicting earnings. Surprisingly (or unsurprisingly?) we’re seeing the latest frontier models (Fable 5.1, GPT-6 Astra) outperform expert analysts.

Every quarter companies report earnings, talk about the drivers and analysts try to predict key financial metrics beforehand. This makes it a very interesting numerical forecasting task: there is a ground truth, a human baseline and also drivers to understand post-hoc reasoning.

To test AI models, we built an environment where we could rewind time, provide highest quality financial information (at parity with expert analysts) and ask the models to make and explain their predictions. To do so, we started from the production Samaya harness, with all of the data sources, added a point-in-time gate for every single data source, structured or unstructured and added some expert-guided instructions on top.

Under this prediction harness, GPT-6 Astra, Claude Fable 5.1 and also Opus 5.5 are at the top with Astra beating not just expert consensus but also a harder “bias-corrected” version of it. Qualitatively, we find that the models’ ability to gather new evidence and willingness to adjust their view away from the consensus drive their wins over consensus, and lack of it drives their misses.

We’re alpha testing this in the product (reach out to collaborate!). Interestingly, the gap between the best and the worst reasoning traces is massive, we’re looking to how far we can push the financial reasoning with RL on this environment.

[Maithra Raghu @maithra_raghu]

Link to full research: https://t.co/nDzHkF0pu2

[Samaya AI @samaya_AI]

👉 https://t.co/ZSr7yuakFO

[Arash Alidoust @ArashAlidoust]

@KavehGhoreishi هم�� خبرا که تا الان برعکس اینو میگفتن؟

[Arash Alidoust @ArashAlidoust]

@GregHBurnham Apart from writing you said above, what would be your vote for the hardest question (or relatively defined task) you could ask the most frontier models right now?