Past life in search/recsys, auto-research/evolutionary loops taught me:
1The loop is only as fast as its eval.
2The agent will exploit eval bugs: a diverged run with all scores tied got a near-top AUC.
Fix: tie-correct, 2,300x faster than sklearn.
https://t.co/Uap3KBVKPG
Introducing our new agentic version of Max: Arena’s model router that achieves frontier-level performance at 37% lower cost.
Built on 2.5M real-world agentic sessions from Agent Arena, Max learns to dynamically select the best model for each task, optimizing for both performance and cost.
Max achieves a +14.7% net improvement score in Agent Arena at a median cost of $3.27 per task, matching the performance of Claude Fable 5.1 (Max) at 37% lower cost.
Try Max today for a limited time, and dive into more technical details in the thread.
@alex_conneau In these applications a popular approach is to use a system that is trained to interact with the market (environment). RL is a popular framework to do this. We’ve tried using pretrained LM to generate policies. That didn’t work too well. Would like to see more development too.
@alex_conneau One interesting area I’ve been looking into is how LM can predict engagements in complicated dynamics like the e-commerce market. This so far seems to be an area that LLM does not do too well. There are indeed more applications of LLM that seems to be quite powerful these days.
My students keep using the term "overfit" like it has any meaning. Every model has a trillion parameters and 100% train accuracy. The barbarians have long since burned down that cathedral.
Stable-Baselines3 v1.3.0 is out!
We mostly worked on polishing the library to improve ease of use and fix bugs.
For instance, we added `sb3.get_system_info()` to help debug issues due to Python environment (see image).
Full changelog: https://t.co/theXMPYW6l
1. Today marks my one month after leaving Google. Google was great, but over time the ML space got more crowded. It became harder to make impact without competing with other teams. Upper management was mostly only chasing giant language models and was not technical enough.
5.
C. On the NLP research side, i think its really cool to work on large scale data set not in English and make big impact . This is quite important for me personally. US companies developed the best NL technologies but did not always work well outside English. The impact of