Can today’s LLMs generate rubrics that reliably evaluate tomorrow’s smarter models? Automated rubric generation can save enormous expert effort, but prior work didn’t evaluate responses from LLMs released after the rubric-generator model.
Our new paper investigates how to best scale test-time compute for this, developing a new rubric generation pipeline called Rubricon. Rather than just relying on LLM self-critique, Rubricon grounds rubric-refinement in how candidate rubrics actually score example responses.
Compared to 4 other synthetic rubric generation methods across 5 diverse benchmarks, Rubricon rubrics consistently score model responses more like human experts do [1/n]
I went through every data vendor that launched in the last 3 weeks. Almost all of them are lying about their revenue, their lab partnerships, and their backgrounds.
(skip to 1:04 if you want to skip context)
Too many founders and VCs have been fooled by these people. Stop believing everything you hear. Data is much harder than most people think.
@paperinstr is one of those teams that just ruthlessly executes with complete clarity and earnestness.
This is a GPT3 moment for open source AI in knowledge work. Congrats @bertgodel , @joeybesgen , @thegavinbains .
We are releasing Ultramarine 1 Flash, a frontier-class model built for knowledge work. It is open-source and achieves similar performance to Fable 5, while being 60x cheaper.
It was post-trained by a team of 5 for ~$300k all-in. Available on API now, with weights releasing in 2 weeks.
We are releasing Ultramarine 1 Flash, a frontier-class model built for knowledge work. It is open-source and achieves similar performance to Fable 5, while being 60x cheaper.
It was post-trained by a team of 5 for ~$300k all-in. Available on API now, with weights releasing in 2 weeks.