@yuris Yes. Hence, 250K is upper limit, and a lot of brute force. Would be interesting to see how the reduction of k in k-provability would work with AI.
How do you evaluate creative work when there is no single correct answer? There are no unit tests created for taste. And that becomes a problem when you’re trying to measure progress in creative AI.
At HumanSignal, we ran an experiment: we gave 10 AI models the same 10 creative briefs, with no reference designs, and asked them to build websites.
Then we had humans evaluate the results.
We’ve scored 10 models, on 100 designs using 4500 human preference votes. A few interesting findings:
- Models frequently converged on similar fonts, colors, and design patterns, despite working independently.
- The top three models were separated by just 4 percentage points in overall human preference.
- Seven different models ranked first on at least one brief. Being good at one type of design doesn't necessarily translate to another.
We've published an interactive research report where you can explore all 100 designs, see the models' creative decisions, and compare how humans ranked them.
Take a look at rankings via link in comments:
Reposting this article we've recently published on rubric design. A lot of subjectivity in human data can be reduced with a better rubric design. Our solution is to treat rubrics as it's own layer: a reasoning model sitting between requirements and annotation interface.