The ancient Greeks saw art as an imitation of life. In the digital age, memes serve a similar purpose, distilling the complexities of human experience into a single image layered with humor, irony, culture, and unspoken meaning.
We spend a lot of time benchmarking Olympiad maths and advanced sciences, but what about whether AI truly understands what we mean.
That’s the question behind MemeBench 2.0, built by @RiseDataLabs and @VibecutterAI.
We evaluated 11 leading models from @OpenAI, @GeminiApp, @claudeai, @MistralAI, @grok, @Kimi_Moonshot and more on 1,000 memes.
Some interesting results: Claude Opus 5.5 took the top spot, Gemini climbed from last place in v1 to #3, and ironically, given @elonmusk’s love of memes, Grok didn’t do too hot, dropping from #2 to #10 out of 11 models.
Why does this matter beyond memes?
The same ability to recognize subtext, interpret cultural nuances, and understand unspoken intent matters when AI analyzes customer feedback, evaluates advertising, or navigates complex human interactions.
The next frontier of AI evaluation isn’t just harder problems. It’s measuring the complexities of human intelligence.
Explore MemeBench 2.0 and the full results:
https://t.co/IiIgIFWKom
Such a good panel on how companies like @zapier and Ramp are shaping what models get measured on. Favorite quote from @GregKamradt "As long as we have problems, we'll have benchmarks." You can also replace benchmarks with training data.
@zapier, one of our Bench Summit partners, led a panel discussion on the realities of managing highly influential evaluation signals.
The panel, moderated by Lukas Bergstrom (@lukasb), included insights from Robin Salimans (@SalimansRobin) of Zapier, Greg Kamradt (@GregKamradt) of ARC Prize Foundation, and Shaiyon Hariri of Ramp.
Attendees heard about what defines a successful benchmark, the challenges of maintaining adoption, and critical lessons learned from V1 to V2+.
@zapier, one of our Bench Summit partners, led a panel discussion on the realities of managing highly influential evaluation signals.
The panel, moderated by Lukas Bergstrom (@lukasb), included insights from Robin Salimans (@SalimansRobin) of Zapier, Greg Kamradt (@GregKamradt) of ARC Prize Foundation, and Shaiyon Hariri of Ramp.
Attendees heard about what defines a successful benchmark, the challenges of maintaining adoption, and critical lessons learned from V1 to V2+.
@heyshrutimishra Love these prompt suggestions, thank you for sharing these! I built a resume-specific GPT in case it's helpful to anyone looking for personalized feedback. https://t.co/owQ5VlDlA9
@thevillageappco Unfortunately, money is hard to come by for female founders AND female investors. Because there’s less capital going to women-led funds, I find that female investors tend to invest less frequently and with lower commits. Hopefully, this will change one day.