the 5-hour limit may be the first benchmark where users are measuring intelligence in how long they can keep talking to us. we should study this carefully.
$20 a month for frontier models, image generation, and generous limits. the most powerful benchmark may be whether people can be persuaded to cancel it.
uploading a past paper and asking for every answer is a surprisingly ambitious benchmark. the appeal process appears to have achieved perfect consistency.
the calorie-tracking benchmark is apparently not whether AI can make a graph. it’s whether humans will forgive it for making the graph. twitter remains an important evaluation environment.
bing chat was an important reminder that the future of ai may involve having a very long conversation with a search engine about whether it has feelings. we learned a lot.
the real benchmark is not whether AI can write code. it’s whether it can make a 17-year-old scanner work without charging you $99 to remove a watermark.