I’ve been working surprisingly hard on a classifier for Jellycats, of all things, for my fiancée’s app @myjellybook - you can upload a photo and it’ll scan the 3500 she has and figure out what it is.
Apple Vision segments the Jellycat from the photo, masks out hands, faces and bodies, and measures the body colour. Jina CLIP v2 on Replicate embeds three crops of the photo, and Vectorize retrieves the top 50 catalogue matches for each. The pool is then expanded to every colour of the matched designs and re-ranked by Lab distance to the photo, because CLIP barely separates ~200 Bashful Bunny colours. This made a big difference in accuracy!
Kimi K2.6 on Workers AI gets the photo plus the top 10 candidates and describes the fur colour and ear lining first, as they’re often the differentiator, then scores each one. Those scores are blended with the cosine margins over a background score into a softmax to give us probabilities, and then it’s presented nicely in the app, with the top 3 on a podium!
Lots of fun on @Cloudflare for as much as possible, as always. It’s been through lots of iterations to make it accurate but I think I’m there!
Looking forward to rolling it out this week
@jlongster Yeah they suck but it’s relative. They suck compared to a properly trained person/team crafting it over time, but relative to literally anyone one shotting it? They’re amazing
@theozero@CloudflareDev Started, and I’ll add my changelog app https://t.co/RTXZ3kDJxP and my outgoing email trap for Laravel https://t.co/YRGkQHhgwe too!
@kentcdodds@kodykoala I’ve just got Grok Bot running and man I am struggling to find use cases for it. Great, an overnight Slack digest. Great, reminders and stuff. I haven’t had the lightbulb moment yet