Silei Xu

[Silei Xu @sileixu]

𝟱𝟯% 𝗼𝗻 𝗛𝘂𝗺𝗮𝗻𝗶𝘁𝘆’𝘀 𝗟𝗮𝘀𝘁 𝗘𝘅𝗮𝗺! 🚀

B𝗎𝗂𝗅𝖽𝗂𝗇𝗀 𝗈𝗇 𝗍𝗁𝖾 𝗌𝗍𝗋𝗈𝗇𝗀 𝖿𝗈𝗎𝗇𝖽𝖺𝗍𝗂𝗈𝗇𝗌 𝗈𝖿 𝖦𝖯𝖳-𝟧.𝟤 𝖺𝗇𝖽 𝖦𝖾𝗆𝗂𝗇𝗂 𝟥 𝖯𝗋𝗈 𝖯𝗋𝖾𝗏𝗂𝖾𝗐, 𝗐𝖾’𝗏𝖾 𝗋𝖺𝗂𝗌𝖾𝖽 𝗍𝗁𝖾 𝖻𝖺𝗋 for HLE - 𝖺𝖼𝗁𝗂𝖾𝗏𝗂𝗇𝗀 𝟧𝟥.𝟢𝟦% on the full set!

[Silei Xu @sileixu]

@HenkPoley @Zoom @OpenAI Zoom is ahead again :)

[Silei Xu @sileixu]

@HenkPoley @cocogoatmain @Zoom Zoom now raised the bar again to 53% on the full HLE set. For the text only subset that Kimi is reporting on, the accuracy is even higher - 55.2% https://t.co/jUuKxOQaER

[Silei Xu @sileixu]

@DeepwriterAI @Zoom We are now at 53% on the full set and 55.2% on text only subset: https://t.co/jUuKxOQaER

[Silei Xu @sileixu]

𝖫𝗂𝗇𝗄: https://t.co/jUuKxOQaER

[Silei Xu @sileixu]

𝟱𝟯% 𝗼𝗻 𝗛𝘂𝗺𝗮𝗻𝗶𝘁𝘆’𝘀 𝗟𝗮𝘀𝘁 𝗘𝘅𝗮𝗺! 🚀

B𝗎𝗂𝗅𝖽𝗂𝗇𝗀 𝗈𝗇 𝗍𝗁𝖾 𝗌𝗍𝗋𝗈𝗇𝗀 𝖿𝗈𝗎𝗇𝖽𝖺𝗍𝗂𝗈𝗇𝗌 𝗈𝖿 𝖦𝖯𝖳-𝟧.𝟤 𝖺𝗇𝖽 𝖦𝖾𝗆𝗂𝗇𝗂 𝟥 𝖯𝗋𝗈 𝖯𝗋𝖾𝗏𝗂𝖾𝗐, 𝗐𝖾’𝗏𝖾 𝗋𝖺𝗂𝗌𝖾𝖽 𝗍𝗁𝖾 𝖻𝖺𝗋 for HLE - 𝖺𝖼𝗁𝗂𝖾𝗏𝗂𝗇𝗀 𝟧𝟥.𝟢𝟦% on the full set!

[Silei Xu @sileixu]

𝖶𝖾 𝖺𝗅𝗌𝗈 𝗌𝖾𝗍 𝖺 𝗇𝖾𝗐 𝖲𝖮𝖳𝖠 𝖿𝗈𝗋 𝖣𝖾𝖾𝗉𝖲𝖾𝖺𝗋𝖼𝗁𝖰𝖠, 𝖺 𝗇𝖾𝗐 𝖽𝖺𝗍𝖺𝗌𝖾𝗍 𝖿𝗋𝗈𝗆 𝖣𝖾𝖾𝗉𝖬𝗂𝗇𝖽, 𝗌𝗎𝗋𝗉𝖺𝗌𝗌𝗂𝗇𝗀 𝗉𝗋𝖾𝗏𝗂𝗈𝗎𝗌 𝖲𝖮𝖳𝖠 𝖻𝗒 𝟣𝟢%!

[Silei Xu @sileixu]

@DeepwriterAI @Zoom Nevertheless, 50.91 is still a very impressive number if it’s on the complete text only subset. Congrats on the achievement.

[Silei Xu @sileixu]

@DeepwriterAI @Zoom "similar to the other reported scores" - actually no, in your chart, numbers for Gemini/GPT-5 Pro/Opus are all on the full HLE set.

[Silei Xu @sileixu]

@DeepwriterAI @Zoom It seems you only reported result on a subset? Zoom is talking about the full HLE set with text and image. The subset of HLE with images is noticeably harder than the text only ones

[Silei Xu @sileixu]

@omarsar0 Thanks for sharing! Glad that you find it interesting. This is a fun side project I did last December. I didn’t get enough time to run more comprehensive experiments, so I would love to learn if you find CoD being less effective in some domains

[Silei Xu @sileixu]

@billyuchenlin Is the checklist for each example automatically generated by LLM? It would be great if the prompt to generate the checklists could be shared. Thanks.

[Silei Xu @sileixu]

@nishant_ch @MKBHD You can’t unlock a phone on a table with sensor on the back; sensor on the side only works for one hand. So both are suboptimal actually

[Silei Xu @sileixu]

@gcampax haha exactly what happens to genienlp nowadays

[David Doel @daviddoel]

Me: Here’s half my income.

Daycare: Cool. Here’s a new virus every other week.

[Stanford OVAL @StanfordOVAL]

We're at #acl2022. Come see our work "A Few-Shot Semantic Parser for Wizard-of-Oz Dialogues with the Precise ThingTalk Representation" (by @gcampax @sina_semnani Ryan Kearns, Lucas Sato, @sileixu and @MonicaSLam ) - right now in the VPS2!

[Silei Xu @sileixu]

@gcampax I got it right in the first try! Hahaha it was totally a random guess

[Silei Xu @sileixu]

@gcampax Yeah sounds like you

[Stanford OVAL @StanfordOVAL]

Our second #emnlp2020 paper this year wonders, do we really need humans to build conversational agents? https://t.co/YdG1nzBlqG by @sileixu (*) Sina Semnani (*) @gcampax @MonicaSLam (1/n)

[Stanford OVAL @StanfordOVAL]

We are pleased to announce that @SloanFoundation is funding our new open-source initiative to create a usable virtual assistant for consumers that protects privacy, by building upon our Almond research prototype.

[Silei Xu @sileixu]

@gcampax meanwhile, I am waiting for my git pull at speed of 3 KiB/s