@satyanadella A real external JevBench measurement of Microsoft-Decision-1 latency does not agree with Microsoft's numbers. Also independent accuracy does not agree with their rankings as well. See
@satyanadella The latency shown in the chart shared is not right. H2O-Lightning-4B has latency of about 29ms rather than 210ms. https://t.co/CtqmeuxB3v
@satyanadella Corrected latency for the Microsoft animation. Also, accuracy presented is not an independent benchmark like JevBench, so training and test may be too similar since generated by Microsoft themselves.
🚨 H2O-Lightning-4B is #1 among open-weight models on JevBench, ahead of Jev itself.
72.5 vs 71.5 on the official composite (intelligence, calibration, cost and speed), ahead of every open 12B, 26B and 31B model.
4B parameters. Apache-2.0. 🧵
@an_vo12 Acts like a feature not a bug. The anomalous legs are properly discounted as not real enough for that particular animal. Do it for an unknown animal for which leg counting is not a strongly known prior.
For GAIA, the dataset https://t.co/LT2hNgdnvo used is called a "validation" data set that is leaked all over internet. They should submit to the official test set. They also excluded results from Trase and https://t.co/PHQmCxqIFh referring to a 4 month old result as "previous SOTA": https://t.co/zFKx7Lu7kI and https://t.co/ysQbJ7b028
@manusai Need to post to GAIA test set. Your agent may be finding the many validation datasets online that allow the agent to cheat and get high validation score.
@MFarajtabar Similar results for gsm8k hard https://t.co/PPhialLB8t, which changes all the numbers and if LLMs were just applying some basic math it shouldn't matter much, but it matters alot.
@MFarajtabar Adding irrelevant items and seeing performance drop isn't new. I remember AI explained channel talking about this year(s) ago and relevant for his Simple Bench.
🚨 BREAKING: Open-Strawberry aims to recreate OpenAI's o1 as open-source! 🍓
🔓 Democratizing AI
🚀 Accelerating innovation
🌐 Community-driven development
Join the revolution: https://t.co/2OcLwKArsi
RT to support open AI! 🔄
#OpenSourceAI#AIRevolution