David Foster

Author of Generative Deep Learning: Teaching Machines how to Paint, Write, Compose and Play (O'Reilly), #generativeAI,
Founding Partner of ADSP.

[David Foster @davidADSP]

@mattshumer_ Yeah looks awesome - any idea how they calculated the $0.19-$0.49 PPM tokens? They say it's based on $2/hour H100 cost and serve rate of 0.03 ms / token I think?

[David Foster @davidADSP]

@Thom_Wolf It's a reference to the fact that an ensemble of all submissions would have scored 81% on the private test set (i.e. 19% of solutions were unsolved by any solution) https://t.co/pwdwjJFn2P

[François Chollet @fchollet]

Does this mean the ARC-AGI benchmark has saturated?

Yes -- the v1 version of the benchmark is starting to saturate. There were already signs of this in the Kaggle competition this year -- an ensemble of all submissions would score 81%.

The competition next year will run on ARC-AGI-2, an updated version of the dataset that keeps the same format as v1, but features fewer tasks that can be easily brute-forced.

Early indications are that ARC-AGI-v2 will represent a complete reset of the state-of-the-art, and it will remain extremely difficult for o3. Meanwhile, a smart human or a small panel of average humans would still be able to score >95%.

[David Foster @davidADSP]

@arcprize @OpenAI o3 solves ARC-AGI? Huuuuge news if that's it...

[David Foster @davidADSP]

@fchollet Out of interest @fchollet, what % of arc test set puzzles remain unsolved by any submitted solution? And what would the top 2 entries score if ensembled (I know this means they'd have 4 attempts). Just curious how much they overlap.

[David Foster @davidADSP]

@OfficialLoganK What's the rate limit?

[David Foster @davidADSP]

@jsuarez @hirschibar Awesome write up! What about action masking - i.e. how do you handle cases where certain actions aren't possible (and the env returns you the mask at each timestep). Is this something PufferLib supports?

[David Foster @davidADSP]

@lmarena_ai @01AI_Yi Will the multimodal Llama 3.2 models be added to the overall leaderboard?

[David Foster @davidADSP]

@lmarena_ai @lmsysorg Will Llama 3.2 (the multimodal models) and Gemini 1.5-002 be added to the main leaderboard?

[David Foster @davidADSP]

@SullyOmarr Would you be willing to share the leaderboard from your evals?

[David Foster @davidADSP]

Spot the data viz fail 🤦‍♂️@BBC @BBCPolitics @BBCNews

[David Foster @davidADSP]

@giffmana @pastaraspberry What is it about Gemma that makes it open, but not open source? Thanks!

[David Foster @davidADSP]

@giffmana @pastaraspberry How are you defining open vs open source. Thanks!

[David Foster @davidADSP]

@NPCollapse Funny story - William Peebles co-authored the Mar 2023 Diffusion Transformer paper on which Sora is based, whilst at Meta as an intern.

But then joined OpenAI last year to co-lead Sora.

So I guess they did know how to do it, but let him leave 😂

[David Foster @davidADSP]

@Thom_Wolf Theory of Everything: https://t.co/gkxjVsmI8t

[David Foster @davidADSP]

@realGeorgeHotz Given the current breakthroughs, "linguistics" is a left-field candidate 🤔

[David Foster @davidADSP]

@nickfloats Does the --iw parameter affect remixes? In the docs it says it doesn't, but I'm never sure how much to trist the docs :)

[David Foster @davidADSP]

@nickfloats Related question / challenge - how do you get Midjourney to output the usual meaning of 'fork in the road', rather than this? Changing the prompt to use different words isn't allowed 😃

[David Foster @davidADSP]

@SullyOmarr Nice idea! Can you create a short example as a demo?

[David Foster @davidADSP]

@StabilityAI When your stochastic parrot drinks too much coffee.

🚀 Awesome work @StabilityAI !

[David Foster @davidADSP]

@EMostaque Is your stochastic parrot related to my stochastic parrot? 😄

#generativeai #stablediffusion #ai @StabilityAI