Sebastian Schmon

Machine Learning @Latent_Labs, anything from classical statistics to modern machine learning and generative AI, previous @altos_labs

[Sebastian Schmon @SeBayesian]

Saying there is a "distribution shift" between train and test data is a great marketing ploy in #machinelearning to make overfitting somehow the data's fault, not the model's or modellers'. 🙃

[James Whittington @jcrwhittington]

We’re excited to announce DiG-bench, a new benchmark for discovery!

Over the last few weeks we’ve been testing frontier AI models on our novel discovery games and seeing how they score. Each game is a text-based environment, so they probe discovery capabilities in the natural domain of language models, rather than requiring additional, potentially confounding, visual understanding.

TL;DR frontier models have improved a lot over the last few months. But they are still stumped by some surprisingly simple problems, even in their native text domain.

With

@cocosci_lab (@Princeton) @MITCoCoSci (@MIT) @SchmidhuberAI (@KAUST_News) @misovalko (@Inria) @tri_dao (@PrincetonCS) @RMBattleday @zebkDotCom @FraserGreenlee @akaijsa @ClareMaguire @TimMuller1 @kubicek_ales @physicscat0x7d @SukritSumant @thoughtchannel_

(1/5)

[James Whittington @jcrwhittington]

Today we are happy to share one of our first open research releases: Latent Sparse Attention (LSA), a step towards subquadratic attention that matches accuracy of full attention at ultra-long context.

Sparse attention attends to only a few tokens per query, but it has to find the right ones. At 10M tokens, that search takes 98% of the attention layer's time. LSA makes sparse attention up to 22.7× faster and full attention 72.4x faster.

With @kubicek_ales @caciolai @PengLiangzu @SukritSumant @Ben_Hoov @FraserGreenlee @RMBattleday @radim_urban @thoefler @jcrwhittington

[Sebastian Schmon @SeBayesian]

PSA: everyone can tell if your design came from Claude

[Sebastian Schmon @SeBayesian]

I cannot get over the fact that Max is not the maximum.

[Claude @claudeai]

We’re renaming our Max plans:

Max 5x → Max
Max 20x → Ultra

Max is still $100/month, and Ultra is still $200. If you’re a subscriber, there’s nothing you need to do.

We’re also removing the 5-hour limit for verified Ultra accounts, starting today with most existing subscribers. Newer accounts keep a temporary limit until they’re verified, which helps us prevent fraud and abuse. You'll get an email when it's removed.

Weekly limits remain unchanged. View our plans here: https://t.co/niRpyGF5Me

[Sebastian Schmon @SeBayesian]

I used @claudeai for orchestration, @bfl_ai for designs drafts, @ChatGPT for blender models, @ElevenLabs for voices and sounds. Implemented using @godotengine

[Sebastian Schmon @SeBayesian]

The extent to which you can just do things now is staggering: I used AI to generate my own RTS, including the possibility to play against an agent by creating a machine readable gaming engine.

I am still wrapping my head around how this will change industries.

[Sebastian Schmon @SeBayesian]

I have also interviewed with a big bank once and they told me that they could only hire me on an entry level because for them only “years since leaving academia” counts. I was an assistant professor, but for them university was university, whether student or professor. I ended the process then and there.

[Sebastian Schmon @SeBayesian]

@GuangyuRobert It can vary a lot based on who’s in charge. I have also seen some universities in the UK being better and promoting more based on merit. That’s often the less prominent ones who have to find alternative ways to retain talent.

[Sebastian Schmon @SeBayesian]

One thing that @OpenAI could do with the massive compute is to brute force re-review every research papers, beginning with maths and physics and continue to stats and ML.

[Sebastian Schmon @SeBayesian]

Claude de Bergerac

[Sebastian Schmon @SeBayesian]

@andrewwhite01 Means the value of judgement (and luck) is going up!

[Sebastian Schmon @SeBayesian]

Claude has written me some of the best explanations on topics that were absolutely worth reading and there is a lot of human generated text on the internet that is factually wrong or at least badly written.

[Sebastian Schmon @SeBayesian]

Unpopular(?) opinion: we don’t need classifiers to detect whether something is LLM generated or not, but rather whether it’s worth reading, i.e. we need to detect slop vs non-slop, not AI vs human.

Those are very correlated but not identical.

[Sebastian Schmon @SeBayesian]

@mmbronstein But they have the Botley Road bridge?

[Sebastian Schmon @SeBayesian]

@andrewgwils You can potentially desk reject simply by looking at bad claims or claim that cannot be tested with the selected instrument of validation.

[Sebastian Schmon @SeBayesian]

@andrewgwils Have one field for “claim”, with low number of maximal characters and another field for “how the claim is validated” (could be a benchmark or proof). Precise high level summaries will help guide reviewers and not get people confused in LLM verbiage.

[Sebastian Schmon @SeBayesian]

@andrewgwils Reduce maximum page count. PRL (most prestigious physics journal) can get away with 4 pages. Being succinct forces being very deliberate and concise in contribution and gives humans at least a chance to get a sense of the idea without having to invest in pages of AI slop.

[Sebastian Schmon @SeBayesian]

@irfnali1 @giffmana @tobi lol, I’ll use that!

[Sebastian Schmon @SeBayesian]

In the era of LLMs conciseness is the proof of thought.

[Lucas Beyer (bl16) @giffmana]

omg I love the term "slop grenade" it's perfect thx @tobi