Max B. Paulus

PhD student in machine learning @ETH and @MPI_IS

[Richelle🧬 @Richelle_Ji]

Not buying only here for the freebies

We tested them all @ClickHouseDB lounge @Data_AI_Summit, so you know what to get!

https://t.co/AolvaxtJ6u

[LAION @laion_ai]

We're introducing Project AELLA, in partnership with
@inference_net
&
@wyndlabs_ai

AELLA is an open initiative to make 100M scientific papers accessible via LLM made structured summaries.

Available now:
- Dataset of 100K summaries
- 2 fine-tuned LLMs
- 3d visualizer

👇

[gabriel @gabriel1]

i have the most liked video on sora 2 right now, i will be enjoying this short moment while it lasts

cctv footage of sam stealing gpus at target for sora inference

[Chris J. Maddison @cjmaddison]

I love this. It's the same story, every time: simple, generic architectures are good enough. Focus on the data.

[Max B. Paulus @mbpaulus]

@cjmaddison Congrats @HaonanDuan @cjmaddison and team!

[Chris J. Maddison @cjmaddison]

What makes a great scientist? Most AI scientist benchmarks miss the key skill: designing and analyzing experiments.

🧪 We're introducing SciGym: the first simulated lab environment to benchmark #LLM on experimental design and analysis capabilities.
#AI4SCIENCE #ICML25

[Jason Wei @_jasonwei]

We don’t have AI self-improves yet, and when we do it will be a game-changer. With more wisdom now compared to the GPT-4 days, it's obvious that it will not be a “fast takeoff”, but rather extremely gradual across many years, probably a decade.

The first thing to know is that self-improvement, i.e., models training themselves, is not binary. Consider the scenario of GPT-5 training GPT-6, which would be incredible. Would GPT-5 suddenly go from not being able to train GPT-6 at all to training it extremely proficiently? Definitely not. The first GPT-6 training runs would probably be extremely inefficient in time and compute compared to human researchers. And only after many trials, would GPT-5 actually be able to train GPT-6 better than humans.

Second, even if a model could train itself, it would not suddenly get better at all domains. There is a gradient of difficulty in how hard it is to improve oneself in various domains. For example, maybe self-improvement only works at first on domains that we already know how to easily fix in post-training, like basic hallucinations or style. Next would be math and coding, which takes more work but has established methods for improving models. And then at the extreme, you can imagine that there are some tasks that are very hard for self-improvement. For example, the ability to speak Tlingit, a native american language spoken by ~500 people. It will be very hard for the model to self-improve on speaking Tlingit as we don’t have ways of solving low resource languages like this yet except collecting more data which would take time. So because of the gradient of difficulty-of-self-improvement, it will not all happen at once.

Finally, maybe this is controversial but ultimately progress in science is bottlenecked by real-world experiments. Some may believe that reading all biology papers would tell us the cure for cancer, or that reading all ML papers and mastering all of math would allow you to train GPT-10 perfectly. If this were the case, then the people who read the most papers and studied the most theory would be the best AI researchers. But what really happened is that AI (and many other fields) became dominated by ruthlessly empirical researchers, which reflects how much progress is based on real-world experiments rather than raw intelligence. So my point is, although a super smart agent might design 2x or even 5x better experiments than our best human researchers, at the end of the day they still have to wait for experiments to run, which would be an acceleration but not a fast takeoff.

In summary there are many bottlenecks for progress, not just raw intelligence or a self-improvement system. AI will solve many domains but each domain has its own rate of progress. And even the highest intelligence will still require experiments in the real world. So it will be an acceleration and not a fast takeoff, thank you for reading my rant

[Max B. Paulus @mbpaulus]

@DimitrisPapail Nice idea. It inspired me to ask for a solved sudoku. And it fails. The intuition is the same reason I believe.

[Yoshua Bengio @Yoshua_Bengio]

ICLR 2025 will have another blogpost track!

If you have new intuitions on past work, noticed key implementation details for reproducibility, have insights into the societal implications of AI, or an interesting negative result, consider writing and submitting a blogpost. Accepted entries will have the opportunity to be presented as a poster at ICLR2025 (in Singapore)! Deadline: Before Nov. 16 2024

More info: https://t.co/0PH7FinGbz

[Magic @magicailabs]

LTM-2-Mini is our first model with a 100 million token context window. That’s 10 million lines of code, or 750 novels.

Full blog: https://t.co/oFz4A9ynVZ

Evals, efficiency, and more ↓

[Chris J. Maddison @cjmaddison]

📣Come work with us 🤗! We're recruiting a postdoc fellow in AI for small molecule drug discovery. A special opportunity to benefit from huge data efforts @thesgconline and have your predictions tested in the lab.

apply here 👉 https://t.co/0f2QUoznws
deadline 🗓️ 31 Dec '24

[Marcel Fratzscher @MFratzscher]

Fast nirgends in der Welt arbeiten so viele Frauen in #Teilzeit, viele davon würden gerne mehr Stunden arbeiten. Für manche ist die fehlende Infrastruktur bei #Kitas und Schulen das zentrale Problem, für viele lohnt es sich auch wegen des Ehegattensplittings finanziell nicht.

[Max B. Paulus @mbpaulus]

@firsyard @LezLuthor Source?

[Magic @magicailabs]

Meet LTM-1: LLM with *5,000,000 prompt tokens*

That's ~500k lines of code or ~5k files, enough to fully cover most repositories.

LTM-1 is a prototype of a neural network architecture we designed for giant context windows.

[Max B. Paulus @mbpaulus]

@itechnologynet @moritzkremb Exactly!

[Max B. Paulus @mbpaulus]

@giomdal @LudovicoEinaud @ThePianoGuys I’d go for @paulkalkbrenner: https://t.co/0wMVtJ8BBl

[Raging Capital Ventures @RagingVentures]

Silicon Valley Bank $SIVB reports earnings tomorrow

Investors have rightfully been fixated on $SIVB's large exposure to the stressed venture world, with the stock down a lot.

However, dig just a little deeper, and you will find a much bigger set of problems at $SIVB... 1/10

[Bob Elliott @BobEUnlimited]

The US banking system is built on the expectation that equity and bond holders accept the bank economic risk and depositors, particularly the small folks, do not.

While that is not legally the structure, its important to keep in mind that's functionally how it works. Thread.

[Max B. Paulus @mbpaulus]

@keinSpekulant @wast8357 @b_riexinger Und dazu der Gewinn bei MunichRe in der Sparte Naturkatastrophen wohl tendenziell höher ausfällt, wenn es im abgelaufenen Geschäftsjahr weniger Katastrophen und Schadensmeldungen gab. Die Aussage von Wenning also durchaus Anlass zur Freude sein könnte, selbst für @b_riexinger

[Max B. Paulus @mbpaulus]

@pip_net Why do you believe it is able to read linked content?