Fabian Fuchs

Senior Research Scientist @ DeepMind. I work on LLM post-training, focusing on improving training data. Prior to that, I worked on AlphaFold.

Typos are my own.

[Fabian Fuchs @FabianFuchsML]

A year ago I asked: Is there more than Self-Attention and Deep Sets? - and got very insightful answers. 🙏 Now, Ed, Martin and I wrote up our own take on the various neural networks architectures for sets.

Have a look and tell us what you think! :)

➡️https://t.co/Z1aprTcLQV ☕️

[Fabian Fuchs @FabianFuchsML]

Both Max-Pooling (e.g. DeepSets) and Self-Attention are permutation invariant/equivariant neural network architectures for set-based problems. I am aware of a couple of variations for both of these. Are there additional, fundamentally different architectures for sets? 🤔

[Fabian Fuchs @FabianFuchsML]

I wrote a blog post trying to understand TrackStar, a gradient-based method tracing LLM predictions to influential training examples.

Mostly: take the main equation, read it from right to left, and poke at the pieces with a small MNIST toy example. 🙂

☕️ https://t.co/JCWblqh956

[Adam Golinski @adam_golinski]

Our Apple ML Research team in Barcelona is looking for a PhD intern! 🎓
Curiosity-driven research 🧠 with the goal to publish 📝

Topics: Confidence/uncertainty quantification and reliability of LLMs 🤖

Apple here: https://t.co/ZiN3ecWGo7

[Fabian Fuchs @FabianFuchsML]

Graphs , Sets, Universality

We put more work into this and are presenting it via the ICLR blogpost track (thanks to organisers and reviewers!). Have a read and let us know what you think:

https://t.co/MtpPYYUfTm

better in light mode💡, dark mode🌙 messes with the latex a bit

[Petar Veličković @PetarV_93]

📢 New blog post!
Realising an intricate connection between PNA (@GabriCorso @lukecavabarrett @dom_beaini @pl219_Cambridge) & the seminal work on set representations (Wagstaff @FabianFuchsML @martinengelcke @IngmarPosner @maosbot), Fabian and I join forces to attempt to explain!

[Adam R. Kosiorek @arkosiorek]

Text-to-image diffusion models seem to have a good idea of geometry. Can we extract that geometry? Or maybe we can nudge these models to create large 3D consistent environments? Here's a blog summarizing some ideas in this space :)
https://t.co/INlMuslCdU

[Fabian Fuchs @FabianFuchsML]

@GabriCorso That is fantastic to hear!

[Fabian Fuchs @FabianFuchsML]

@PetarV_93 @GabriCorso @lukecavabarrett @dom_beaini @pl219_Cambridge @martinengelcke @IngmarPosner @maosbot Ed's link is broken, here it is: @EdWagstaff :)

[Petar Veličković @PetarV_93]

📢 New blog post!
Realising an intricate connection between PNA (@GabriCorso @lukecavabarrett @dom_beaini @pl219_Cambridge) & the seminal work on set representations (Wagstaff @FabianFuchsML @martinengelcke @IngmarPosner @maosbot), Fabian and I join forces to attempt to explain!

[Fabian Fuchs @FabianFuchsML]

I have recently had a range of very insightful conversations with @PetarV_93 about graph neural networks, networks on sets, universality and how ideas have spread in the two communities. This is our write up, feedback welcome as always! :)

➡️https://t.co/vbmCGHBHxd ☕️

[Sergey Ovchinnikov @sokrypton]

Anyone know of a department looking to hire faculty in the protein/genome+evolution+ML space? Also RNA biology (asking for a friend) 🙂🥼🧪

[Adam R. Kosiorek @arkosiorek]

New blog post! Find out:
- what reconstructing masked images and our brains have in common,
- why reconstructing masked images is a good idea for learning representations,
- what makes a good mask and how to learn one

https://t.co/bFrmQvITEz

[Fabian Fuchs @FabianFuchsML]

@Padarn Gradient descent :)

[Fabian Fuchs @FabianFuchsML]

Graph neural networks often have to globally aggregate over all nodes. How we do this can have a significant impact on performance 🎯. After we recently finished a project on this, I wrote a blog post on this topic. Let me know what you think! :)

➡️https://t.co/OGJJAF9w9C ☕️

[Fabian Fuchs @FabianFuchsML]

@jonkhler That's very true, nice observation! Let me know how it goes in case you do, I am curious :)

[Emiel Hoogeboom @emiel_hoogeboom]

Molecule Generation in 3D with Equivariant Diffusion (https://t.co/4ZgiHdswER). Very happy to share this project (the last of my PhD woohoo 🥳) and a super nice collab with @vgsatorras @ClementVignac (equal contrib shared among three of us) and of course @wellingmax

[Fabian Fuchs @FabianFuchsML]

@newplatonism Depends, are you happy with treating cars as point masses in vacuum? :P

More seriously: people do work on making the L/H-based NNs more general (like allowing for friction & external forces), but, to my understanding, it's still mostly constrained to physical particle systems

[Fabian Fuchs @FabianFuchsML]

Emmy Noether connected symmetries and conserved quantities in physics - how is this related to exploiting symmetries with neural networks? 🤔

I've tried to answer this question in a blog post (no background knowledge required!):

➡️https://t.co/VfxMu3fkAK ☕️

[Fabian Fuchs @FabianFuchsML]

@lzamparo The book was already linked but I now added a few more links for further reading, including the ICLR keynote. Thanks for the suggestion!

[Fabian Fuchs @FabianFuchsML]

@_onionesque @Blendenfleck Thank you!

[Fabian Fuchs @FabianFuchsML]

I should have said 'no physics background knowledge required' - the blog post does assume general machine learning background knowledge :)

[Fabian Fuchs @FabianFuchsML]

@william_woof @andrewwhite01 +1; also, this equivalence does not need to be obvious at all. In some cases, like with max(), it might even seem counterintuitive that the function >can< be written as a sum-decomposition (ie in a Deep-Sets form)