Robert Mueller

Forest Pathologist by training, Data Scientist during the day. I would like to dive deeper into Computational Biology and Software Development.

[Ravid Shwartz Ziv @ziv_ravid]

1/8 Looks like my paper "Tabular Data: Deep Learning is Not All You Need" just hit 1,000+ citations 🥳🥳🥳
Here's the story of how we almost didn't publish it...
https://t.co/KiZ9dUTYWn

[David Smith @revodavid]

Today is the 6th anniversary of the R language.

R 1.0.0 was released on February 29, 2000. R 4.3.3 was released today.

Happy anniversary, #Rstats!

[atc1441 @atc1441]

DOOM on a Toothbrush? Sure!

[Edel Pérez-López @Edel_PLopez]

Anybody else noticed that statistics in plant biology/pathology papers is often wrong? We have been noticing that almost nobody check normality 🥲

[Andrej Karpathy @karpathy]

# on technical accessibility

One interesting observation I think back to often:
- when I first published the micrograd repo, it got some traction on GitHub but then somewhat stagnated and it didn't seem that people cared much.
- then I made the video building it from scratch, and the repo immediately went through hockey stick growth and became a verty often cited reference for people learning backpropagation.

This was interesting because the micrograd code itself didn't change at all and it was up on GitHub for many months before, stagnating. The code made sense to me (because I wrote it), it was only ~200 lines of code, it was extensively commented in the .py files and in the Readme, so I thought surely it was clear and/or self-explanatory. I was very happy with myself about how minimal the code was for explaining backprop - it strips away a ton of complexity and just gets to the very heart of an autograd engine on one page of code. But others didn't seem to think so, so I just kind of brushed it off and moved on.

Except it turned out that what stood in its way was "just" a matter of accessibility. When I made the video that built it and walked through it, it suddenly almost 100X'd the overall interest and engagement with that exact same piece of code. Not only from beginners in the field who needed the full intro and explanation, but even from more technical/expert friends, who I think could have understood it if they looked at it long enough, but were deterred by a barrier to entry.

I think as technical people we have a strong bias to put up code or papers or the final thing and feel like things are mostly self-explanatory. It's there, and also it's commented, there is a Readme, so all is well, and if people don't engage then it's just because the thing is not good enough. But the reality is that there is still a large barrier to engage with your thing (even for other experts who might not feel like spending time/effort!), and you might be leaving somewhere 10-100X of the potential of that exact same piece of work on the table just because you haven't made it sufficiently accessible.

TLDR: Step 1 build the thing. Step 2 build the ramp. 📈
Some voice in your head will tell you that this is not necessary, but it is wrong.

[Yann LeCun @ylecun]

@sirbayes 100% agree.

[Harrison Kinsley @Sentdex]

Sora got me like

[Andrej Karpathy @karpathy]

# on shortification of "learning"

There are a lot of videos on YouTube/TikTok etc. that give the appearance of education, but if you look closely they are really just entertainment. This is very convenient for everyone involved : the people watching enjoy thinking they are learning (but actually they are just having fun). The people creating this content also enjoy it because fun has a much larger audience, fame and revenue. But as far as learning goes, this is a trap. This content is an epsilon away from watching the Bachelorette. It's like snacking on those "Garden Veggie Straws", which feel like you're eating healthy vegetables until you look at the ingredients.

Learning is not supposed to be fun. It doesn't have to be actively not fun either, but the primary feeling should be that of effort. It should look a lot less like that "10 minute full body" workout from your local digital media creator and a lot more like a serious session at the gym. You want the mental equivalent of sweating. It's not that the quickie doesn't do anything, it's just that it is wildly suboptimal if you actually care to learn.

I find it helpful to explicitly declare your intent up front as a sharp, binary variable in your mind. If you are consuming content: are you trying to be entertained or are you trying to learn? And if you are creating content: are you trying to entertain or are you trying to teach? You'll go down a different path in each case. Attempts to seek the stuff in between actually clamp to zero.

So for those who actually want to learn. Unless you are trying to learn something narrow and specific, close those tabs with quick blog posts. Close those tabs of "Learn XYZ in 10 minutes". Consider the opportunity cost of snacking and seek the meal - the textbooks, docs, papers, manuals, longform. Allocate a 4 hour window. Don't just read, take notes, re-read, re-phrase, process, manipulate, learn.

And for those actually trying to educate, please consider writing/recording longform, designed for someone to get "sweaty", especially in today's era of quantity over quality. Give someone a real workout. This is what I aspire to in my own educational work too. My audience will decrease. The ones that remain might not even like it. But at least we'll learn something.

[Andrej Karpathy @karpathy]

Thinking about the ideal blogging platform:

1. Writing:
- in markdown
- with full WYSIWYG, not just split view (think: Typora)
- super easy to copy paste and add images
2. Deploying:
- renders into static pages (think: Jekyll)
- super simple, super minimal html with no bloat
- hosting at a nice url
3. Maintaining:
- analytics (think: Google Analytics)
- comments section (think: Disqus)
4. Ownership:
- full export, access/ownership of the raw files to perpetuity should the need arise to move elsewhere.

I don't believe this exists.

Github hosting (my primary blog atm) comes close. I use VS Code + extensions to write, but dealing with images is a bit of a pain and no WYSIWYG. I also experimented with Typora for writing, and then export to markdown, but still a bit clunky. Jekyll is ~ok but is very heavy and keeps breaking. Deploy is super easy (git push). Maintanace is non-existent, have to separately use and pay for Disqus and Analytics.

Platforms like Medium/Substack are quick and convenient, but extremely annoying with all their log in requirements, popups, unnecessary features (e.g. highlights), various other dark patterns they invent over time and you down "own" your files, and can't download them as simple markdown if you wanted to.

Right now feeling this close |---| to trying to build the thing 🤦‍♂️🥲

[MrBeast @MrBeast]

I’m gonna give 10 random people that repost this and follow me $25,000 for fun (the $250,000 my X video made)

I’ll pick the winners in 72 hours

[EMBL Heidelberg @EMBLHeidelberg]

The @EMBL Heidelberg campus has been covered in snow! ❄️

Enjoy these aerial shots by our photographer Kinga Lubowiecka 📸

[Elon Musk @elonmusk]

Earth already receives about the same energy from the Sun in an hour than humanity consumes in a year.

Solar panels just need to catch a tiny amount of it to power our entire civilization!

[Andrej Karpathy @karpathy]

I touched on the idea of sleeper agent LLMs at the end of my recent video, as a likely major security challenge for LLMs (perhaps more devious than prompt injection).

The concern I described is that an attacker might be able to craft special kind of text (e.g. with a trigger phrase), put it up somewhere on the internet, so that when it later gets pick up and trained on, it poisons the base model in specific, narrow settings (e.g. when it sees that trigger phrase) to carry out actions in some controllable manner (e.g. jailbreak, or data exfiltration). Perhaps the attack might not even look like readable text - it could be obfuscated in weird UTF-8 characters, byte64 encodings, or carefully perturbed images, making it very hard to detect by simply inspecting data. One could imagine computer security equivalents of zero-day vulnerability markets, selling these trigger phrases.

To my knowledge the above attack hasn't been convincingly demonstrated yet. This paper studies a similar (slightly weaker?) setting, showing that given some (potentially poisoned) model, you can't "make it safe" just by applying the current/standard safety finetuning. The model doesn't learn to become safe across the board and can continue to misbehave in narrow ways that potentially only the attacker knows how to exploit. Here, the attack hides in the model weights instead of hiding in some data, so the more direct attack here looks like someone releasing a (secretly poisoned) open weights model, which others pick up, finetune and deploy, only to become secretly vulnerable.

Well-worth studying directions in LLM security and expecting a lot more to follow.

[Andrew Ng @AndrewYNg]

It is only rarely that, after reading a research paper, I feel like giving the authors a standing ovation. But I felt that way after finishing Direct Preference Optimization (DPO) by @rm_rafailov @archit_sharma97 @ericmitchellai @StefanoErmon @chrmanning and @chelseabfinn. This beautiful paper proposes a much simpler alternative to RLHF (reinforcement learning from human feedback) for aligning language models to human preferences.

RLHF has been a key technique for training LLMs. In brief, RLHF (i) Gets humans to specify their preferences by ranking LLM outputs, (ii) Trains a reward model (used to score LLM outputs) -- typically represented using a transformer network -- to be consistent with the human rankings, (iii) Uses reinforcement learning to tune an LLM, also represented as a transformer, to maximize rewards. This requires two transformer networks, and RLHF is also finicky to the choice of hyperparameters.

DPO simplifies the whole thing. Via clever mathematical insight, the authors show that given an LLM, there is a specific reward function for which that LLM is optimal. DPO then trains the LLM directly to make the reward function (that’s now implicitly defined by the LLM) consistent with the human rankings. So you no longer need to deal with a separately represented reward function, and you can train the LLM directly to optimize the same objective as RLHF.

Although it’s still too early to be sure, I am cautiously optimistic that DPO will have a huge impact on LLMs and beyond in the next few years.

You can read the paper here: https://t.co/m14qRYszVa I also write more about this in The Batch (linked to below).
https://t.co/8h2ag2plIa

[Bruno Rodrigues @brodriguesco]

I've just read that that Yihui Xie @xieyihui left @posit_pbc

https://t.co/ZeXwMMfZHa

he will continue to work on his packages as a contractor

that's really surprising (not in a good way) news to start 2024...

[Jon Erlichman @JonErlichman]

Steve Jobs on one of the secrets of life.

[Yann LeCun @ylecun]

René Barjavel foresaw our addiction to smartphones!

[Baltimore Ravens @Ravens]

A VERY MERRY CHRISTMAS TO ALL!!!!

[John Carmack @ID_AA_Carmack]

This is a great little book to take you from “vaguely understand neural networks” to the modern broad state of practice. I saw very little to quibble with. https://t.co/4zbAxKv5Vq