DW

Father. Machine Learning Life's Languages & Building biological software.

[DW @dirkweissenborn]

@ylecun @_rockt That's not to say that today's auto-regressive is the way to go btw, just pointing out that there are no fundamental reasons why it should not be able to do such tasks albeit maybe not the most efficient.

[DW @dirkweissenborn]

@ylecun @_rockt There is no reason why auto-regressive models shouldn't in principle be able to self correct when allowed to iterate on a task. It's like asking someone to one shot a hard coding problem without letting them iterate, write tests, etc. I think the flaw is in the evaluation here.

[DW @dirkweissenborn]

@jeremyphoward @levelsio Yes and interestingly enough this turned around after that in the sense that training on unlabeled data never really worked well in vision so this only got more traction with the advent of LLMs again.

[DW @dirkweissenborn]

@jeremyphoward @levelsio That might be true though I would say that for many of us it still was. I think this is really the biggest achievement, i.e. the focus on universality. The problem in this discussion is the focus on the term LLM imo. BERT did also not embrace the same kind of universality as GPT.

[DW @dirkweissenborn]

@jeremyphoward @levelsio Also, agreed wrt the universality argument.

[DW @dirkweissenborn]

@jeremyphoward @levelsio I guess my main point is that neither gpt nor ulmfit fell out of the sky really. The time was ripe. There was continuous progress towards it starting with word embeddings which are also based on a crude form of language modeling. Of course kudos to the people who made it happen!

[DW @dirkweissenborn]

@giffmana @jeremyphoward To be fair, I think BERT 2018 was earlier which is arguably one of the first larger LMs.

[Nando de Freitas @NandoDF]

Oh no Albert Einstein didn’t work as hard as Elon Musk and Sergey Brin. What a slacker 😅

Productivity is not a consequence of number of working hours only. Focus and motivation do matter. Doing one thing well can be far more impactful than doing many insignificant things, like texting, emails, Twitter …

[DW @dirkweissenborn]

@karpathy @plasticlistorg While some debate exists, most reputable scientific institutions, including the EPA and EFSA, haven't found glyphosate to be carcinogenic. In fact, there is no convincing scientific study. It's generally considered less toxic than many alternatives.

[Stephanie Chan @scychan_brains]

Devastatingly, we have lost a bright light in our field. Felix Hill was not only a deeply insightful thinker -- he was also a generous, thoughtful mentor to many researchers. He majorly changed my life, and I can't express how much I owe to him.

Even now, Felix still has so much to give -- to all of us. Below, I've compiled some of his writings that I think would be inspiring to anyone in the field. They reflect his unusual perspective, his incisive writing, and his irreverent, witty sense of humor. It's a pleasure and an epiphany to read any of these.

I hope this is a small way to help ensure that Felix's light continues to shine, as it deserves to

--

1. Why do transformers work so well on language? (https://t.co/hKHwwmx3xM) Atypically few jokes in this one, but starting around 0:32:00 Felix answers a fundamental question that I've never seen addressed so well. He ties the architectural properties of the transformer 1:1 to properties of language. I think this perspective is so illuminating and incredibly underappreciated.

2. What deep learning can teach us about linguistics and science (https://t.co/2RqobF1Hzn). It starts with a pretty entertaining refutal of Chomskian linguistics, and then dives into an optimistic and pretty irrefutable argument about why neural networks can teach us about language.

3. How to write a paper (https://t.co/jseEALGcys). One of many things I learned from Felix -- he taught me to put together a research story from the beginning, including imagined results. This narrative helps guide the research, even if the story might change with each new set of experiments.

4. The bittersweet lesson (https://t.co/c8ANdiOq1v). Felix wrote this a few weeks before he died, and it's a unique, insightful take on the bitter lesson. He asked for feedback but sadly I got overwhelmed with other things and it stayed on my todo list for too long.. I didn't get to tell him what a beautiful essay I thought it was, and how much he still had to contribute...

5. Mental health in (AI) research (https://t.co/91fR4jeyJw). If you are struggling, please do reach out for help, and please do hold on. There may not always be a complete fix, but there are always ways to make things significantly better. I believe this from the bottom of my heart

[DW @dirkweissenborn]

@kchonyc @FelixHill84 Didn't know until today and I am quite shocked myself. Even though I haven't seen Felix since interning in his group at DM he made a lasting impression on me for the deeply caring person that he was. I enjoyed conversations over a beer or playing football together. May he RIP ;(

[DW @dirkweissenborn]

@_philschmid @LightOnIO @answerdotai Why BERT and not something that's also useful as a generative model like discrete diffusion? Should in theory subsume any BERT like model in terms of capabilities.

[DW @dirkweissenborn]

@_rockt Looks amazing! Congrats Tim and team!

[DW @dirkweissenborn]

@giffmana Congrats Lucas. Curious to see what you'll be cooking :)

[DW @dirkweissenborn]

@yoavgo Wouldn't disagree with that but I don't see a big issue in this particular case, i.e. writing about something as technical as hparam sweeps.

[DW @dirkweissenborn]

@karpathy I guess the chat transcripts you'll find online are likely the ones that are better than average so it's almost like a weak form of supervision.

[DW @dirkweissenborn]

@_rockt @j_foerst @ucl @uclcs @ai_ucl @UCL_DARK Amazing. Well deserved!!

[Edward Grefenstette @egrefen]

Oof. What a read!

https://t.co/KmlMLFCtrd