Lolo A.

Educating and building cool stuff.

prev-@Microsoft @Doordash @Netflix

[Sid Sijbrandij @sytses]

Video available at https://t.co/fGF9QMKXet

[Bryan Marley @_bryanmarley]

7 ChatGPT-4o Prompts That Will Make Your Jaw Drop:

[Tom Yeh @ProfTomYeh]

Transformer by Hand✍️

To study the transformer architecture, it is like opening up the hood of a car and seeing all sorts of engine parts: embeddings, positional encoding, feed-forward network, attention weighting, self-attention, cross-attention, multi-head attention, layer norm, skip connections, softmax, linear, Nx, shifted right, query, key, value, masking. This list of jargons feels overwhelming!

What are the key parts that really make the transformer (🚗) run?

In my opinion, the 🔑 key is the combination of: [attention weighting] and [feed-forward network].

All the other parts are enhancements to make the transformer (🚗) run faster and longer, which is still important because those enhancements are what lead us to "large" language models. 🚗 -> 🚚

Walkthrough

[1] Given
↳ Input features from the previous block (5 positions)

[2] Attention
↳ Feed all 5 features to a query-key attention module (QK) to obtain an attention weight matrix (A). I will skip the details of this module. In a follow-up post I will unpack this module.

[3] Attention Weighting
↳ Multiply the input features with the attention weight matrix to obtain attention weighted features (Z). Note that there are still 5 positions.
↳ The effect is to combine features across positions (horizontally), in this case, X1 := X1 + X2, X2 := X2 + X3....etc.

[4] FFN: First Layer
↳ Feed all 5 attention weighted features into the first layer.
↳ Multiply these features with the weights and biases.
↳ The effect is to combine features across feature dimensions (vertically).
↳ The dimensionality of each feature is increased from 3 to 4.
↳ Note that each position is processed by the same weight matrix. This is what the term "position-wise" is referring to.
↳ Note that the FFN is essentially a multi layer perceptron.

[5] ReLU
↳ Negative values are set to zeros by ReLU.

[6] FFN: Second Layer
↳ Feed all 5 features (d=3) into the second layer.
↳ The dimensionality of each feature is decreased from 4 back to 3.
↳ The output is fed to the next block to repeat this process.
↳ Note that the next block would have a completely separate set of parameters.

Together, the two key parts: attention and FFN, transform features both across positions and across feature dimensions. This is what makes the transformer (🚗) run!

[Rahul @sairahul1]

Less than 2 weeks since OpenAI started rolling out GPT-4o.

And people have been busy executing god-like tasks.

50 truly mind-boggling examples:

(29th is my favorite)

[, @dubs4o8]

An Incredible 1:44 from Giannis Antetokounmpo: https://t.co/FnYOvjzBc8

[Lolo A. @MMawulolo]

@YuHuang_yh @KevinJLeach @VanderbiltU Congrats Yu, this is awesome!

[Elizabeth Gulleen, MD @EGulleen]

Our paper regarding predictive models of adverse outcomes in neutropenic fever is finally out! Big thanks to the team from @UVA_ID and UVA Systems Engineering for making it happen. Shout out to modeling wizard @MMawulolo and mentors Drs. Chris Moore and Laura Barnes.

[Maithra Raghu @maithra_raghu]

New blog post: "Reflections on my (Machine Learning) PhD Journey" https://t.co/abXiTFntIp

2020 has marked the end of my six year PhD journey, filled with struggles, success and evolution of personal & research perspectives. In the post I share experiences and lessons learned ⬇️

[AIhub @aihuborg]

Recent and forthcoming machine learning and AI seminars: December edition - https://t.co/PbtygsVhqY

[Sergey Levine @svlevine]

Distribution shift is a fact of life for models in the real world: the test distribution will never really match the training distribution. But evaluating methods that account for distribution shift is difficult. WILDS aims to address that, across a range of realistic domains.

[Lolo A. @MMawulolo]

@EmmaBrunskill Looking forward!

[David Sontag @david_sontag]

New dataset alert!
AMR-UTI: Antimicrobial Resistance in Urinary Tract Infections
De-identified data derived from 80,000 patients with urinary tract infections (UTI) treated at Massachusetts General Hospital and Brigham & Women��s Hospital in Boston
https://t.co/5zwgHqTMgC

[David Sontag @david_sontag]

MIT's class on Machine Learning in Healthcare is now available for free on MIT's OpenCourseWare! All videos, slides, and lecture notes can be found here:
https://t.co/2WN7UZYMuU
https://t.co/vmBRAx54Oo

[Lolo A. @MMawulolo]

@david_sontag Super interesting! Looking forward to watching them. Thanks for sharing.

[Neil Lawrence @lawrennd]

Dear Mr Twitter,

I think we've all understood that you don't want us to have an edit button to fix typos and misspellings in our tweets.

But can we at least have a button for
"delete and copy to new draft"?

I would find that such a button very convenient.

Thank you,

Neil

[Rishabh Mehrotra @erishabh]

The 2020 #NobelPrize in Economics has been awarded to
Paul R. Milgrom and Robert B. Wilson “for improvements to auction theory and inventions of new auction formats.”

[Rishabh Mehrotra @erishabh]

#KDD2020: @BenCarterette & I will be giving our tutorial on "Recommendations in a Marketplace" this Sunday, as part of a full day tutorial on "Advances in Recommendation Systems".

Updated website with detailed outline: https://t.co/azgwzRg3N3

@kdd_news #RecSys #DataMining

[Irene Chen @irenetrampoline]

NEW DATASET ALERT! MIMIC-IV is an open dataset of over 200k patients from the ICU and ED with labs, notes, and chest x-rays.

Shoutout to the LCP lab at @MIT for the commitment and stewardship to open medical data and @alistairewj 's tutorial at @CHILconference (today!)

[Lolo A. @MMawulolo]

@alansaid @ACMRecSys @crmpla67 Where can authors find reviewers' comments?

[Victor Veitch 🔸 @victorveitch]

The presence of unobserved confounders can bias estimation of causal effects. But, if unobserved confounding is 'weak' then we might hope bias is small. Austen plots aid such judgements by visualizing sensitivity to unobserved confounding.
w @anishazaveri
https://t.co/sJFD7LFu3u