Chung

[Chung @love17697]

@probnstat Binomial and Gaussian.

[Romee Panchal @romitheguru]

All ML enthusiasts tapping into Deep Learning and NLP may have heard about the famous "Attention Is All You Need" paper. Implementing it from scratch is a great way to understand it. However, it has many prerequisites to understand it fully.

Below is the list of papers that you should read and implement before proceeding to the "Attention Is All You Need" paper.

Start with Basic Neural Networks:
- Implement a simple feed-forward neural network from scratch
- Focus on backpropagation and gradient descent
- Paper: "Gradient-Based Learning Applied to Document Recognition" (LeCun et al., 1998) - implement a simpler version

Sequence Models Basics:
- Build a basic RNN from scratch
- Implement LSTM cells
- Paper: "Long Short-Term Memory" (Hochreiter & Schmidhuber, 1997) - start with a basic version

Word Embeddings:
- Implement Word2Vec
- Paper: "Efficient Estimation of Word Representations in Vector Space" (Mikolov et al., 2013)
- Understanding embeddings is crucial for transformers

Basic Attention Mechanisms:
- Implement Bahdanau attention
- Paper: "Neural Machine Translation by Jointly Learning to Align and Translate" (Bahdanau et al., 2014)
- This is directly relevant to transformer attention

Sequence-to-Sequence Models:
- Build a basic seq2seq model with attention
- Paper: "Sequence to Sequence Learning with Neural Networks" (Sutskever et al., 2014)

Once you've implemented these, you'll have the foundation to understand:
- Attention mechanisms
- Positional encodings
- Multi-head attention
- Layer normalization
- Residual connections

These components all come together in the Transformer architecture.

[Chung @love17697]

@cuukhonganchay Hi em

[Chung @love17697]

@probnstat So beautiful

[Chung @love17697]

A bit nervous about uncertainty that future challenges will bring; they can all turn left or right, here and there; I cannot know.