Autoencoder by hand ✍️ ~ 7 steps walkthrough below
Goal: squeeze four numbers down to two, then rebuild the original four from them.
1. Given
Let us start with four training examples: X1, X2, X3, X4.
2. Auto (copy to targets)
We copy the training examples straight into the targets. That is the whole trick behind the name: "auto" is Greek for "self", and the data is its own label.
3. Encoder, layer 1
Let us multiply the inputs by the weights, add the biases, and apply ReLU. Negative values get crossed out and become zero.
4. Encoder, layer 2 (the bottleneck)
We do it again, and now the four dimensions have become two. This layer is called the bottleneck, because everything has to fit through it.
5. Decoder, layer 1
Let us go back the other way: multiply, add, ReLU. This time there are no negatives to cross out.
6. Decoder, layer 2
We multiply once more and get the outputs Y. This is the decoder's attempt to rebuild the four original numbers from the two it was given.
7. MSE loss gradients
Let us compare Y with the targets Y'. The gradient is 2 x (Y - Y'): subtract, then double. Those gradients kick off backpropagation, and the weights start to learn.
🙆 I do a physical exercise in class for this one. Everybody stands up.
Stretch your arms out wide. Imagine you are holding a heavy textbook (like Introduction to Algorithms by Prof. Cormen), the whole thing, every page. Now bring your hands slowly together until they almost touch your neck. The bottle "neck."
The final exam is tomorrow and you are allowed one cheat sheet: whatever you can scribble on your palm. All nine hundred pages have to survive the squeeze. That is the encoder.
Now imagine you sit in the exam. Push your arms back out to where they started. You try to rebuild the textbook from your palm notes. That is the decoder.
Of course you cannot get every page back. What you get back is what mattered enough to write down, and the gap between the two is the loss the network is trying to shrink. Your entire education is all about encoding and decoding!
I call it AI by Arms 🙆. It gets a laugh, and then it gets remembered.
Try it yourself!
#AIbyHand #DeepLearning #MachineLearning
Relational, NoSQL, Graph, Time-Series, Vector, In-Memory & Spatial Databases—each serves a unique purpose. Understanding database types helps you choose the right tool for the right problem.
#Database#SQL#NoSQL#DataEngineering#DataScience