Top Tweets for #DeepLearningMadeSimple

Day 197 : DataScience Journey Continuing on The Paper That Changed Deep Learning Forever: "Deep Residual Learning for Image Recognition" (He et al., 2015) Practical intuition – what it actually FEELS like when you train one: You know how with a normal deep net, after ~30 layers, adding more layers makes training loss go UP instead of down? It’s not overfitting , the network literally forgets how to copy the input to the output. The optimizer is screaming “I can’t make this identity function with 30 layers of Conv+ReLU!” ResNet’s fix is brutally simple: Stop asking the block to output the full thing. Just ask it “what should I ADD to the input?” (the crux of ILSVRC) . So instead of forcing : output = SomeHeavyStuff( input) you do : output = SomeHeavyStuff (input) + inputNow, if the layer has nothing useful to add → it just learns to output ≈ 0 → output ≈ input. That’s trivial for the optimizer. Literally one forward pass and it sees “oh, just put zeros everywhere in these weights”. Result? You can stack 100–1000 layers and the network trains like a dream. Most layers early on just learn “do nothing, pass it through”, and only the layers that actually see useful patterns activate. It’s like giving the network an “I’m already good” button. That’s why ResNet-152 trains faster and better than a 34-layer plain net, even though it has 4–5× more layers. Now the actual math (super direct, no fluff): Forward pass in ResNet is just recursive addition Because of y = F(x) + x, if you unroll the whole network you get: Final output = input + F₁(x) + F₂(x) + F₃(x) + … + Fₙ(x). There is a direct path (the +input) that is completely untouched by any weights or non-linearities. The signal has an express highway straight to the end.