Mourad ZAIED

[Mourad ZAIED @Mourad_ZAIED]

What if we could delete BatchNorm and LayerNorm entirely? 🚀 Introducing SaluNet: a framework that replaces normalization with a simple learnable geometry (just 2 scalars per layer). It works on both CNNs and ViTs. 🧵👇

[Mourad ZAIED @Mourad_ZAIED]

@liuzhuang1234 Great video! To add:
Normalization rescales activations using stats. Derf learns a scaled and shifted erf. SALU learns saturation amplitude and transition width around a fixed center at zero.
Different geometric choices for normalization-free networks.
https://t.co/nwj0mLd2MS

[Mourad ZAIED @Mourad_ZAIED]

What if we could delete BatchNorm and LayerNorm entirely? 🚀 Introducing SaluNet: a framework that replaces normalization with a simple learnable geometry (just 2 scalars per layer). It works on both CNNs and ViTs. 🧵👇

[Mourad ZAIED @Mourad_ZAIED]

To put it simply:
Normalization moves the distribution.
Derf moves the function.
SALU deforms the function around a fixed center.
A different geometric perspective on going normalization-free. 👇

[Mourad ZAIED @Mourad_ZAIED]

What if we could delete BatchNorm and LayerNorm entirely? 🚀 Introducing SaluNet: a framework that replaces normalization with a simple learnable geometry (just 2 scalars per layer). It works on both CNNs and ViTs. 🧵👇

[Mourad ZAIED @Mourad_ZAIED]

@chouebmahamatis Thank you Choueb

[Mourad ZAIED @Mourad_ZAIED]

Paper & Code:
📄 arXiv: https://t.co/eOiPC2fQRl
💻 GitHub: https://t.co/B0dRtOWb9R (Pretrained checkpoints included)
Looking for collaborators on:
• Matched baselines & seeds
• Large-batch training
RTs appreciated! 🙏 #DeepLearning #ComputerVision #ViT

[Mourad ZAIED @Mourad_ZAIED]

What if we could delete BatchNorm and LayerNorm entirely? 🚀 Introducing SaluNet: a framework that replaces normalization with a simple learnable geometry (just 2 scalars per layer). It works on both CNNs and ViTs. 🧵👇

[Mourad ZAIED @Mourad_ZAIED]

The proof on CIFAR-100 👇

ResNet-18 + BatchNorm? Accuracy collapses at batch size 1. 😱

SaluNet-C-18? Retains 76.23% Top-1 at batch size 1.

It also scales to ImageNet-1K:
• ResNet-50: 80.16%
• ViT-B/16: 80.28%

[Mourad ZAIED @Mourad_ZAIED]

The problem with BatchNorm/LayerNorm? They rely on batch statistics.

So performance collapses at small batch sizes (e.g., large models, high-res images, or limited GPU memory).

SALU solves this by learning an element-wise transformation. No mean/variance. No batch dependency.

[Mourad ZAIED @Mourad_ZAIED]

Paper & Code:
📄 arXiv: https://t.co/eOiPC2fj1N
💻 GitHub: https://t.co/B0dRtOVDkj
Looking for collaborators on:
• Matched baselines & seeds
• Large-batch trainingRTs appreciated! 🙏 #DeepLearning #ComputerVision #ViT

[Mourad ZAIED @Mourad_ZAIED]

The proof on CIFAR-100 👇

ResNet-18 + BatchNorm? Accuracy collapses at batch size 1. 😱

SaluNet-C-18? Retains 76.23% Top-1 at batch size 1.

It also scales to ImageNet-1K:
• ResNet-50: 80.16%
• ViT-B/16: 80.28% (with only BS=128!)

[Mourad ZAIED @Mourad_ZAIED]

The problem with BatchNorm/LayerNorm? They rely on batch statistics.

So performance collapses at small batch sizes (e.g., large models, high-res images, or limited GPU memory).

SALU solves this by learning an element-wise transformation. No mean/variance. No batch dependency.

[Mourad ZAIED @Mourad_ZAIED]

A step by step backpropagation example
https://t.co/0bcUJQnyrI...
YouTube channel:
https://t.co/AMhE3zLDca

[Mourad ZAIED @Mourad_ZAIED]

@slashML A step by step backpropagation example
https://t.co/0bcUJQnyrI...
YouTube channel:
https://t.co/AMhE3zLDca

[kora online @korafans0]

بكاء جورجينا زوجة رونالدو وفرحة ابنه الهيسترية بعد تسجيل رونالدو الهاتريك والتأهل باليوفنتوس لدور ربع النهائي 🔥🔥

#يوفنتوس_اتليتكومدريد

[Mourad ZAIED @Mourad_ZAIED]

https://t.co/6RBCy8idAi

[Mourad ZAIED @Mourad_ZAIED]

Hhh https://t.co/8R0CWrg1pi

[Mourad ZAIED @Mourad_ZAIED]

https://t.co/AbfsBtnYy1

[Mourad ZAIED @Mourad_ZAIED]

https://t.co/rM9pbyPCEs

[Mourad ZAIED @Mourad_ZAIED]

Heart Model for Valentine’s Day https://t.co/usdYd8AzKN

[Mourad ZAIED @Mourad_ZAIED]

https://t.co/1gRXrCQZ6Q

[Mourad ZAIED @Mourad_ZAIED]

https://t.co/nSLLOT1pLl