Can we train neural networks in any shape? Well, we often noticed that they cannot be too thin or too deep, even if the number of parameters looks right. So we came up with Jumpstart Regularization as a way to fix that: https://t.co/T8XUkZH8H6
Coming up at #cpaior2022#orms#ml
Stochastic weight averaging (SWA) just keeps coming back, often rebranded as model merging. A simple equal average of params with constant LR is less sensitive to hypers, generalizes better, and is more adaptable to new settings. Give it a try! https://t.co/8k87b3QBxA
⁉️What do model-neural similarity scores tell us?
To systematically explore this for different metrics, we develop new numerical tools & analytics to characterize what drives similarity scores and what constitutes a "good" score. (1/10)
I am going to open source the code and key experiments from our new paper /Ensemble everything everywhere/. For that I'd like to use a clean, academic-paper-style github pages template. Are there any particularly good ones that people use these days?
https://t.co/9mOL2wzZ7i
For low-precision neural network training, I'm finding that ternary weights ({-1, 0, 1}) are consistently outperformed by septernary ({-2, -1, -0.5, 0, 0.5, 1, 2}). The latter format needs 40% fewer parameters to match fp16 performance, while consuming only 5% more bits.
What is the geometry of data?
🤖The geometry of weights in deep learning?
🧠The geometry of activations in neural networks?
Differential geometry can describe and quantify these structures. Want to use it for your research?
🌐Discover Geomstats!
https://t.co/3kZLlQuaPp
https://t.co/njMEqzH3TF This is our latest work that tries to reveal what diffusion/denoising model is really doing in terms of learning a data distribution: Subspace Clustering and Subspace Denoising. What surprises me the most is the number of samples required to learn the subspaces correctly scales linearly in the intrinsic dimension. No curse of dimensionality, instead, a blessing! Combined with the results from our previous whitebox Transformer paper, it seems that a mixture of low-dimensional Gaussians (or subspaces) is enough to model and learn anything in the world :-)
Our 100+ page mega paper: "White-Box Transformers via Sparse Rate Reduction: Compression Is All There Is?" has been officially accepted by JMLR. The paper has been updated based on the reviews and the new version is now available on arXiv (v3): https://t.co/oNzWzumjRf Hope this work will clarify most mysteries about deep networks for you.
Good afternoon! Lately, I have been really interested in the usefulness of equivariance. Do you know any paper comparing in the same conditions (#parameters, training schedules...) equivariant vs non-equivariant models in different tasks?
✨🎨🏰Super excited to share our new paper Ensemble everything everywhere: Multi-scale aggregation for adversarial robustness
Inspired by biology we 1) get adversarial robustness + interpretability for free, 2) turn classifiers into generators & 3) design attacks on vLLMs 1/12
❓How expressive is persistent homology (PH)?
❓Can such #topology-based learning methods be as effective as message-passing neural networks in graph #MachineLearning tasks?
➡️We (@rballeba, YT) explore this question in our updated preprint.
https://t.co/dt56Lxmrrm
🧵1/n
Excited to share our latest work that shows "Large Step Size Training Cannot Overfit" in univariate ReLU nets 🔥🔥🔥 https://t.co/eFHXlJE9yT
For the first time, we understand how *flatness*, *edge-of-stability* and *large stepsize* imply (near-optimal) generalization. 🧵1/
Cell complexes are everywhere! Ranging from meshes to molecules, cell complexes can naturally represent many forms of real data. So the question is, how can we leverage the properties of this topological representation in learning? Are transformers suitable for this domain? Check our new paper where we tackle this question! 🍩
Topological deep learning meets transformers! It turns out transformers can be naturally realized as topological neural networks & extended to topological domains. Interested ? check out our paper : https://t.co/0CllE7vZoY #TDL
Check out our latest work on human interaction generation, accepted at the @humogen11384 workshop (#CVPR2024)! 🆒 stuff:
➡️Fine-grained control with descriptions of both intra- and inter-person dynamics.
➡️InterHuman is extended with LLM-generated single-person annotations.