ICML2024 Tutorial on
"Neural Operators
& Machine Learning on Function Spaces"
is now out. #NeuralOperators#AInScience@nvidia@NVIDIAAI
https://t.co/24cLebv7A3
Looking forward to building the future of this field with the bigger community, all together.
tomorrow i'm sitting down with @Azizzadenesheli from Prometheus at Miami AI Summit 🔥
physics intelligence. AI that actually understands how the physical world works. this one's gonna be good
fireside chat, 1:30 PM ET 🚀
Proud to have contributed to the @UN@ScienceBoard_UN scientific brief on AI and climate modeling
The last few years has seen tremendous progress in AI-weather modeling, starting with our launch of FourCastNet, the first high-resolution AI weather model.
Climate modeling with AI is harder, with the need for longer term forecasting and extrapolation to new scenarios.
Building in laws of physics at different scales and disentangling different mechanistic pathways is essential in AI-climate models.
https://t.co/OifC6sOnnh
A detailed work on how to extend great neural network architectures to neural operator framework, making them readily applicable to operator learning problems. This work has been one of the mind provoking and principle riddle solving works in MLonFunctions.
Neural operators – Convert popular neural networks into neural operators for scientific modeling
Extending neural networks to function spaces: While many phenomena are inherently described by functions, neural networks define vector-to-vector mappings that rely on fixed discretizations of the input and output.
Neural operators instead define learnable function-to-function mappings that guarantee consistent predictions across different discretizations of the input and output functions. By respecting the functional nature of the data, neural operators can achieve improved performance and generalization.
Translating the success of deep learning to operator learning: Careful engineering of neural architectures has been a key factor in deep learning’s success. Translating these architectures to neural operators is crucial for operator learning to enjoy the same empirical optimizations. Our new intro to neural operators (https://t.co/hrlkddRZeI) is officially published in @NatMachIntell It includes:
Key principles for constructing NOs
Recipes for converting popular architectures (CNNs, GNNs, transformers, etc.) into NOs
Guidance for practitioners
Joint work w/ @julberner@mliuschi@JeanKossaifi Valentin Duruisseaux Boris Bonev @Azizzadenesheli@Caltech
Today at NeurIPS (4:30-7:30pm), I will present our work on adaptive sampling for bridging the gap between physics informed models and numerical methods. It exploits many interesting links from orthogonal polynomials, quadrature rules and parametrization with neural networks.
#NeuralOperators learn physics through data.
We study long term prediction capability of #NeuralOperator on a hard task of ocean emulation with variable forcing, making me think very seriously about coupled weather ocean model, #THEModel
Excited to share our recently published paper in @WileyGlobal on "Ocean Emulation With Fourier Neural Operators: Double Gyre" https://t.co/MWXSZtbFBY
We used Fourier Neural Operators to build the first high-resolution weather model, FourCastNet. Since it works so well for atmospheric emulation a natural progression is to extend them to emulate ocean simulations.
We propose learning the dynamics of a simplified ocean simulation using Fourier neural operators. Fourier neural operators.
We are able to generate long forecasts using trained Fourier neural operators, and find that they are more accurate than using climatology or persistence on short-term forecasts and approach the accuracy of the physics-based model.
On long-term forecasts, the neural operators can still predict future scenarios with realistic physics like propagating waves and meandering currents. This is impressive because no physics is explicitly programmed into the neural operators. Physics is learned from data. @Azizzadenesheli
My @MLSysConf keynote is now online. https://t.co/VX6od4R8jA
The scaling of large language models has led to impressive gains in language understanding, but at a cost of insatiable memory and bandwidth requirements.
I advocated a principled approach of designing optimization and quantization algorithms that can reduce memory requirements without sacrificing accuracy.
This includes our research into gradient compression methods (GaLore, SignSGD) and logarithmic number system for representation.
We also design fine-grained memory reduction schemes such as KV cache compression, chunking and offloading to overcome memory bottlenecks in language models, especially in the reasoning mode where current memory requirements are massive.
Such principles are broadly applicable and especially relevant to physical AI where the memory and bandwidth requirements are even greater than frontier LLMs.
FALCON: built on centuries of knowledge from fluid dynamics, turbulent flows, control, RL, and ML, to deliver foundational approach for aviation, tested and trained on real world wing, validated on the real world.
This is very exciting, thanks @BBCBreaking for featuring our work.
AI+Science book is now up,
After years of work, our book on
Artificial Intelligence for Science in Quantum, Atomistic, and Continuum Systems
is available at Foundations and Trends® in Machine Learning.
We cover the history of works in the intersection of AI and Science, to novel breakthroughs and advance in foundation of AI.
From neural networks to neural operators,
From PINNs to PINO and so on,
From computer vision to quantum, graph data, grid data, point cloud data, and GenAI in sciences,
From classical reduced order methods to learned ones.
Etc.
With an extraordinary list of co-authors
Xuan Zhang, Limei Wang, Jacob Helwig, Youzhi Luo, Cong Fu, Yaochen Xie, Meng Liu, Yuchao Lin, Zhao Xu, Keqiang Yan, Keir Adams, Maurice Weiler, Xiner Li, Tianfan Fu, Yucheng Wang, Alex Strasser, Haiyang Yu, YuQing Xie, Xiang Fu, Shenglong Xu, Yi Liu, Yuanqi Du, Alexandra Saxton, Hongyi Ling, Hannah Lawrence, Hannes Stärk, Shurui Gui, Carl Edwards, Nicholas Gao, Adriana Ladera, Tailin Wu, Elyssa F. Hofgard, Aria Mansouri Tehrani, Rui Wang, Ameya Daigavane, Montgomery Bohde, Jerry Kurtin, Qian Huang, Tuong Phung, @MinkaiX , Chaitanya K. Joshi, Simon V. Mathis, Kamyar Azizzadenesheli, Ada Fang, Alán Aspuru-Guzik, Erik Bekkers, Michael Bronstein, Marinka Zitnik, @AnimaAnandkumar , @StefanoErmon , Pietro Liò, @yuqirose , Stephan Günnemann, @jure, Heng Ji, Jimeng Sun, Regina Barzilay, Tommi Jaakkola, Connor W. Coley, Xiaoning Qian, Xiaofeng Qian, Tess Smidt, @ShuiwangJi
Our 500+ page AI4Science paper is finally published:
Artificial Intelligence for Science in Quantum, Atomistic, and Continuum Systems. Foundations and Trends® in Machine Learning, Vol. 18, No. 4, 385–912, 2025
https://t.co/RzxYTDOJwx