Jose González-Abad

Deep learning for climate science

[Jose González-Abad @jogonzab]

@productaizery I think there’s a deeper distinction between the program humans write (architecture, optimization) and the program(s) AI generates (weights). We control the former. The latter is something that “happens”, we don't have much control over it.

[Andrej Karpathy @karpathy]

https://t.co/Lb6T42n5jl

[Nichole “Vapor” Ayers @Astro_Vapor]

Just. Wow. As we went over Mexico and the U.S. this morning, I caught this sprite.

Sprites are TLEs or Transient Luminous Events, that happen above the clouds and are triggered by intense electrical activity in the thunderstorms below. We have a great view above the clouds, so scientists can use these types of pictures to better understand the formation, characteristics, and relationship of TLEs to thunderstorms.

[Eric J. Michaud @ericjmichaud_]

tl;dr: Maybe learning simple things (basic knowledge, heuristics, etc) actually lowers the loss more than learning sophisticated things (algorithms associated with higher cognition that we really care about), and the sophisticated things will eventually be learned as scaling further drives down the loss.

Some rough thoughts on this issue:
As we know, LLMs are (pre)trained to minimize next-token prediction loss across human text. Ignoring fine-tuning, RLHF, etc., what they learn will be determined both by (1) what is __possible__ to learn and (2) what is __optimal__ to learn under this objective.

So what do models learn under this objective? The low-hanging fruit might be to learn token frequencies, then common bigrams, then to learn how to repeat subsequences in context (https://t.co/DDt0sY7lgq) then simple grammatical rules, then all sorts of knowledge, and so on and so on. With a small network you can only learn the low-hanging fruit (https://t.co/tsebT22ICB), but of course the *hope* with scaling is that eventually, to reduce the loss slightly more, you'll need to learn the more sophisticated algorithms for reasoning or planning or drawing connections between ideas or coming up with new explanations (https://t.co/YKN6M4D63m).

So why might current models have not learned the algorithms associated with higher cognition yet? It could be that simply (1) they are not possible for the models to learn. But one defense of scaling would be that (2) is the bottleneck:

It could be possible that the (loss improvement) ÷ (network capacity) of learning the algorithms/circuits we associate with higher cognition is still worse than for learning many additional pieces of knowledge or statistical patterns in language. Like maybe the model could have learned the "making new connections" algorithm but actually that is suboptimal and you reduce the loss more by learning lots of dumb things instead. Presumably though eventually the algorithms for reasoning, planning, making new connections, etc. would be the optimal things to learn to further reduce loss, and they would be learned with further scaling. This explanation feels slightly odd to me since I would have expected that learning the sophisticated general parts of cognition would be very frequently useful when predicting human text and therefore would have been learned by now. But maybe it's just a fact about predicting language that there is a *lot* of low-hanging fruit in learning the dumber things: https://t.co/5xykYjioeN

Explanation (1) -- there's a fundamental limitation -- is complicated. Maybe there are so many steps or so much compute needed to implement certain algorithms that it's just really hard to implement them in a forward pass of a transformer. Further scaling could maybe solve this (maybe you just needed to make the network deeper), but maybe you'd need such a massive network that it's practically impossible. Maybe it's not about expressivity but instead about whether SGD can learn the algorithms we care about and whether we have the data needed to learn them: https://t.co/EG62wIQ7L8.

So a rebuttal to your point would be about (2), but personally I don't know if that explanation is right! The bottleneck might really be about expressivity or learnability and we're just using the wrong architectures or objectives or data. Super interesting question.

[ECMWF @ECMWF]

🌍🌐 A new climate emulator for Destination Earth (#DestinE)! @predictia and partners will develop an #ML-based #climate emulator, reproducing some features of the Climate DT at a reduced computational cost. Learn more ➡️ https://t.co/twrbm9Ah0p

[Alex Hernandez-Garcia @alexhdezgcia]

I'll be hiring PhD and MSc students for Fall 2025 to do research on machine learning for science (details in 🧵)

I especially encourage applicants from under-represented groups and regions in conflict (Palestine, Ukraine, Sudan, Myanmar...)

See 🧵 for how to apply.

[Jose González-Abad @jogonzab]

@tsarnick I think the “in practice” part is relevant, otherwise you are implying that the Boltzmann machines were not useful at all, when they were from a theoretical perspective

[AIhub @aihuborg]

In this blogpost, Jose González-Abad (@jogonzab) writes about work on statistical downscaling for climate models, presented at the @RealAAAI Fall Symposium.
https://t.co/JSs98crrpf

[AIhub @aihuborg]

Generating physically-consistent local-scale climate change projections https://t.co/bYchQzOrfI

[IFCA Advanced Computing and e-Science Group @IFCA_Computing]

🚨 Bringing together #HPC and #AI: in our latest work we train, in a distributed fashion, #DeepLearning models in a multi GPU scenario using #Docker containers and @udocker_indigo

➡️ https://t.co/2h6545TBZv

[paulaharder.bsky.social @PaulaHarder8]

Working in ML for downscaling? Overwhelmed by the amount of literature? Some friends and me starting collecting all the papers out there: https://t.co/k56BVCikKw
@JuliaKaltenborn @jogonzab @venka_r @alexhdezgcia #downscaling #superresolution #aiforclimate

[LAION @laion_ai]

We launche a petition to democratize AI research by establishing an international, publicly funded supercomputing facility equipped with 100,000 state-of-the-art AI accelerators to train open source foundation models.
https://t.co/jbkQnlEHYH
https://t.co/agmh1Ptjky