Jiri Klic

Programming, Statistics, GIS, Machine Learning and Earth Observation. Passionate about @GlobalGoalsUN

[Yann LeCun @ylecun]

@ClementDelangue @Dan_Jeffries1 Everyone, please join Project Tapestry
https://t.co/5MOgouVplV

[Andrej Karpathy @karpathy]

Three days ago I left autoresearch tuning nanochat for ~2 days on depth=12 model. It found ~20 changes that improved the validation loss. I tested these changes yesterday and all of them were additive and transferred to larger (depth=24) models. Stacking up all of these changes, today I measured that the leaderboard's "Time to GPT-2" drops from 2.02 hours to 1.80 hours (~11% improvement), this will be the new leaderboard entry. So yes, these are real improvements and they make an actual difference. I am mildly surprised that my very first naive attempt already worked this well on top of what I thought was already a fairly manually well-tuned project.

This is a first for me because I am very used to doing the iterative optimization of neural network training manually. You come up with ideas, you implement them, you check if they work (better validation loss), you come up with new ideas based on that, you read some papers for inspiration, etc etc. This is the bread and butter of what I do daily for 2 decades. Seeing the agent do this entire workflow end-to-end and all by itself as it worked through approx. 700 changes autonomously is wild. It really looked at the sequence of results of experiments and used that to plan the next ones. It's not novel, ground-breaking "research" (yet), but all the adjustments are "real", I didn't find them manually previously, and they stack up and actually improved nanochat. Among the bigger things e.g.:

- It noticed an oversight that my parameterless QKnorm didn't have a scaler multiplier attached, so my attention was too diffuse. The agent found multipliers to sharpen it, pointing to future work.
- It found that the Value Embeddings really like regularization and I wasn't applying any (oops).
- It found that my banded attention was too conservative (i forgot to tune it).
- It found that AdamW betas were all messed up.
- It tuned the weight decay schedule.
- It tuned the network initialization.

This is on top of all the tuning I've already done over a good amount of time. The exact commit is here, from this "round 1" of autoresearch. I am going to kick off "round 2", and in parallel I am looking at how multiple agents can collaborate to unlock parallelism.
https://t.co/WAz8aIztKT

All LLM frontier labs will do this. It's the final boss battle. It's a lot more complex at scale of course - you don't just have a single train. py file to tune. But doing it is "just engineering" and it's going to work. You spin up a swarm of agents, you have them collaborate to tune smaller models, you promote the most promising ideas to increasingly larger scales, and humans (optionally) contribute on the edges.

And more generally, *any* metric you care about that is reasonably efficient to evaluate (or that has more efficient proxy metrics such as training a smaller network) can be autoresearched by an agent swarm. It's worth thinking about whether your problem falls into this bucket too.

[ComplexityExplorer @ComplexExplorer]

Happy New year! Kick off 2024 by joining us for Nonlinear Dynamics with Instructor Liz Bradley beginning Jan. 16
https://t.co/osffaMFoJB

[ETP4HPC @Etp4HPC]

New ETP4HPC White Paper: Federated HPC, cloud and data infrastructures. The authors identify the challenges and formulate recommendations. Illustrated with the examples of @Fenix_RI_eu and @gaiax_aisbl
https://t.co/YdMo2a70A5

[GeoPandas @geopandas]

Long time in the making, but we are happy to officially announce:

✨ Dask-GeoPandas for scalable spatial analysis in Python! ✨

Blogpost: https://t.co/mGcqscjlxX
Docs: https://t.co/GJ3FXyhN8b

Small thread ⬇️

[Person @ElectricWeegie]

It’s finally here! #book #MachineLearning #Python

[Jiri Klic @deltodon]

@Tasting_Tea It is an interesting subject. Unless you've done it already, I would suggest looking into M343 - Random processes in continuous time. It covers epidemics quite extensively. I am always happy to chat if you are interested. All the best with your work.

[The Ocean Cleanup @TheOceanCleanup]

BREAKING: @Coldplay adopts Interceptor 005, aka Neon Moon 1, to support The Ocean Cleanup’s mission to rid the oceans of plastic.

[Olivier Hagolle @OHagolle]

How wide can we extend the field of view of next generation of #Sentinel2 satellites ? We have studied the limitation due to directional effects using @CesbioLab 's #DART3D radiative transfer simulator. See our thread of blog posts, starting from here:
https://t.co/AYbpFF4l4r

[EODC GmbH @EODC_GmbH]

@ESA's openEO platform project was successfully kicked off yesterday, with full support from over 20 colleagues from the consortium partners @WWU_Muenster @VITO_RS_ @sinergise @EURAC @EGI_eInfra @GEOSEC2025 @planetek and the Austrian Hail Insurance.

[Steven Brunton @eigensteve]

I'm excited to announce the first video in a new series on sparsity and compression. This is one of my favorite topics in applied math and statistics.

Check out the video on YouTube: https://t.co/Z4kjonwkgE

[Stanford University @Stanford]

Global emissions of methane have reached the highest levels on record. https://t.co/LwfOehR9gt

[Jiri Klic @deltodon]

#EarthObservation for @GlobalGoalsUN by @RUS_Copernicus https://t.co/z8DRwp7jOe

[Farm-Hand @FarmHandAgTech]

'India, the world’s largest groundwater user, extracts a dangerously high amount of 250 km^3 of water per year. As water becomes increasingly scarce, this will make... irrigation...more difficult and expensive.' See how Farm-Hand can help. @giz_india
https://t.co/fNxkVrNqf4

[Jiri Klic @deltodon]

Great @apachekafka talk by @ciudilo during the last Edinburgh Meetup 👍

[Jiri Klic @deltodon]

Just found out that my friend from @AbertayUni Yannic Kawan worked on concept art for The Elder Scrolls Online: Summerset - Cinematic Trailer. 👍 Enjoy 🙂
https://t.co/T1ZJ9WGv0U

[Jiri Klic @deltodon]

I have never seen anyone live code the compiler during 60min presentation before, but @TartanLlama did it. 👍 Dependency chain management, cross-platform compilation in WSL and gems just kept coming. 💡 It felt like Xmas came to @cppedinburgh a bit early this year. 🌟

[Jiri Klic @deltodon]

Very interesting talk on workings of C++ committee by @AerialMantis during the last @cppedinburgh 👍

[Jiri Klic @deltodon]

Excellent talk by @LizzieEardley on Experimentation and data-driven decision making during #ScotML One of those moments when you come home so inspired that you cannot wait to pick up your notes and textbooks even when it is almost a midnight 👍

[Jiri Klic @deltodon]

Spatial Analysis of Scotland’s Highest Peaks with google maps mapbox and alteryx by @JohnTdata during #ScotML