Swetava Ganguli

ML/AI Research @Apple. MS/PhD (#ComputationalMath) + MS (#ComputerScience) @Stanford. Formerly @GoldmanSachs. BTech @iitgn

[山景城小路 @Zen_with_AI]

斯坦福把一门从零手搓 agent 的新课全免费公开了，第一个作业就是自己写一个 Claude Code。

视频 讲义 作业 都可以直接下载学习

https://t.co/VEfA5NwcGE

[Michael Ryan @michaelryan207]

🔊 Big Update!

All materials are now being released publicly for our NEW Stanford course CS329Z: Engineering AI Agents!

Starting with the lectures, which are going live daily on the Stanford Online YouTube channel!

Lecture 1: https://t.co/Gpjxoyi5UM

🧵 (1/5)

[Joshua Jebaraj @joshva_jebaraj]

If you want to learn how AI agents work, I highly recommend Harvard’s CS2680.

It covers how Claude Code works, how to build your own agent, and how to serve LLMs on GPUs.

https://t.co/DBoWdDeyGb

[Larry Dial @classiclarryd]

Sharing the thought process behind the architecture for Marin 535B-A23B, which finished pretraining on 18T tokens last week: https://t.co/ZdeshNHhcy

Earlier this year the Marin project had 6 full-time employees at Open Athena, a mix of AI veterans and people newer to the space, like me. I learned a lot from the team as an IC. Since then we got our B200 GPU stack running in a month, scaled up 300x on GPU to a 535 billion parameter model, publicly forecasted the run, live-streamed the entire thing, and made it through pretraining slightly better than predicted with zero instabilities.

In the blog I call out specific contributions on architecture- but the broader gratitude is to the ecosystem. 2 years ago this would have been inconceivable on a first large scale attempt. Now it’s possible, in large part due to published work from Llama, DeepSeek, Qwen, Gemma, Olmo, Cohere, Nemotron, Arcee, Inkling, Mistral, Microsoft AI, Kimi, Muse, academia, and many others. The recipes are out there. A small team with big dreams (and some funding) can train a big model. (And if you have doubts on that, you can copy our fully open recipe from data to code).

Of course, training a model does not produce a binary outcome. I am proud of how far we’ve come and optimistic about how much more there is to explore! The team has grown recently and will be taking the model through SFT and RL soon, sharing the findings along the way.

[Percy Liang @percyliang]

Why was Marin 535B-A23B the size it was? Why 8 of 384 experts? Why GatedNorm, sconv, XSA? Why linear learning rate? All of these factors were driven by tradeoffs between timelines, compute efficiency, MoE router stability, etc. Read about the rationale of each design decision:

[Thomas Wolf @Thom_Wolf]

18 of the 20 best open models today are Mixture-of-Experts. Dense is basically over.

And yet the know-how to train a good MoE still lives mostly in the heads of a handful of teams.

So we spent 800k GPU hours running the experiments ourselves and wrote everything down. Super happy to finally release "The Ultimate MoE Training Handbook", free and open, organized around the questions training teams hit:
- how many experts, how big, shared or not?
- why is half my step spent in all-to-all, and how do I hide it?
- why is my run 25x slower with no error in the logs?
- softmax, sigmoid or sqrt(softplus) router, and why keep it in fp32?
- aux loss or DeepSeek's loss-free bias?
- upcycle a dense checkpoint or train from scratch?
- can I serve with fewer experts per token than I trained with?

⚠️ Warning: it's long (~70k words, budget a few days, sorry) but with 55 interactive figures, code you can run and numbers from our own runs.

Read it here: https://t.co/6XihlUrj92

[Tatsunori Hashimoto @tatsu_hashimoto]

How well do you understand training dynamics for deep nets?

Here’s a CS312 (Deep Learning Alchemy) prediction question: midway through training an LM, we temporarily lower the learning rate and then restore it. How does validation loss change relative to the original run? ↓

We train a 35M-parameter LM with a WSD learning-rate schedule (peak LR 0.003, tuned roughly to be optimal for this setting). Consider two different learning rate schedules

A short LR drop: between 40-50% of training, dropping to 0.1x the original LR.
A longer, shallower LR drop: between 35-55%, dropping to 0.55x the original LR.

These LR drops are matched so that the total sum of the removed LRs is the same in both.

How does the shape of the dip affect validation loss during the dip and after the LR returns?

The image shows four qualitative predictions.
A: The deeper LR drop leads to a larger loss drop and a large penalty after
B: A deeper LR drop leads to a large temporary loss drop, but both match soon after.
C: Both LR drops keep the loss curve monotone, but a larger LR drop leads to slower progress
D: The deeper LR drop leads to a larger temporary loss drop, and this gain persists long term

Try the full quiz: https://t.co/E4Utj54GiU

[Andrej Karpathy @karpathy]

Weekend hack: HackerTube. I was annoyed with YouTube's recommendations so I had Claude filter down 43M comments over 20 years of HN to 6,454 videos that keep being recommended. *Significantly* better viewing experience, ~2 hours with Opus 5.5:
https://t.co/JIhXN7OSh1

- Excellent recs without any logins (state is kept in local browser session)
- Thumbnail is a representative frame from middle of the video instead of the super duper A/B test hyper-optimized user uploads
- Underlying data from the comment analysis as one zip file, point your Claude at it and ask for your own version
- HackerTube Longs ✊

A bit more on the process:
I use Claude Code (desktop app for Mac), Opus 5.5 on Medium effort as my default. I only talk to Claude and have it do everything, the only "manual work" I helped with in tandem was in the browser (reminiscent of MenuGen): 1) I had to create some kind of API key from Google Cloud UI to query YouTube for video metadata, 2) I booted up a VPS box for serving and 3) I bought the domain name. Once the box was up I gave Claude the IP and told it update, secure, harden, deploy and optimize the code, and it did the whole thing.

Many awesome projects that would have normally taken weeks/months even can now be few-shotted for breakfast. What else are we not thinking of.

[Horace He @cHHillee]

Feels like it's been some time since I've learned something new about non-determinism on GPUs 😆 Pretty cool!

[Russ Salakhutdinov @rsalakhu]

New work on Training Foundation Models with a Single Algorithm:

Paper: https://t.co/bH6eDSgSbd
Code, Models: https://t.co/TDiMe3qLUM

This work introduces Unified Training, a framework that unifies pretraining, midtraining, supervised fine-tuning and RL under a single maximum-likelihood objective. Building on MaxRL, it enables static data and RL-generated data to be trained together within the same system.

Empirically, Unified Training improves accuracy and compute efficiency, generalizes better to harder reasoning problems, and reduces forgetting.

Interestingly, reusing data across training stages can improve performance even when conventional staged training fails. It also produces models that adapt more effectively to new tasks, leading to stronger continual learning.

See an excellent thread by @yus167.

[Jia-Bin Huang @jbhuang0604]

How thousands of GPUs train one AI

Data parallelism helps, but not sufficient. We need new ways to split the workload along different axes.

Let's learn about 5D (data, tensor, context, pipeline, experts) parallelism and how they work together.

https://t.co/sIjxxeD4z5

[Sebastian Raschka @rasbt]

The complete reasoning-from-scratch video series is now up (and linked) here: https://t.co/5IOHLTCIGt

(Taking a short break but will be back with LLM model and research updates in November!)

[Gergely Orosz @GergelyOrosz]

Finding it very educational and interesting to learn about the hardware layer of GPUs: how they are structured, the obvious advantages and disadvantages bs CPUs.

This is from Modal’s excellent “GPU Glossary” print book @can gave me (the PDF is free here: https://t.co/ltzxXm1GSi)

[Yi Ma @YiMaTweets]

The lectures of my new course "Principles of Deep Representation Learning" are now available at the Youtube site: https://t.co/9eCNxhxWLM with new lectures continuing to be added there. I will open-source all the slides later in the textbook website: https://t.co/ihPBCkI3x5

[Dominik Schnaus @dominik_schnaus]

DINOv2 has never seen a caption, and Qwen3 has never seen an image. We still aligned their embedding spaces without a single image-caption pair.

It even works when the images and the captions come from different datasets.

Project page: https://t.co/2YHSkXVz1D

⬇️

[Sayak Paul @RisingSayak]

Was meant to be a Valentine's gift, but I'm releasing it early ❤️

Wrote a long piece on KV caching in flow models for image generation.

Includes intuition, commentary, implementation details, benchmarks, & some quirky experiments.

https://t.co/Ufrp5jZIf1

[Lucas Beyer (bl16) @giffmana]

So they say "three things we scaled". Let me translate:

1. "compute" - yep, that's compute.
2. "environments and harnesses" - actually, also compute.
3. "and grader compute" - you guessed it, that's also compute.

joke aside, pretty cool to see their public live dashboard, including "cost so far" (!)

(due to recent events/discussions: no, this QT is not sponsored. I don't do sponsor stuff, I'm here for the fun.)

[Steven Strogatz @stevenstrogatz]

Marvelously instructive, for anyone who wants a visual, physical understanding of the recent Navier-Stokes (Millennium) singular solution.

[Sam Altman @sama]

I agree with Dario that we need to pace the frontier. This has been a primary topic of discussions we've had at OpenAI in recent weeks.

Committing to having independent evaluators with employee-like access is a great idea, and we will do the same. We'll have more to share soon.

[Dario Amodei @DarioAmodei]

We Must Pace the Frontier: I’ve written a new essay on why the AI industry should slow down, with a three-part plan for doing so.

Anthropic is unilaterally committing to the first of these steps. We’ll provide third-party evaluators with permanent, employee-level access to our systems, so that they can verify adherence to our safety measures, report on incidents, and assess models’ alignment during training.

You can read the full post here: https://t.co/OGyPb7yaYt