HMD

[Mark Schmidt @MarkSchmidtUBC]

Slides from my ICML tutorial "Is numerical optimization theory irrelevant to machine learning practice in 2026?":
https://t.co/nRmKdzTrdA

(Updated to fix some typos, incorporate feedback, and add some things I did not have time for. Will link to the video here when posted.)

[Andrej Karpathy @karpathy]

One pattern I find useful for working with LLMs is a nice long ramble session. Sometimes the LLM needs more bits to understand what you're trying to achieve, but you're too lazy to type them. In these cases I like to lean back, switch to /voice and just ramble for like 10 minutes, total mess, anything goes, full stream of consciousness. Sometimes I declare it up top, something like "switching to speech recognition sorry for any typos...". Sometimes I turn it into a small interview of a few turns. But I find that the LLMs are somehow very good at reconstructing long incoherent rambles and often their echo of your own tangle of thoughts comes out quite a bit cleaner than what you started with. The result is that you improve the mind meld and have to correct things less from that point on.

[Anuj @anujcodes_21]

🚨 Anthropic just showed a 24-minute workshop on how to actually do prompts for Claude.

Taught by the people who built it.

Free. No registration. No paywall.

I've seen $300 courses that don't cover what they teach in the first 8 minutes.

Watch it and bookmark it now.

[Panzer_kommandant @Pz_kommandant]

بدون دید سیاسی و احساسی این ویدئو رو نگاه کنید یک دید حدود به شما میده که الان ادمینستریشن ترامپ نسبت به ایران و مذاکرات و جنگ چطوری فکر میکنن
ویدئو از @NickJFreitas
https://t.co/AUKLi1lChW

[3four3 @3four3]

It's not that he doesn't run.
He doesn't even walk.
He just stands there. 😂

[Robert M. Gower @gowerrobert]

Are you interested in the new Muon/Scion/Gluon method for training LLMs?
To run Muon, you need to approximate the matrix sign (or polar factor) of the momentum matrix. We've developed an optimal method *The PolarExpress* just for this! If you're interested, climb aboard 1/x

[Keller Jordan @kellerjordan0]

Modded-NanoGPT Optimization Benchmark

Hundreds of neural network optimizers have been proposed in the literature, recently including dozens citing Muon: MARS, SWAN, REG, ADANA, Newton-Muon, TrasMuon, AdaMuon, HTMuon, COSMOS, Conda, ASGO, SAGE, and Magma, to name a few.

The majority of this innovation is happening in the public research community. But the community currently lacks a widely accepted, easily accessible way to compare and make sense of the deluge of methods. As a result, promising new ideas get buried, and spurious results go unchallenged.

To help address these issues, I'm releasing a new optimization benchmark. It's designed for maximum simplicity and speed: Just a single file containing ~350 lines of plain PyTorch, which can complete a baseline LM training within 20 minutes of booting up a fresh 8xH100 machine. It also works with {1,2,4}xH100 or A100. These attributes make the new benchmark more accessible than any prior work.

The rules are simple: The optimization algorithm can be changed arbitrarily, with the goal being to minimize the number of training steps needed to reach 3.28 val loss on FineWeb (this is the same target loss as in the main speedrun). Modifying the architecture or dataloader, on the other hand, is not allowed. Wallclock time is unlimited, in order to give a fair chance to optimizers which would need kernel work or larger scale to become wallclock-efficient.

Like the main NanoGPT speedrun, submissions are open, and new results will be publicly broadcast. Beyond just improving the step count record, another goal of the benchmark is to collaboratively produce well-tuned baselines for as many optimizers as possible. For example, any improvement to the benchmark's best hyperparameters for AdamW would be considered a worthwhile new result.

This benchmark is not intended to be the final measure of optimizer quality across all domains. Convenient shared experimental infrastructure which covers the full space of possibilities -- across varying batch size, tokens per parameter, model scale, epoch count, and architecture -- is desirable, but far beyond the current status quo. This benchmark is only meant to be one step towards that goal.

To start the benchmark off, I've spent ~20 runs tuning baselines for Muon and AdamW. From time to time over the next few weeks, I'll add another optimizer from the literature, with my best effort at finding good hyperparameters. Researchers interested in neural network optimization are invited to join in by picking an optimizer and giving it a try on the benchmark. All optimizers are welcome, and even runs that don't necessarily have the best hyperparameters are desirable additions to the repo, because each new run adds to the collective knowledge.

[bluepoint @bluepoint_021]

روشی که دوستمون aminiyt گذاشته جواب میده

این روش برا من جواب داد ، سرعتشم خیلی خوبه ! فقط برا استفاده شخصی ، اگه بیشتر نیاز دارید پلن pro سایت vercel ماهی 20 دلار هست با حجم 1 ترابایت و 1 میلیون رکوئست ( نیاز به سرور خارجی و دامنه دارید )

ممکنه فردا vercel رو دوباره ببندن یا خود پروژه گیت هاب به خاطر fork بالا مشکلی براش پیش بیاد یا هر چیزی ....

حواستون باشه همون اول 20 دلار پلن vercel رو نخرید همون حالت اولیش 10 گیگ میده

https://t.co/IfG28PtYy9

[Mark Schmidt @MarkSchmidtUBC]

The first result of this type appeared in von Neumann's 1933-34 lecture notes. Here is a scan I made during my PhD.

[Sam Walters ☕️ @SamuelGWalters]

A kind of folklore theorem. If P is orthogonal projection onto a subspace M, and Q is orthogonal projection onto a subspace N, then the orthogonal projection R onto the intersection M ∩ N can be written as a simple limit of powers of PQ. #math #algebra

[difficultyang @difficultyang]

Let's say we wanted to rewrite PyTorch from scratch, because such a thing is topic du jour in the age of LLMs. What would the goals of such a rewrite be? What problems could a rewrite solve that incremental evolution from where the code is today not? 🧵

[HMD @hmdusf]

@bremen79 @DamienTeney And I am not saying all journal papers have a readable main text and appendix. But when the emphasis is on theory not application, the reviewers should be asked to check all proofs and if it's not readable, they should reject the paper.

[HMD @hmdusf]

@bremen79 @DamienTeney I think the emphasis in COLT is still on how well you write and motivate your problem in the 12-page main text. But there are so many cases that the appendix and proofs are unreadable. I think the readability of the proofs should equally translates to higher quality.

[HMD @hmdusf]

@bremen79 @DamienTeney I understand your comment. But, COLT prefers certain set of problems. You do online learning, they like it. You do robust statistics, they like it. There are communities inside COLT, and they make sure they get enough presence. Doesn't mean all the published papers are correct.

[Interesting STEM @InterestingSTEM]

They capture the exact moment when a developing heart shifts from silence to its first beat. There is no “switch”: many cells gradually become active and, upon crossing a critical threshold, the entire tissue suddenly synchronizes.

[Mehdi Allahyari @MehdiAllahyari]

این ریپو یک کتاب خیلی خوب و کامل راجع به طراحی و پیاده سازی سیستمهای ایجنتیک هست. هم میتونید آنلاین بخونید و هم دانلود کنید.
Github: https://t.co/xd1siTPO5o

[Lars van der Laan @LarsvanderLaan3]

I put together a guide on regret theory for empirical risk minimization (ERM) as I understand it.

The goal was to compile results and proof techniques I’ve found useful in my own work. I hope people find it useful more broadly

[Lucas Beyer (bl16) @giffmana]

Paper reading thread! Complete(d)P by fine folks at Apple (@jramapuram did the sigmoid attention, for example).

They try to further extend hparam transferability across ALL things one typically scales: width, depth, batch-size, training tokens. (Though they forget ctx len imo).

[Tomek Czajka @TCzajka]

I trained a board game playing bot that won an AI tournament: CodeCup 2026.

It learned to play by playing 100 million games against itself on 32 cores over the course of a week.

[Dawei Zhu @dwzhu128]

[1/n]

Super excited to introduce PaperBanana 🍌! (PKU x Google Cloud AI)

As AI researchers, we often spend way too much time crafting diagrams and plots instead of focusing on the ideas 🤯. To rescue us from this burden, we built an Agentic Framework to auto-generate NeurIPS-quality paper illustrations!

📄 Paper: https://t.co/2NbQeEhzMv
🌐 Page: https://t.co/05dKkjVs7f

Key Features:
🌟 Human-like Workflow: Retrieve 🔍 -> Plan 📝 -> Style 🎨 -> Render 🖼️ -> Critique 🔄. This ensures both academic fidelity and aesthetics.
🌟 Versatile: Supports both illustrative diagrams and statistical plots.
🌟 Polishing: Also effective for polishing existing human-drawn diagrams.

Here are some example diagrams and plots generated by our PaperBanana: