apaz

GPUs go brrr

[apaz @apaz_cli]

Releasing mlsweep, a sweep scheduler and visualizer for distributed ML training. It aims to make launching runs across groups GPUs frictionless and achieve near feature-parity with wandb. But you can use it with whatever frameworks or loggers you like, wandb included.

[apaz @apaz_cli]

@stochasticchasm This is my understanding as well, yeah. There may also be other benefits, but I think this is a very reasonable arch.

[Experiential Labs @explabsai]

DeepSeek V4.1 Flash 1017 uncensored is now 50% off

Openrouter:
doesn't serve it

Competitors:
$0.60 in / $1.60 out

Experiential:
$0.15 in / $0.60 out

[apaz @apaz_cli]

@parth220 sure lol

[apaz @apaz_cli]

@code_star https://t.co/Ev7SKzsWLo

[apaz @apaz_cli]

@liz_love_lace May I recommend listening to dwarf music on repeat for multiple days while going slowly insane, almost succumbing to delirium before finishing your data cleaning experiments
https://t.co/BC2O6ZL8Km

[apaz @apaz_cli]

@willcb Disappointing

[apaz @apaz_cli]

Why does this take two years to build what am I missing, I feel like I could do this in a couple weeks.

[DataVendor @datavendor_ai]

We're announcing the public beta of https://t.co/YgMb6QF79k

We spent the last two years building the best solution for people buying and selling data, whether you're a researcher in a lab or a company trying to sell data.

Data is the fundamental bottleneck for AI progress, and DataVendor will be the infra that solves it.

[apaz @apaz_cli]

@username_is_unq Which subgradient methods are you referring to?

[apaz @apaz_cli]

@username_is_unq The comms overhead of Dust is higher (like DDP not like sending scalars) but it is worth it due to the SNR issues I was talking about earlier. How your SNR scales with model size is really really important and is most of what made evolution strategies not viable for pretraining.

[apaz @apaz_cli]

@username_is_unq No, but there are non-differentiable results for es at scale/eggroll (they train an integer RNN in one of the papers, I forget which) and there's no reason you can't also do this with activation perturbations.

[tender @tenderizzation]

what if you just shot your infra engineers

[apaz @apaz_cli]

@username_is_unq It has a SNR of 1, so yes. Backprop just tells you the grad. It's superior in every way except that it requires the architecture and loss function to be differentiable. Hence why people use it over ZO.

But these are not an assumption for ZO, which is why it may be worth a look.

[apaz @apaz_cli]

@liz_love_lace May I recommend listening to dwarf music on repeat for multiple days while going slowly insane, almost succumbing to delirium before finishing your data cleaning experiments
https://t.co/BC2O6ZL8Km

[apaz @apaz_cli]

Apologies to those I have neglected while I have been dwarfmaxxing, hit me up. I have left the data mines and returned to the land of the living.

[apaz @apaz_cli]

@gabriel1 @menhguin @Apple @Google @MIT You got cooked but you can just take it in stride. You don't have to harbor petty resentments.

[apaz @apaz_cli]

@lu_sichu I agree with this but also I have money in a bank, don't have three weeks of backup food in my apartment, can't exactly airgap anything, etc. I am not sure how to take the advice seriously in a way that's practical.

[apaz @apaz_cli]

There are many ZO papers with results like this. Many different optimizers which choose directions to perturb in, using one manner of trickery or another.

It's pretty obvious that these need to be adapted to Dust, and re-tested. I do not expect the results to move the needle. But it's probably worth setting a bunch of research agents on.

The holy grail would be solving it for large networks, in a pretraining context. I think we still do not understand what happens to these loss landscapes to make SPSA-like algorithms fail, and better understanding of what's going on would be huge.

[Pyuyi @Pyuyi2333]

Conventional zeroth-order optimizers such as MeZO estimate gradients through random parameter perturbations, wasting evaluations on directions unrelated to meaningful updates.

AGZO from ICML26 exploits a simple but powerful geometric property: the weight gradient of a linear layer lies in the subspace spanned by its input activations. Rather than sampling perturbations across the entire parameter space, it extracts low-rank activation subspaces during forward computation and restricts gradient estimation to those directions.

Under spectral concentration assumptions, the authors prove better expected alignment with the true gradient than isotropic perturbations. On Qwen3-0.6B, AGZO achieves 89.2% accuracy on CB versus 80.3% for MeZO, while enabling Qwen3-4B fine-tuning in a hardware setting where full first-order training runs out of memory.

The deeper insight is that avoiding backpropagation does not require treating neural networks as structureless black boxes. Forward activations already reveal useful information about optimization geometry, connecting zeroth-order learning with randomized linear algebra and low-dimensional representation structure. However, the method remains substantially more iteration-intensive than first-order training, and its advantages have not been established at frontier pretraining scale.

The next question is whether adaptive activation subspaces can reduce both gradient-estimation noise and the number of forward evaluations required for effective learning. Perhaps the next generation of optimizers will use forward computation not merely to evaluate loss, but to determine where parameter updates should occur.

https://t.co/3MoXumbFpw

[apaz @apaz_cli]

This concurs with my own findings on training looped transformers, by the way. Although their work is much more polished than mine.

[Siyu Chen @siyuc3141]

Looped transformers often save compute by truncating backpropagation to the last few loops.

Is it worth it? We trained ~600 models to find out 👀

Short answer: only if the loop count is fixed. Otherwise, run fewer loops and train them all!

📄 https://t.co/4vPIXEUdDJ
🧵👇

[apaz @apaz_cli]

@apaz_cli EGGROLL and dust scaling isn’t the same. EGGROLL only needs the seeds

[apaz @apaz_cli]

The Dust paper is really big. A breakthrough.

After spending a few days to really digest it though, I'm somewhat disheartened that so much of it is implementation details which are not explained. I'm working on a blog post to reverse engineer the math, methods, and choices.

[Samip @industriaalist]

Backprop has been the only credit assignment algorithm capable of training large neural nets.

Introducing Dust, the first zeroth-order method to pretrain transformers to approach and sometimes even *exceed* backprop with large amounts of computation (bitter lesson!)

- Dust is on the order of 1,000 to 10,000x more compute efficient than EGGROLL, the state-of-the-art ES method, for training transformers.

- We chose pretraining transformers because it's arguably the hardest task possible for zeroth-order optimization.

- Search in high dimensions is really misunderstood: zeroth-order usually gets better with overparameterization, not worse, i.e. at a fixed population size, larger models (up to 120x) reach a lower loss than smaller ones.

- Backprop-like gradients emerge from a large population of activation perturbations, and the alignment holds up at every scale we tested, up to 1B tokens, which is critical for scaling.

The core idea is a *virtual population*, which helps scale to large populations in transformers by perturbing activations in parallel.

w/ @bishmdl76, @cs_serdar, @akshayvegesna

[apaz @apaz_cli]

@username_is_unq If you want to scale eggroll or dust horizontally it's the same, you can increase the batch size or perturbation count.

The case for Dust is that it's an estimator that gets you a much better SNR for free, so you have more accurate gradients. This turns out to be very important.

[apaz @apaz_cli]

@willcb Bro I have been talking about this for literally months what do you mean

[apaz @apaz_cli]

I actually don't see any original ideas though?

Like, isn't this just Forward Gradients With Local Losses (https://t.co/7Ox75OeF0c), but with a full Hadamard basis instead of random directions, and using a shared logit lens head as the local loss?

@cs_serdar and I were just talking about it in ZO Chat