Top Tweets for #paperaday
@ID_AA_Carmack @mov_axbx Great #PaperADay recap! How does this compare to RL world models in your AGI pursuits? Thoughts, experts?
#PaperADay recap
On January 8th, I set out to read and take notes on one paper each weekday for the rest of the month. I missed one day due to a funeral, and another day due to bad time management, but not too bad.
I probably averaged a bit over 2 hours on each of them, which is only a rough read in some cases, but still enough to put a pinch in my work days. You can easily spend all day on a single paper if you dig in deep.
I have written code based on six of the papers so far, and the others are still kicking around in my head.
For now, back to my previous habits, but I may consider doing “week of papers” in the future after I digest where this fits in the exploration / exploitation time tradeoff.
15: Mastering Diverse Domains through World Models
14: MASTERING ATARI WITH DISCRETE WORLD MODELS
13: DREAM TO CONTROL: LEARNING BEHAVIORS BY LATENT IMAGINATION
12: Learning Latent Dynamics for Planning from Pixels
11: Discovering state-of-the-art reinforcement learning algorithms
10: LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics
9: floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RL
8: Beyond Gradient Averaging in Parallel Optimization: Improved Robustness through Gradient Agreement Filtering
7: Cautious Weight Decay
6: LOCAL FEATURE SWAPPING FOR GENERALIZATION IN REINFORCEMENT LEARNING
5: Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wasteful
4: Patches Are All You Need?
3: Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture
2: Deep Delta Learning
1: Emergent temporal abstractions in autoregressive models enable hierarchical reinforcement learning
#PaperADay 15
2024: Mastering Diverse Domains through World Models
(DreamerV3)
https://t.co/a5WCrd2uVW
https://t.co/bXbgtNJvYH
Applies the latest Dreamer model to over 150 diverse tasks, getting state of the art scores on many of them, but most notably, applies it to mining diamonds in Minecraft, a substantially harder challenge than most RL tasks.
The press reported this as “AI solves Minecraft”, which is misleading. After 30 million (20 hz) environment steps (17 days non stop) it mined a diamond. Unlike the Atari games, which are played with the same pixels and controls that a human uses, this is a modified interface with the inventory and stats presented directly to the model, and a categorical action space – no mousing around the inventory and crafting screens.
Mining had to be modified to instant-break instead of the normal multi-second hold of the mining button because Dreamer uses stochastic action policies, which are almost incapable of holding a button for hundreds of frames in a row. Similarly, the jump action required multiple frames of holding, so it was made instant.
Still, it was the first time an RL agent had gotten this far without having used imitation learning from human players, and significant improvements were made on all the other benchmarks as well.
The improvements were largely engineering grinds, rather than completely different architectures. I missed the “things we tried that didn’t work out” section from V2.
With the changes, they can profitably scale the model from 12M to 400M parameters, and the replay ratio from 1 to 64 times the environment rate.
The paper terminology is now closer to other RL papers: “Continue predictor” instead of “discount predictor” and using Pi for policy networks. The diagrams are improved.
With the jointly trained models, there is a tension between the representation model wanting to degenerate to make prediction easier and being useful for predicting following states. One of the tricks they use is “free bits”, clipping the losses when below a certain level so they don’t try to drive all the way to zero, allowing the opposing force to then make progress unopposed.
For the categorical distributions they use 1% label smoothing on the categorical distributions to avoid spikes in the KL loss. They call this “unimix” for mixing a uniform distribution on top of the existing distribution. Tthis is non-standard (versus label smoothing), but arguably better terminology.
They use a two-hot categorical value instead of MSE regression for the critic, but unlike most other implementations, use exponentially spaced bins instead of linearly spaced so they can cover several orders of magnitude. They define functions symlog() / symexp() to allow the networks to handle widely varying values in both positive and negative ranges. Reportedly works better than the similar non-linear transformation used in MuZero and Muesli.
This apparently required some care: “For computing the expected prediction of the softmax distribution under bins that span many orders of magnitude, the summation order matters and positive and negative bins should be summed up separately, from small to large bins, and then added.”
The final layer of the reward and critic models are zero-initialized instead of randomly initialized to avoid potentially large spurious values at the beginning of training.
The target model for the value function is now an EMA instead of a periodic copy.
To get the same amount of exploration from their policy gradient regardless of the scale of the value functions, they scale the (exponentially spaced, so potentially very large) returns to a bounded range, only considering the 5% to 95% range seen to exclude outliers.
They do 16 step prediction in imagination, but action selection in real environments is strictly by the policy model. Performance could likely be increased by doing imaginary lookaheads for every action selection like MuZero, at the expense of much slower runtime.
#PaperADay 14
2022: MASTERING ATARI WITH DISCRETE WORLD MODELS
(DreamerV2)
https://t.co/oWwHOYOvma
https://t.co/eSwdE95z6B
DreamerV1 was mostly targeted at continuous control tasks, but it also demonstrated basic playing of Atari games and DMLab tasks. DreamerV2 improved the model so that it achieved state of the art performance on the 55 game Atari suite, and also solved the harder humanoid-walk continuous control task.
This is very much an engineering paper, and I am here for it! In appendix C they summarize the changes that led to improved performance, and also (very rare in papers!) a list of things they tried that didn’t work out. Algorithms are shown in actual code with names instead of greek letters.
It is notable that they are only using 64x64 grey scale images as input, and those were downscaled from the common 84x84 resolution used by DQN, so it isn’t even a perfect 64x64 image from the source. Those are very blurry inputs for such good scores. I am curious if using 128x128xRGB images with an extra conv layer would improve performance, or if the extra detail would make it harder for the world model to train.
Their biggest change was replacing the VAE style gaussian latents, which were just 32 mean/var pairs, with categorical variables: 32 variables of 32 categories. They do not have a conclusive theory why this is so much better, but offer several theories. It would have been interesting to compare more gaussians against the larger categorical outputs.
The other big algorithmic change was “KL balancing”, or using a different learning rate for the prior and posterior weights, so the predictor trains faster than the representation. The joint optimization was apparently problematic for V1.
DreamerV1 struggled with exploration, and still had an epsilon-random action on top of the stochastic action policy. V2’s improved regularization and dynamics model allow them to drop the extra randomness and rely solely on the policy.
They do make some substantial changes in the KL loss and training setup for the continuous control versus discrete Atari control tasks.
They also scaled the models up and used ELU activation everywhere.
Their Atari evaluation protocol is good: full action space with sticky actions enabled. The scores are high enough that they recommend a new metric: “clipped record mean” scores – normalize to the human world record, clipping if it is above that, then taking the mean of all games. The historic Atari RL results have compared against “human” scores, which were originally some random people, then eventually a professional gamer, but for powerful agents in the 200M frame regime, this clipped record metric has merit.
During training over 200 million real environment frames, or 50 million action selections with action_repeat 4, 468 billion latent states were imagined, for nearly 10x the experience that a model-free agent would have seen.
The real environment experience is trained in batches of 50 sequences of 50 steps each. Sequences are constrained to not cross episode boundaries.
When training the policy and value functions, imaginary sequences are rolled out for 15 steps.
Values are MSE trained, not categorical. A traditional value target network is used, updating every 100 gradient steps.
#PaperADay 13
2020: DREAM TO CONTROL: LEARNING BEHAVIORS BY LATENT IMAGINATION
https://t.co/FbStuzbGFH
More than doubled the performance of PlaNet, and beat the state-of-the-art model-free algorithm of the day that used many more environment steps.
PlaNet (#PaperADay 12) wasn’t really a reinforcement learning algorithm – it learned to predict which states would get a reward, but it didn’t learn how to get more of them. If it was possible to get rewards inside the planning horizon of some dozens of steps, it could often find them, but the behavior was often “shortsighted”. There were several tasks where PlaNet didn’t do any better than random at an action repeat of 2, because it took more than a couple seconds of behavior to score. They crutched around it by using larger action repeats on those tasks in the original paper, but re-ran them all with a fixed action repeat in the tables here.
Traditional RL algorithms like Q learning and SARSA can propagate rewards far back in time using temporal difference learning and state or action value functions.
Dreamer basically adds traditional RL policy and value functions for the latent states, and learns them strictly in the imagined state transitions instead of from the online real environment interactions. This allows long term credit assignment like the model-free agents, but exploration can still happen in imaginary traces without having to test them in the environment (which can sometimes be misleading).
Some of their terminology confused me for a bit relative to other RL papers: What they call an “action model” I expected to be called a “policy model” (although action model might be better terminology!), and they predict a “discount factor” as a measure of the chance of an episode terminating, but discount factor usually refers to the exponential gamma factor in target bootstrap calculations.
They do a TD-lambda style weighting of all the possible imagined trajectory lengths to get their bootstrap targets, and they do stochastic backpropagation through the entire imagined trajectory to account for the stochastic action model decisions.
For learning the world dynamics models (in real experience, not in imagination), they predict latent-to-latent, but the loss is the pixel reconstruction from the predicted latent to the real observation. They experimented with contrastive estimation and pure reward prediction, but pixel reconstruction is simplest and worked best. It is basically like an autoencoder across multiple latent steps before going back to pixels.
The policy model is a little unusual: “The action model outputs a tanh mean scaled by a factor of 5 and a softplus standard deviation for the Normal distribution that is then transformed using tanh (Haarnoja et al., 2018). The scaling factor allows the agent to saturate the action distribution.”
In addition to the continuous control tasks from PlaNet, they also test on a subset of discrete control tasks from Atari and DMLab. I always smile at the use of the DeepMind Lab (DMLab) benchmarks, which were built on Quake 3, but with all the violence removed. These more complex tasks only use an imagination horizon of 10 instead of 50, presumably because the model was counterproductively bad beyond that horizon.
#PaperADay 12
2019: Learning Latent Dynamics for Planning from Pixels (PlaNet)
https://t.co/BYK9pysMxu
This was the forerunner to the Dreamer 1/2/3/4 series of RL agents / papers, which I am going to read in sequence.
Planning is common in tasks with fully specified transition and reward dynamics like board games, but it is much more challenging when you have to learn the “rules of the game” at the same time you are trying to improve your performance, especially when trying to do it from raw pixels instead of perfectly observed state features.
I sometimes semi-jokingly defend the position that “planning” might not actually be a thing, at least at low levels like this, and it just feels like planning when relevant experiences are called up from memory and bootstrap training on them results in changes to the current policy decision. There is a classic Atari paper that makes the case that replay buffers *are* a sort of non-parametric world model.
This paper gets peak performance “close to” strong model-free algorithms, but with far less real world experience necessary, because most of the work is happening in planning. Often, model based methods have to struggle to get to parity with the simpler model-free algorithms, and that is still going on with the Atari100k benchmark today.
This is a classic model-based system with a state transition and reward model. The biggest problem with transition models is usually that errors accumulate rapidly, so you can’t predict many steps into the future.
The transition models take a state plus an action, and predict the next state and the reward that results from taking the action. One of the key findings of the paper was that trying to learn a deterministic transition model basically failed. A stochastic model could be trained, but performance improved when they combined both deterministic and stochastic calculations in the model.
Looking closely at the video prediction frames in appendix H is interesting: once the deterministic GRU transition model lost the plot on a frame, everything after stayed broken, while the stochastic model could transition into something nonsensical on one frame, but then back to something sensible afterwards. I would not have guessed that. Their full combined model gave good looking predictions throughout.
There is no policy or value network as in model-free RL. Actions are selected by trying out a sequence of them using the modeled transition and reward functions, and the action that led to the best results is taken. Thousands of action sequences are evaluated for each action selected, but because they operate on compact latent vectors, this is relatively efficient. The Cross-Entropy Method (CEM) is used for planning multiple steps ahead with the transition models. This has to be heuristic for continuous action spaces or any more than a few modeled steps into the future.
The input to the state network is a 64x64 RGB observation (quantized to 5 bits like GLOW; I’m not clear why this is necessary). During training, they have an observation model that tries to go backwards from a state to a pixel observation. This is generally impossible to do perfectly when the state is smaller than the image, but attempting it provides a rich feedback signal for what to put in the state. This is not used for any part of the action decision process, it is just a training aid.
Action-repeat of 2 to 8, depending on the task.
Latent overshooting as a regularizer in latent space that encourages iterated one-step and multi-step predictions to match.
#PaperADay 11
Discovering state-of-the-art reinforcement learning algorithms
https://t.co/CVC6mwAevK
The paper is about replacing the traditional RL target generation algorithms (policy gradient, GAE, TD-lambda, etc) that look at a sequential block of agent frame predictions plus environment feedback with a generic LSTM meta-network that processes the same environment data and opaque prediction vectors from the agent to generate the targets.
After a lot of training, the meta-network can be frozen and used like any other hard-coded target generation algorithm, giving better performance than standard algorithms. Notably, even if the meta-network was trained on Atari games, it still provides state of the art performance on ProcGen and other environments that it had never seen before. A true example of transfer learning!
The meta-network is agnostic to the main agent network architecture – as long as the network outputs the right block of vectors, the meta network will generate targets for it.
They talk about an “observation conditioned prediction” Y and an “action conditioned prediction” Z, but the action conditioned prediction is still also conditioned by the observation (or state, more correctly, since it might be a recurrent state). These are the equivalents of classic V and Q functions, but of an arbitrary dimension instead of just being a scalar (just as you would for categorical value functions).
The additional Q and P output vectors are hard coded auxiliary tasks: standard Q learning with Retrace targets, and next-step policy prediction. They don’t impact the policy learning or the agent’s behavior directly, they just let the base model learn some additional features. Including them increases the agent performance by about 10%, but it clutters the point of the paper. It would be interesting to see if the same resources were allowed to be meta-learned alongside Y and Z if it would discover equally useful options.
All of the outputs are softmax normalized, and KL divergence between the predictions and targets is used to generate the loss.
To build the targets, the meta network LSTM is run on 29 multisteps at a time. They found the LSTM was as effective as a transformer, and more efficient. They have an additional meta-RNN for longer term adaptation that slightly improves overall performance, but it also seems to clutter the point a bit.
Meta-learning is often discussed in relation to optimization parameters, but here all of those were swept like normal: optimizer, learning rate, weight decay, and gradient clip point. Meta-optimizing these could probably piggy-back on the same meta architecture as the target meta learning, but the paper is clearer without it. There is still an order of magnitude difference in performance between the best and (still plausible) worst manually tuned hyperparameters.
A normal training run with the frozen meta network only took 40% of the time that MuZero did, because the target generation is faster, but presumably it is slower than Muesli.
They train with a supervised learning style warmup and cosine-decays learning rate for Atari, which is a bit unusual.
The agent parameters are reset after mostly shorter trajectories like 20M, with only a few going for 200M, because it was observed that most of the learning happened early on. I’m curious if there was much benefit to the longer runs at all, or if they could have still learned an equivalent algorithm with just 20M or even shorter training sessions.
Google filed a patent application on aspects of this work – Yikes!
#PaperADay 10
LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics
https://t.co/9Q5hkRogbr
The comments on #PaperADay 3 recommended this paper as the state of the art JEPA paper, and it does look much better!
They acknowledge that much of the prior JEPA research is ad-hoc and full of heuristics, but here they make strong theoretical claims of optimality and provide proofs (which I did not read).
The first claim is that isotropic gaussian is the unique optimal embedding distribution for both linear and nonlinear probing, minimizing worst-case risk across downstream tasks. I would have taken that on faith with just a “sounds good to me”, but they go into it with details and examples.
Actually getting an isotropic gaussian in high dimensions is easier said than done. They present Sketched Isotropic Gaussian Regularization (SIGReg) as a well behaved loss function to achieve this after analyzing a number of different statistical tests, and they claim it beats the curse of dimensionality with linear scalability.
The final loss is just a blend factor to weight the JEPA prediction loss against the SIGReg isotropy loss. This is the one tunable hyperparameter for LeJEPA.
Despite the P in JEPA, they don’t use predictor networks here, they just directly compare view embeddings for the JEPA loss. Predictor networks could still be useful for video sequences, especially when conditioned with action information for agents / robots.
Each training image is augmented to produce 2 global views and 6 local views with different spatial scales but the same set of color and geometric transformations. The loss is the average MSE between the average of the global view embeddings and each of the local view embeddings.
I don’t have a good feel for the tradeoffs in their view transforms, which still seem very much in the ad-hoc space, but they will determine the nature of what gets filtered out of the representation. Learning what doesn’t matter is critical, but the specification of “matters” is only implicit in the view transformations.
LeJEPA itself is architecture independent – anything that digests a batch of samples from a dataset into vectors can be used. Vision transformers, MLP, ConvNets, etc. The specific augmentations for views would be input modality specific, but the LeJEPA algorithm could work on audio, images, video, or other things.
They show that the LeJEPA loss on a large foundation model is very indicative of downstream task performance, both directly, and with a heuristic to improve the predictive power of the loss farther.
They also show that it can be used to train from scratch on small datasets with as few as 1000 samples and achieve better results than probing a conventional general foundation model.
I was pleased to see sample code blocks in the paper instead of greek-laden pseudocode, as well as a github repo.
Appendix D has interesting details on generating good coverage of unit hyperspheres with low discrepancy samples by transforming Sobol sequences, but this is only for their theoretical analysis, and they show you are better off just making new random hypervectors every batch, with even 16 random vectors outperforming a fixed set of thousands.
Some questions:
In the discussion of non-linear probing, only kNN and kernel methods are mentioned, presumably for their theoretical analysis tractability, but would an MLP generally perform better?
A JEPA embedding is not fully reversible like NICE or a RevNet, so how does it react to inputs that are far outside the training set? Will novel inputs map to unique embeddings, or could they be collapsed onto the codes from the training set?
How would the embeddings evolve in a continuous learning environment, as novel inputs are added to the training mix?
Can a JEPA be overtrained – is lower training loss always better, or would there be an optimal early stopping point?
#PaperADay 9
floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RL
https://t.co/V4kCMfUkEK
In theory, value based reinforcement learning is a regression problem, which is most naturally addressed with an MSE loss. However, there are a bunch of subtle reasons why a neural network terminating in a single activation may not be an ideal value function approximator.
Most models today use a categorical value representation to mitigate some of the training issues, but the proposal here is to generate the categorical value with around 8 steps of a flow model, which plausibly makes it easier to represent complex mappings.
This kind of refinement-from-noise is appealing to me as a possible way to be sort of “error correcting” and combat the problem of catastrophic forgetting. If you learn something well, then train on something else for a few million optimizer steps, the odds are good that your original task performance will be almost back to random if you try it again. Hints of the original behavior remain, but covered in a lot of noise. Some kind of de-noising on the values would be a big help.
I’m not familiar with the benchmarks they use, but they appear to be comparing with a previous model, which may not be State Of The Art, and there is also some per-environment hyperparameter tuning. They target offline RL, optionally followed by a period of online fine-tuning, so there may be issues with the learning dynamics when going completely from scratch.
Their flow model is a 4-layer MLP, and I’m guessing they have a modestly sized explicit state. I’m not sure what the training dynamics would be if you had to co-learn a CNN feature extractor with the flow model.
They compare the flow model against both an 8x ensemble of value functions (going wide) and a ResNet with 8x the layers (going deep) and find that the flow model is notably superior.
The details of the ensemble could be important – an ensemble of models trained on the same sequence of samples doesn’t help much in RL, but an ensemble trained with completely different IID samples can make a difference.
One of the key points about flow models is that they get supervision at every step, unlike a very deep traditional model that only gets supervision at the very end, which is backpropagated through all the layers. I wonder if you could train the deeper res-net with similar supervision at multiple interior points.
They presume to know ahead of time what the range of possible Q values are to set the range of the initial input noise, and it is a sensitive parameter on some of the benchmarks, which could be problematic.
Broad categorical value representation was essential; doing flow based on a scalar worked poorly.
The time input to the velocity network is a multi-channel fourier basis, using a scalar time also worked poorly.
I have been meaning to learn more about flow models, this has been a good prod.
#PaperADay 8
Beyond Gradient Averaging in Parallel Optimization: Improved Robustness through Gradient Agreement Filtering
https://t.co/C1KP2qx27Y
Recommended by @FrancoisChauba1 after yesterday’s paper.
Proposes Gradient Agreement Filtering (GAF), where independent gradients that have large cosine distances between them are filtered out instead of being averaged in. The implication is that some calculated gradients that may help with training loss are actively harmful to generalization, and should be discarded instead of used.
This is presented in terms of micro batches in multi-GPU distributed training, but the same idea should apply to partitions of any training batch.
I was coincidentally just trying something similar – clipping gradients from IID replay buffer samples so they don’t contradict the gradient from the current online experience sample. It hasn’t shown positive results yet, but I have a few more angles to try.
The observation that motivates the theory is that if you train an image classifier on completely random noise and random labels, it will have 100% train accuracy and only random chance validation accuracy, clearly overfit to the training data. They note that if you look at cosine similarity between the gradients of minibatches on this overfit model, it is always above 0.99, or essentially orthogonal. If orthogonal gradients are a reliable sign of overfitting, maybe you can skip minbatches with orthogonal gradients and reduce overfitting / increase generalization.
For the simplest case of two microbatches, this comes down to either keeping or rejecting both of them based on the cosine similarity, but with more microbatches they propose comparing all of the microbatches to the first one, and averaging together all the ones that pass the test.
Some of the comments about batch size are in tension with the thesis in #PaperADay 5 that claims failures of batch size scaling are due to not adjusting beta2, but justifications don’t matter nearly as much as empirical performance. I will probably try something like this in the next few days on our RL codebase.
#PaperADay 7
Cautious Weight Decay
https://t.co/EzgZbK4WRJ
This is a 36 page paper about a very simple idea:
Don’t apply weight decay when it is in opposition to the current optimizer step.
If the step is moving the weight farther from zero, there is no decay. If the step is towards zero, decay moves it in faster.
They spent 20,000 H100 GPU hours (about $60k!) testing this across multiple optimizers and models, and it looks like it is basically always a modest improvement, with no changes to any hyperparameters.
My current models use weight norm on most of the parameters, but there are still some with traditional weight decay. A first test with this idea does seem to be a tiny improvement, but I will need to do more runs to have confidence in it.
Two modifications of the idea come to mind:
Use the current gradient instead of the optimizer step (which includes momentum) for masking, as in https://t.co/Q7yiRAyw19. If using a cautious optimizer, explicitly masking the optimizer step before calculating cautious weight decay would do this automatically.
Weight decay is an exponential effect, which mixes with a linear learning rate. It might be interesting to just have a larger learning rate when the step is heading towards zero than away from it, which would do similar things, but with different learning dynamics.
#PaperADay 6
LOCAL FEATURE SWAPPING FOR GENERALIZATION IN REINFORCEMENT LEARNING
https://t.co/n1xj5BRqNX
There is a good discussion of generalization, both in general (ha) and more specifically in RL, but the idea presented is very simple, and I’m going to give it a try:
CLOP: Channel-consistent local permutations
Given a 3D tensor (4D with batch), with some probability at each location, randomly swap position with a neighbor, swapping all channels as a unit. Like dropout, this reduces overfitting by co-adaptation, but it doesn’t zero any channels out, it just moves them.
I agree with the idea that data augmentation in the latent space is more efficient for generalization than in the input space. They suggest doing it as low in the spatial hierarchy as possible, but it probably wouldn’t be a good idea at a 2x2 level, where there are only four possible permutations and any of them disturb half the spatial information.
Note that they tuned the swap chance per-game, which is generally not done when reporting results on a suite of games.
The results on pure supervised learning tasks weren’t noteworthy, but might be better with the CLOP inserted in different places and with different training recipes.
#PaperADay 5
Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wasteful
https://t.co/rPtY2GMN8i
This is written in terms of LLMs, but I believe the result should be true across other models.
The main point is that prior works that showed small batch size training working worse on a per-sample basis were only true if Adam’s beta2 parameter was not also adjusted with the batch size. With proper adjustment, small batch training should be equal or better than large batch training for a given number of processed samples.
I was always surprised to see people advocate doing gradient accumulation over iterations (as opposed to in parallel across nodes) as a useful trick. It should always be better to take a step!
The proposed scaling rule is: When changing the batch size, raise the existing beta2 to the NewBatchSize/OldBatchSize power. As your batch size shrinks, beta2 gets closer to 1.0. This fits with the typical beta2 parameters used with large-batch LLM training being substantially lower than the default 0.999 commonly used for moderate-batch image processing.
Learning rates need to shrink with smaller batch sizes, but they don’t propose a scaling rule. They note that the optimal Adam learning rate scales much less than even the common sqrt(batch) suggestion. Going from batch 1 to batch 1024 only shifted the optimal lr by 3x, not 32x.
A somewhat surprising result is that small batch sizes are more robust to hyperparameters like lr and beta, with much larger basins of near optimal performance, in contrast to peaky optimums for large batch training.
Also surprising is that the differences between fancy optimizers shrink as the batch size shrinks. At batch size one, even momentum is unnecessary, and vanilla SGD can match Adam’s typical large batch performance. Properly tuned batch-1 Adam gets a little better.
They set weight decay to zero for their batch-1 experiments, but that is probably a mistake. Some decay is important to reduce the impact of “noisy features”, regardless of the optimizer.
They point out that the memory savings from a stateless optimizer can be 75%, so it may become practical to do true fine tuning of an entire model instead of LowRank Adaptation.
In general, you still want to use a batch size large enough to get full GPU utilization, but you should be able to change beta2 at that point and match the final model performance of larger batch training.
#PaperADay 4
Patches Are All You Need?
https://t.co/dGXF57Viqr
This is an older paper that is very relevant to my interests.
In value based RL, modern big vision models have been surprisingly unhelpful. I have tried multiple times to drop in ConvNeXT and other high end models, and they wind up performing worse than my simple MaxPooled seven layer network, despite having many more parameters and running many times slower. Trying to use vision transformers for value regression has also not worked out well.
Clearly these big models are doing something right to perform well on the traditional benchmarks, but it is hard to disentangle which architectural features are harmful for the RL task.
This paper proposes an extremely simple model architecture (it can fit in a 280 character X post!) that, while not as good as the most sophisticated and highly tuned models, is better than baselines. That is often valuable!
The idea is that while models like ViT and MLP-Mixer have very different block structure, in common they have the design of "patchifying" once at the top level, then working isotropically for all the layers. Maybe this is the most important aspect.
The ConvMixer patchifies, then just alternates unusually wide depthwise convolutions with pointwise convolutions, and it looks like it works well.
Wide depthwise kernels are parameter efficient, and give the layers close to the global reach of attention layers, while being fast and easy.
I think the isotropic nature is the real win, and non-overlapping patches are just a detail. I wouldn’t be surprised if you got slightly better performance by making the initial patch embedding layer moderately overlapping.
I suspect that for deeper networks, reorganizing the residual so it is pre-activation around the entire block would help.
I was able to start running experiments with these ideas right after reading the paper, which is always gratifying!
#PaperADay 3 (hoping embedded links deboost enough that not too many people are annoyed with this content)
@ylecun has been topical recently, so today I went through:
Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture https://t.co/kYP2E8rhD0
I am broadly in agreement with the idea that the important predictions are of internal representations, not pixels, so generative models may be somewhat counterproductive, or at least needlessly inefficient for many tasks.
However, I tend to think that the internal prediction has to be happening at a more granular level than full image processing, at the minicolumn or even neural level, and with more of a temporal component than local masking.
Self supervised training works on a large dataset with no idea what will be asked of the model later, just building up knowledge from the data. Afterwards, you can train a simple linear classifier (linear probe) on the output and get quite good performance. The best linear probes on frozen self supervised models are not as strong as end-to-end trained classifiers, but the exact same SSM can be strong for many different tasks at the same time.
The paper notes that in contrast to JEPA, Invariance-based training methods that take the same image and augment it two different ways while maintaining representational similarity get their performance at the expense of a researcher biased set of image augmentations, which doesn't transfer to other modalities like audio or text. I note that JEPA is very sensitive to the exact masking performed (table 6), which doesn’t feel too different.
The target encoder is superficially similar to the modern formulation of the target model in DQN RL networks with an EMA of the weights instead of an occasional copy, but while it was a stability aid for RL (and isn’t always necessary), it has a more fundamental purpose here to prevent the model from collapsing representations into trivial to predict ones. This, along with LayerNorm also being a crucial element of that, is not spelled out in the paper, and I had to find references to it elsewhere.
Kind of quirky that they apply a random 0.85-1.0 crop to the context, but only remove blocks from the right and bottom. I expected to see an ablation of that crop.
Increasing the image resolution is a bit of an odd way to scale the model. It probably isn’t actually resolution that helps, but total patch count.
There is a large body of work on self supervision that I am only vaguely familiar with, so I’m probably missing some key distinguishing aspects of JEPA. I’m still grappling with the core question of exactly what the contexts learn, and how the model architecture and training guide it away from collapse.
#PaperADay 2
2026: Deep Delta Learning https://t.co/nKj9NE1ri6
The standard residual network blocks are limited to adding on top of the existing state, which limits the expressivity of each layer. It is still a universal approximator, but we can always hope for function blocks that are more parameter / performance / training efficient.
This paper proposes a new block based on generalizing the Householder matrix so that the state can be partially or completely collapsed onto (or past) a hyperplane in addition to having new values added on top. This allows information along a vector to be multiplicatively erased while a different vector is added.
This is an all-math, no-experiments paper, which usually doesn’t bode well.
The same scalar gates both the directional collapse and the value addition. Those might be better off independent, like input and forget gates in an LSTM.
When the gating vector is 0.0, the layer is an identity. At 1.0, the previous state has been collapsed onto the learnable hyperplane before adding the new value, and at 2.0 the previous state has been reflected across the hyperplane.
They force the gate scalar to be bounded between 0 and 2 by using 2*sigmoid, but that means that default initializations will tend to project everything onto the plane, collapsing values quickly, and you can’t actually reach a perfect identity.
Is it actually necessary to bound the gate value? Letting it extend past those bounds just results in scaling along the vector, and avoids the sigmoid gradient issues.
They normalize the reflection direction after calculating it with an MLP, which can cause some learning dynamics issues since the weight norm can grow without bound, which reduces the effective learning rate with Adam.
I like and bookmark so many interesting sounding papers here, and don’t get back to most of them. Time to start making a dent. I’m going to try to at least skim one of the papers in my bookmarks each weekday for the rest of the month.
#PaperADay
2025: Emergent temporal abstractions in autoregressive models enable hierarchical reinforcement learning (Google)
I like their statement of the hierarchical goal problem as “how long does it take a twitching hand to win a game of chess?” @RichardSSutton is fond of the “options” framework in RL, but we don’t have a clear method to learn them from scratch.
Their Ant environment is designed to require two levels of planning: the standard mujoco Ant locomotion work to be able to move at all, and routing decisions to get to the colored squares in the correct order, which will happen hundreds of frames apart.
Basically, this takes a pre-trained sequence predicting model that predicts what separately trained expert models (manually steered) do, and inserts a metacontroller midway through it, which can tweak the residual values to perform high level “steering”, and can be RL’d at high level switch points to much greater performance than the base pre-trained model.
A key claim here is that learning to predict actions in a supervised next-token manner from lots of existing expert examples, even if you don’t know the goals, results in inferring useful higher level goals. This sounds plausible, but their experiment makes it rather easy for the model: the expert RL models that generated the training data were explicitly given one of four goals in each segment, and the option learning model just classifies the sequences into one of four categories. This is a vastly simpler problem than free form option discovery.
A State Space Model is used for the more complex Ant environments, while a transformer is used for the simpler grid world environments. I didn’t see an explanation for the change.
The internal “walls” are more like “poison tiles”, since they don’t block movement like the map edges, they just kill the ant when its center passes into them.
The 3D renderings (with shadow errors that hurt my gamedev eyes) are somewhat misleading, since it is really a 2D world that the agent gets to fully observe in a low dimensional one-hot format. It doesn’t do any kind of partially observed or pixel based sensing.
Everything is done with massively parallel environments, avoiding the harder online learning challenges.
The success rates still aren’t great after a million episodes.
I would like to see this applied to Atari, basically doing GATO with less capable experts or lower episode quantities, then trying to identify free form options that can be usefully used to RL to higher performance.
Standard reinforcement learning in raw tokens is a disaster for sparse rewards!
Here, we propose 𝗜𝗻𝘁𝗲𝗿𝗻𝗮𝗹 𝗥𝗟: acting on abstract actions emerging in the residual stream representation.
A paradigm shift in using pretrained models to solve hard, long-horizon tasks! 🧵
Today's #PaperADay is :
Believe it or not: how much can we rely on published data on potential drug targets?
https://t.co/Kx24yRtCbd @NatRevDrugDisc #scientificreproducibility #biotech #YesStillReadOldPapers
This is pretty wild. Has me thinking about all the pitfalls, but I love it when ppl approach a things from new angles https://t.co/BqY7bQGKqf
Basically: take a bunch of genomes, make "pseudo images" from the data, do image classification/clustering #paperAday #bioinformatics
DeepGOMeta: Predicting functions for microbes} https://t.co/QjjVIylyj1 #biorxiv_bioinfo
Most Popular Users
Elon Musk 
@elonmusk
240.8M followers
Barack Obama
@barackobama
119.2M followers
Donald J. Trump
@realdonaldtrump
111.7M followers
Cristiano Ronaldo
@cristiano
111.2M followers
Narendra Modi
@narendramodi
107M followers
Rihanna
@rihanna
97.8M followers
NASA
@nasa
92.2M followers
Justin Bieber
@justinbieber
91M followers
KATY PERRY
@katyperry
88M followers
Taylor Swift
@taylorswift13
81.8M followers
Lady Gaga
@ladygaga
73.3M followers
Virat Kohli
@imvkohli
70.4M followers
Kim Kardashian
@kimkardashian
69.9M followers
YouTube
@youtube
68.7M followers
Bill Gates
@billgates
64.1M followers
Neymar Jr
@neymarjr
63.2M followers
The Ellen Show
@theellenshow
62.4M followers
CNN
@cnn
61.9M followers
Selena Gomez
@selenagomez
61M followers
X
@x
60.8M followers