Thanh-Dung Le

Deja vu

[alphaXiv @askalphaxiv]

"Contrastive World Models"

If world model has to reconstruct every pixel, it'll pretty much waste most of its capacity modeling irrelevant background noise.

So this paper removes Google DeepMind's Dreamer pixel decoder and instead trains the latent state to identify features of the correct future observation.

It matches Dreamer on clean environments, but performs much better with moving distractors and natural-video backgrounds.

https://t.co/S5b0QWmwuH

[Harsh Bhatt @harshbhatt7585]

Morning Read.

[Gabriele Berton @gabriberton]

Here are two interesting papers (vision and NLP) on the topic of memorization vs generalization.

Each investigates the issue from a task-specific perspective, and the solution for generalization is different for the 2 domains. [1/4]

[Probability and Statistics @probnstat]

Variational inference is a method for approximating complex probability distributions by turning inference into an optimization problem. Instead of trying to compute an intractable posterior distribution exactly, it selects a simpler family of distributions and finds the closest one by minimizing a divergence such as the Kullback–Leibler distance. In probability theory, this provides a principled way to approximate Bayesian models and study uncertainty in high dimensions. In machine learning, variational inference is central to Bayesian neural networks, topic models, and variational autoencoders, enabling scalable learning from massive datasets while retaining uncertainty estimates. In real life, it supports applications such as recommender systems, medical diagnosis, finance, and robotics, where decisions must be made under uncertainty and models must update efficiently as new data arrives.

[Vishal Misra @vishalmisra]

New work: Do transformers actually do Bayesian inference?
We built “Bayesian wind tunnels” where the true posterior is known exactly.
Result: transformers track Bayes with 10⁻³-bit precision.
And we now know why.
I: https://t.co/e5O5O3QN5U
II: https://t.co/11KMePLUaT
🧵

[Igor Mezic @IgorMezic]

This interesting paper connects
@ylecun's JEPA architecture to Koopman invariants. I am confident more is to come. https://t.co/6bfSYh4VoT #AI #JEPA #ML #DynamicalSystems #ControlTheory

[Mike Shou @MikeShou1]

Very nice tutorial of Linear Attention and its variants -- comprehensive and easy to understand!

[Wenhao Chai @wenhaocha1]

From this project, I mainly learned three things:

1) Representation learning can fully emerge from generative objectives. In the language domain, this has almost become a consensus. However, in vision, discriminative representation learning methods such as CLIP and DINO still dominate. On the other hand, while diffusion-based visual generative models are trained with generative pretraining, their representations are still not very strong, as noted in REPA. We hope that scaling NEPA can enhance visual representations from a different paradigm.

2) Modeling different modalities with the same autoregressive pattern is feasible. We show that the simplest next-token (embedding) prediction works equally well for vision. Going further, we believe this should be done in latent space rather than pixel space.

3) Architectural design matters. By introducing modern architectural components such as RoPE, LayerScale, SwiGLU, and QK-Norm, we not only observe faster convergence and better final performance, but also significantly more stable training. The computer vision community should learn more from advances in large language models.

Proud of the team. Take a look!

[Weiyang Liu @Besteuler]

Something related we have done in the past: SphereNet (https://t.co/hX7SfruRUs), Decoupled Networks (https://t.co/GLyh3eazFw). The central idea is to find a bounded activation to replace the normalization back then in CNNs. We have tried a few angular activation functions. Could be interesting to check them out. :)

[Alexia Jolicoeur-Martineau @jm_alexia]

Instead of training an autoencoder from scratch, they start from a pretrained vision encoder to train their own.

It takes 3 training stages to achieve this. But once you have the autoencoder, training a diffusion model in this latent space converges super fast.

[Damien Teney @DamienTeney]

Can vision transformers learn without images?🤔👀
Our latest work shows that pretraining ViTs on procedural symbolic data (eg sequences of balanced parentheses) makes subsequent standard training (eg on ImageNet) more data efficient! How is this possible?! ⬇️🧵

[Frank Nielsen @FrnkNlsn]

Minimizing Kullback-Leibler divergence interpreted as an information projection wrt to Fisher orthogonality and exponential/mixture connection.

Uniqueness of projection proof proved with a dual generalization of the Pythagoras' theorem!

PDF (≈1MB):
https://t.co/VQE7wcTlXn

[return of the research era ꙮ @byebyescaling]

Nice mathematical ideas + empirics in this paper! They adapt frozen DINOv2/SigLIP for generation using a Feature Auto-Encoder (FAE) with just one attention layer.

The core insight is decoupling feature reconstruction from pixel generation. They minimize

$\mathcal{L}_{VAE} = ||\hat{x} - x||_{2}^{2}$

to force the 32-dim latent to preserve the semantic geometry of the teacher model before the pixel decoder takes over. What's really cool is the tricks they use!

1. Gaussian Embedding Decoder: They pre-train the pixel decoder on frozen features perturbed by noise ($\tilde{x} = x + \epsilon, \epsilon \sim \mathcal{N}(0, \sigma^2 I)$). This acts as a robust prior, stabilizing the manifold mapping immediately.

2. Time Shift: Applying time-shifting to the flow-matching trajectory bridges the convergence gap between low-dim (32d) and high-dim latents.

This achieves SOTA FID (1.29) on ImageNet-256 with 10x training efficiency (80 epochs vs. standard 800).

[Kyunghyun Cho @kchonyc]

love it because RNN, because perturbation in the parameter space, and because it works better than PO!

a historically relevant study by @wellecks is at https://t.co/CkAboq1lyM

[MIT CSAIL @MIT_CSAIL]

15 courses for understanding neural networks, v/@tut_ml: https://t.co/lzXxV6Nkyq

[Jason Lee @jasondeanlee]

Haozhe's paper is worth a read, really nice use of fixed point theorems. The new one about 1 to 1 seems almost immediate though Just from reading the thread only I would guess the proof is as follows: Say your input space is discrete in j \in [n] and represented by x_j. The embedding is E: [n] \to R^d . "For almost all E", E x_j are distinct. Then a transformer f is composed of building blocks that are analytic, composition of analytic is analytic and also preserved under many algebraic operations (o-minimal stuff) . Analytic functions are either the zero function, or crosses zero on a measure zero set (they can't have positive measure set f(set) =0) . Thus so long as f \neq 0 uniformly, this should be injective (not bijective necessarily its not onto).

[Rohan Paul @rohanpaul_ai]

New Nvidia + UC San Diego paper builds an automated way to read huge research fields and tell where to work.

It automates high-quality field surveys and trend tracking.

Research moves too fast for humans to track 10,000+ papers a year.

The authors create Real Deep Research, a pipeline that gathers papers from top venues, filters scope with targeted prompts, and turns each paper into a compact structured summary.

For foundation models it logs input, modeling, output, objective, and training recipe, which means data in, how the model works, what it produces, what it learns toward, and how it is trained.

For robotics it logs sensor, body, joint outputs, action space, and environment, which together describe how a robot senses, moves, and acts in the world.

It embeds these summaries as vectors so similar work clusters together.

It then auto-builds surveys, maps topic trends over time, and links clusters across fields.

It also supports semantic retrieval so newcomers get high-quality starting papers.

In expert pairwise tests the system achieves average rank 1.30 and wins across many subdomains.

The trend read shows teleoperation and dexterous manipulation rising while classic reinforcement learning slows.

Researchers get a current map, fast orientation, and concrete entry points.

----

Paper – arxiv. org/abs/2510.20809

Paper Title: "Real Deep Research for AI, Robotics and Beyond"

[echo.hive @hive_echo]

I built a biologically inspired spiking neural network from scratch and it learned with %5 accuracy to do addition :)

There is no backpropagation, no artificial loss functions - just spikes, synapses, and dopamine-like reward signals.

it uses STDP -> "Spike-Timing-Dependent Plasticity" with modulated rewards

This is super fun and I will try to get it to learn with better accuracy. I also need to better understand how all the moving parts fit together

Link to source code in comment which has a detailed readme and html with animations explaining how it all works

[You Jiacheng @YouJiacheng]

this updated my prior

[Jorge Bravo Abad @bravo_abad]

Discovering state-of-the-art reinforcement learning algorithms

Reinforcement learning agents usually learn with rules we program by hand (TD, Q-learning, PPO…). But humans didn’t hand-design our learning rules—evolution did. What if we let machines discover their own RL update rules from experience?

Junhyuk Oh and coauthors present exactly that. They train a population of agents across many environments and use meta-learning to optimize a meta-network that outputs the targets an agent should learn toward—effectively learning the agent’s loss and bootstrapping scheme end-to-end. The agent still emits a policy and predictions, but the semantics of those predictions are discovered rather than hard-coded.

The outcome is striking: a discovered rule (“DiscoRL”) that sets a new bar on long-standing benchmarks. On Atari, a version trained on the 57 games (Disco57) exceeds the performance of hand-engineered algorithms while being more wall-clock efficient. Even more interesting, the same rule generalizes: without being tuned for them, it delivers state-of-the-art results on ProcGen and competitive performance on DMLab, NetHack, Crafter, and Sokoban. Scaling the discovery process to a more diverse set of environments (Disco103) makes the rule stronger still—performance improves simply by exposing it to more varied worlds.

Under the hood, the learned predictions behave differently from classic value functions: they spike before salient events (big rewards, abrupt policy shifts) and are explicitly used to bootstrap and update the policy—showing the system has invented useful intermediate quantities rather than rediscovering old ones. The discovery process is also practical: a few hundred million steps per environment were enough to find a top rule, and the learned rule transfers to larger networks at evaluation time.

This points to a compelling future: instead of manually crafting ever more intricate RL losses and targets, we can train agents whose learning algorithms are themselves learned—improving as we add compute, data diversity, and richer environments. Fewer knobs, more capability.

Paper: https://t.co/1BT1rjc0sg