Pranav

researcher @kubocreate

[Pranav @Noro1729]

At @kubocreate, we were working on learnable novelty and epiplexity using spectral objectives and we thought, why not apply this in an optimiser? We ran some experiments and then realized that it’s basically Muon. I went down the rabbit hole anyway. Read below:

[Pranav @Noro1729]

@sketchbender @mishandmash2 why would you have a staunch anti-ai stance in the first place lol

[Pranav @Noro1729]

@andrewgwils Happy diwali! 🪔

[Pranav @Noro1729]

@aarnav2006 Got an average of 1.66, looking to flip some scores 🙄

[Pranav @Noro1729]

@Keyingkarras @HrrsMjd Yeah, probably

[Pranav @Noro1729]

@sumukhbett @iclr_conf @iclr_conf wyd 🍆💦

[sumukh @sumukhbett]

@iclr_conf u up?

[Pranav @Noro1729]

@eashwarpanchal @peeleraja It's satire

[Nassim Nicholas Taleb @nntaleb]

Another way the current administration is trapped in the ZERO-SUM FALLACY: The H-1B visa situation.
+ You are NOT taking a job from a citizen, rather creating jobs by increasing overal economic output. Plus these people (even vegetarians) SPEND locally.
+ The US has long been benefiting from other countries' brain drain; without it its edge wd collapse. Tesla, Von Braun, von Neumann, Fermi & Einstein did NOT each take the job of a citizen.
+ It the problem is anti-Indian racism, say it, don't play games.

[Nassim Nicholas Taleb @nntaleb]

Capitalism works if and only if all members see themselves as "pre-rich" or "feasibly-rich-if-they-wanted-it-badly-enough". Remove that and the system collapses.

[Pranav @Noro1729]

@kubocreate @kellerjordan0

[Pranav @Noro1729]

At @kubocreate, we were working on learnable novelty and epiplexity using spectral objectives and we thought, why not apply this in an optimiser? We ran some experiments and then realized that it’s basically Muon. I went down the rabbit hole anyway. Read below:

[Pranav @Noro1729]

@sumukhbett @ajan_iyer Read the full explainer here:

https://t.co/26EKE7sunv

[Pranav @Noro1729]

w/ @sumukhbett @ajan_iyer

[Pranav @Noro1729]

We tested a 4.84M-parameter transformer on Tiny Shakespeare for 1,200 steps, with five fresh paired seeds per study.
Mean validation loss versus each study’s Muon baseline:
• Nyström + diagonal: 1.63% lower
• Mean + diagonal: 0.99% lower
Both won all five paired comparisons.

[Pranav @Noro1729]

@sumukhbett @ajan_iyer Read the full explainer here:

https://t.co/26EKE7sunv

[Pranav @Noro1729]

w/ @sumukhbett @ajan_iyer

[Pranav @Noro1729]

Why apply Muon afterward?
Preconditioning changes the matrix’s directions and scales. Muon then approximately equalizes its nonzero singular values.
M → input conditioning → approximate polar factor → weight update

[Pranav @Noro1729]

Our second approach is even simpler: mean + diagonal.
Ĉ = μμᵀ + diag(v)
μ contains mean activations; v contains their variances.
This preserves correlations explained by the mean while approximating centered covariance as diagonal. Its inverse needs only a rank-one correction.

[Pranav @Noro1729]

The challenge is estimating C cheaply.
Our first approach: Nyström + diagonal.
A rank-64 sketch captures part of the correlation structure. A residual diagonal restores each feature’s second moment. We apply its damped inverse without building a dense matrix.

[Pranav @Noro1729]

We call it MUON-IC, since it's MUON but Input Conditioned. Building on Newton–Muon, we use that input geometry before orthogonalization:
P = M A⁻¹
M is momentum. A is a normalized, damped approximation of C = E[aaᵀ].
Then apply Muon’s usual Newton–Schulz iterations to P

[Pranav @Noro1729]

At @kubocreate, we were working on learnable novelty and epiplexity using spectral objectives and we thought, why not apply this in an optimiser? We ran some experiments and then realized that it’s basically Muon. I went down the rabbit hole anyway. Read below: