J.L

pefter

[J.L @Jo1uck]

1/ How can we make Low-Rank Matrices behave more like Full-Rank Matrices—and achieve higher rank efficiency?

We release our new paper: NoRA: Normalized Low-Rank Adaptation.

Paper: https://t.co/hZXe6C5tP5

[alphaXiv @askalphaxiv]

“Normalized Low-Rank Adaptation”

LoRA’s early optimization is dominated by its randomly initialized down-projection, which can produce badly scaled gradients.

This paper introduces NoRA, which simply normalizes that projection along the rank dimension to better match full-finetuning dynamics.

Even normalization only at initialization captures most of the gain, improving convergence, stability, and downstream performance with no extra parameters or inference cost.

https://t.co/30KL1tjPap

[Weiyang Liu @Besteuler]

Great work led by Jiale! NoRA builds on LoRA’s parameterization, with the down-projection matrix normalized. Even more surprisingly, applying this normalization only at initialization is already sufficient to obtain considerable performance gains.

More details in https://t.co/fvGiLWsR8H.

[Zheng Zhan @ZhengZhan13]

Really enjoyed working with Jiale Kang on this.

Well, it’s interesting to see how much room there still is to improve LoRA in 2026.

One thing that stood out to me is that the gap between PEFT and full finetuning is not just about rank or capacity. Initialization and the scale/normalization of the low-rank projection can fundamentally change its optimization dynamics.

Makes me wonder how many other seemingly “small” design choices in PEFT are still hiding in plain sight.

[J.L @Jo1uck]

@Sub_RBala That’s a good question. In principle, with sufficiently long training, the differences introduced by various optimization techniques may eventually diminish. The key question, however, is how to accelerate that process.

[J.L @Jo1uck]

1/ How can we make Low-Rank Matrices behave more like Full-Rank Matrices—and achieve higher rank efficiency?

We release our new paper: NoRA: Normalized Low-Rank Adaptation.

Paper: https://t.co/hZXe6C5tP5

[Hoosha 🧠 @hooshaaii]

Tweet 1:How does one make Low-Rank Adaptation (LoRA) match the optimization dynamics of Full-Rank training? 👩‍💼

We share NoRA—normalizing the projection weights to bridge the gap between efficiency and performance🚀

https://t.co/Z8ExFULwrF

[Daisuke Okanohara / 岡野原 大輔 @hillbig]

NoRAはLoRAのdown-projectionを正規化し、学習速度・安定性・性能を大きく改善する。

LoRAでは、ファインチューニング時に元の重みWに対して低ランクな更新
ΔW = αBA

を加える。ここでAはr×k、Bはd×rの行列であり、rは元の入出力次元よりかなり小さい。

標準的なLoRAでは、Aをランダムに、Bを0に初期化してから学習する。B=0という性質のため、学習初期のLoRAの挙動はほぼAによって決まる。

フルファインチューニングにおける重み勾配を
G = ∂L/∂W

とすると、初期状態では
∂L/∂B = αGA^T

学習率をηとすると、重みの変化幅は
ΔB = -ηαGA^T

また、初期状態ではB=0なので、
∂L/∂A = 0

となり、最初の更新ではAは動かない。

したがって、LoRAによって実際の重みWに生じる変化は

ΔW = αΔBA = -ηα^2GA^T A

となる。

ここで
P = α^2A^T A
と置けば、

ΔW = -ηGP

と書ける。NoRAという手法そのもの以上に、この式によるLoRAの解釈が重要。

もともとフルファインチューニングでは
ΔW = -ηG

なので、LoRAは少なくとも学習初期において、勾配に
P = α^2A^T A

というpreconditionerを掛けた更新とみなせる。

このとき、Aの各列のノルムは、各入力方向に対する実効的な学習率として働き、Aの第j列をa_jとすると、そのスケールは
α^2 ||a_j||^2

となる。
標準LoRAのランダム初期化では、この値が入力方向ごとにばらつくだけでなく、全体として小さくなりやすい。そのため、LoRAの初期gradient normはフルファインチューニングよりかなり小さくなる。
そこでNoRAでは、Aの各列のノルムが1になるように正規化する。
a_j <- a_j / ||a_j||

これにより、入力方向ごとの不要なスケールのばらつきをなくし、LoRAの初期gradient normをフルファインチューニングに近づけることができる。

しかも、この正規化は学習中ずっと続けなくてよく、初期化時に一度だけAを正規化するNoRA-initでも、NoRAによる性能改善の大部分を得られる。

さらに、A���
A = [I_r, I_r, ..., I_r]
このようなブロック単位の単位行列で初期化するBIMI（Block Identity Matrix Initialization）でも、NoRA-initとほぼ同等の性能を達��できる。

これは、初期化分布そのものよりも、Aの各列のノルムを揃えることが重要であることを示唆している。

コメント
===
2層の線形層を重ねて勾配降下法で学習すると、特異値・特異ベクトルに沿った学習ダイナミクスが現れることが知られている（Saxe et al., 2014）。
この場合、今回のNoRAが注目している学習初期だけでなく、学習が進むにつれて大きな特異モードから速く学習されるという現象が現れる。

また、W = BA としたとき、Wに対する更新は概念的には
-BB^T G - GA^T A

という形になり、左右両側からのpreconditioningを暗黙に生み出す。
LoRAでは初期値がB = 0なので、学習開始時には左側のBB^T Gの項が消え、
-GA^T A
という右側からのpreconditioningだけが残る。今回のNoRAは、特にこの初期段階に注目したものと見ることができる。
2つの線形変��を連続して適用する構造はLoRAだけでなく、Transformerを含むニューラルネットワークのさまざまな場所に現れる。その最適化ダイナミクスに対しての基本的な理解が重要となっている。

[J.L @Jo1uck]

Such an honor🥰

[MikaStars★ @MikaStars39]

The best peft paper I've ever read this year.
https://t.co/nSmbrrwjUg

[J.L @Jo1uck]

6/ Huge thanks to my amazing collaborators @ZiyinYue @ZhengZhan13 @YangyiHuang @Besteuler for all the insightful discussions and contributions to this work!

It was a great pleasure working together! 🙌

[J.L @Jo1uck]

/5 Across Pre-training, SFT, and RL, NoRA consistently brings:

• faster convergence
• better performance
• improved training stability
• better rank efficiency
• no extra trainable parameters
• no inference overhead
• fully mergeable weights

[Xidulu @xidulu]

1/ Sharing a new, interesting project I did during my internship at Microsoft AI Frontiers w/ @JohnCLangford

TL;DR: At decoding time, we feed **previous hidden state** into the input together with token embedding, and it boosts performance for free.

https://t.co/za3cZ4UwWV

[Yu Zhang 🐙🌘 @yzhang_cs]

In the early morning, @nathancgy4, @Xinyu2ML, @Yulun_Du and I were preparing some showcases for the blog while watching the World Cup. The moment Argentina beat England, I felt something. I looked up, and saw the most unforgettable Beijing sunrise. So I took the photo.

I knew this was no ordinary day, and now it comes and says, "Hello, world!"

Happy hatch day, K3!

Please enjoy it, I have felt 「眩晕瘫坐」 for weeks.
It is a monster model, yet clean in architecture; elegant in optimization, beautiful in MoE desigins.
It has crossed a certain singularity, it is a year of work, in one release.

🚀🚀🚀 This is a Sputnik moment for OSS.

[Weiyang Liu @Besteuler]

Orthogonal Finetuning (https://t.co/IlBYlgiaae; https://t.co/Mve4Pdptmv) has a unique advantage of preventing catastrophic forgetting. Inspired by this property, we find that merging models within the orthogonal group can effectively reduce model conflicts and preserve both pretraining and downstream knowledge. This is our OrthoMerge framework.

The idea behind OrthoMerge is extremely simple. For OFT-tuned models, we can first map the orthogonal adapters to Lie algebra with inverse Carley transform and then perform merging there. This guarantees the merged model differs from the pretrained model only up to an orthogonal transformation.

A better news is that OrthoMerge can also be applied to non-OFT-tuned models. By solving the orthogonal procrustes problem, we can have the projected component of the adapter onto the orthogonal group. OrthoMerge will then be applied there and the residual component can be merged using conventional merging methods. That said, OrthoMerge can be used together with existing model merging methods!

This is a great example of simple yet effective ideas. Great efforts by my PhD students Sihan Yang and Kexuan Shi. The project is already open-sourced and feel free to give it a try!

Project: https://t.co/Fzjrn0zpaW
Paper: https://t.co/QvFafN1UeY
Code: https://t.co/LjEzcLZ0De

[J.L @Jo1uck]

Effortless win — welcome everyone to try MiSS. Code for MiSS lossless LoRA conversion will be updated soon in PEFT, so you can easily adapt it to frameworks like vLLM and Slime.

[MikaStars★ @MikaStars39]

Accepted by ICML'26, c u @ Seoul ! Thanks for all the coauthors!
#ICML

[Yu Zhang 🐙🌘 @yzhang_cs]

flash-linear-attention is now seeing over 15,000 daily downloads. 📈

We @SonglinYang4 @uniartisan are honored to see fla becoming a piece of the core infrastructure for efficient model archs. Grateful to the community for the trust and support.

https://t.co/VirlvFzgYc

[Hanchi Sun @sun_hanchi]

Introducing Expert Threshold Routing:

- ✅ load balance
- ✅ dynamic computation
- ✅ autoregressive
- ✅ zero train-inference mismatch

At 2.4B params, Expert Threshold achieves 0.067 lower CE loss than Token Choice (equivalent to 1.6× data efficiency).

[kabikabi @jakevin7]

twitter-cli 重大升级!！
逆向了几乎所有的接口, 而不仅仅是之前只有读接口.

现在实现了发帖子、点赞、引用 回复 等基本上所有的写请求！包括搜索等也都实现了！

https://t.co/FefzzXvyq1