三鞭兄想做吃货又想减肥_AI

[三鞭兄想做吃货又想减肥_AI @xcgoner1108]

@Orange41324306 True for original muon, but not quite for muon family. You could modify the abc factors or add a diagonal eps to the preconditioner in order to filter out the small eigenvalues which are typically considered noise, which would make muon work for noisy gradients such as RL.

[三鞭兄想做吃货又想减肥_AI @xcgoner1108]

@tengyuma @HongLiu9903 @zhiyuanli_ @dlwh @percyliang @StanfordAILab @stanfordnlp @StanfordCRFM @Stanford The comparison seems unfair for Adam. I wonder what will happen if you use Adam as the hessian estimator in Sophia (basically AdamW + lazy k-step update of 2nd moment + clipping of 1st moment / 2nd moment). I suggest use the same bag of tricks of sophia on AdamW

[Bert Huang 🐥 @berty38]

NIPS reviewers: IMO, the only acceptable scenario where you don't update your reviews after the author responses is if the authors did not upload a response. If they addressed your review, they deserve some feedback.

[三鞭兄想做吃货又想减肥_AI @xcgoner1108]

@tqchenml Awesome! In that case it will be really convenient to add figs and equations

[三鞭兄想做吃货又想减肥_AI @xcgoner1108]

@tqchenml The sad fact is, simple methods are usually ranked as having “minimal contribution” and thus get rejected.