@Matiuee اگه چیزی میزان سختیش خیلی زیاد به نظر میاد، معمولا معنیش اینه هنوز کامل بکگراند مورد نیاز (مثلا کورس های پیشنیاز) رو نداری. با مطالعه بیشتر بهتر میشه.
The translation of my book Linear Algebra Done Right into Farsi is now freely available at https://t.co/ii1ovMFKvH (click on the fourth link on that webpage to get the Farsi version).
This spring I taught a grad course on “All of Convex Quadratic Optimization.”
That may sound narrow, but quadratic problems are simple enough to analyze precisely, yet rich enough to reveal many of the core phenomena that drive modern optimization and learning.
I wrote up the course notes here:
https://t.co/kuLZi4rFRG
@ehsanasgharzde اگر تو زمینه مشخصی ریسرچ نمیکنی، به نظرم مطالعه کورس های advanced دانشگاه های خوب (ایرانی یا خارجی) فایده بیشتری از خوندن مقاله رندوم داره.
A linear autoencoder learns PCA. This has been understood since Baldi–Hornik ’89.
But what happens if you make the model nonlinear by adding just a ReLU in the middle?
Surprisingly, the training dynamics of even this minimal nonlinear variant remain poorly understood.
In our paper, we take a stab at this by characterizing how a one-hidden-layer ReLU network, trained from random initialization while updating both layers, recovers a linear model.
https://t.co/b5D6M87xR9
No fixed features. No preprocessing tricks. No “train only the last layer” gimmicks.
Just standard gradient descent on the empirical loss, with both layers trained end-to-end.
At first glance this may sound almost comically simple: “Congrats, you proved neural nets can learn linear functions?!”
But the dynamics turn out to be surprisingly subtle, and the proof techniques ended up being some of my favorite ones we’ve developed in the last few years.
Quick thread below on why this problem is challenging 🧵
I also genuinely think the techniques here may be useful for understanding more contemporary models and training phenomena, including reasoning and post-training. Stay tuned!
Challenge for ML theory folks:
What’s the cleanest solvable data model where nonlinear autoencoders provably beat PCA?
We suggest one here: https://t.co/HVpiXLCCIG
Happy to collect pointers & ideas!
1. Ridge regression is heavily used in systematic investing both in the p << n and in the p >> n cases (last one, less so). I don't think that the use is very deeply motivated, other than the old standard argument in favor (see, e.g., El. Stat. Learning).