formalproo_f

formalproo_f Math COVID climate statistics run maths machine learning AI programming quantifiable economics neuronal Network bank

[formalproo_f @formalproo_f]

remote switching f35 on tarmac. @Ginger149 @charles_consult us prob not

[formalproo_f @formalproo_f]

underestimation risk to life hostages @Ginger149 @charles_consult

[formalproo_f @formalproo_f]

f35 genocid @Ginger149
@charles_consult

[formalproo_f @formalproo_f]

h @Ginger149 @charles_consult

[Jeff Masters @DrJeffMasters]

Helene’s Cat 4 landfall gives the U.S. a record eight Cat 4 or Cat 5 Atlantic hurricane landfalls in the past eight years (2017-2024), seven of them being continental U.S. landfalls. That’s as many Cat 4 and 5 landfalls as occurred in the prior 57 years.

[formalproo_f @formalproo_f]

a @charles_consult @Ginger149

[Tom Yeh @ProfTomYeh]

Mamba by hand ✍️ ~ I made this exercise as I studied the math in the Mamba paper. It is a small yet complete Mamba implementation. Thanks @_albertgu for answering my questions and checking. 👇Join the 'AI Math' community. Download xlsx.

[formalproo_f @formalproo_f]

l @Ginger149 @charles_consult

[Alex Bilzerian @alexbilz]

Lectures by Stephen Boyd at Stanford University.

Introduction to applied linear algebra with emphasis on applications: https://t.co/oVjyyBRriy

h/t @dr_sergiomedina

[formalproo_f @formalproo_f]

o @Ginger149 @charles_consult

[Konstantin Mishchenko @konstmish]

A student reached out asking for advice on research directions in optimization, so I wrote a long response with pointers to interesting papers. I thought it'd be worth sharing it here too:

1. Adaptive optimization.
There has been a lot going on in the last year, below are some papers I personally found interesting.
First of all, this paper by Li and Lan on Nesterov's acceleration of adaptive gradient descent:
https://t.co/D6hykeK2tw
Check Corollary 1 for a simple description of their method. There is one thing I don't like about it: the amount by which we can increase the stepsize at each iteration decreases as t grows. That being said, I don't know if this restriction can be lifted, and perhaps it's the best thing we can get.

Yura Malitsky and I also did some work on adaptive gradient descent, making the stepsizes a bit larger, roughly sqrt(2) improvement over our previous result:
https://t.co/exhFgbjChk
We still don't know if that's the best we can do or if a tighter analysis can give us better methods.

I should also mention that there is more push in the literature on Polyak stepsize, see for instance these two papers:
https://t.co/8tKRReEpx2 (a stepsize very similar to Polyak)
https://t.co/kZhWGqI1sE (Polyak stepsize with momentum)

2. Adagrad-like methods still can be studied, I believe it's an underexplored direction. I wish there was more papers on studying the importance of coordinate-wise stepsizes. One paper on the topic I really liked is this study of when Adam is more useful than SGD:
https://t.co/sF5Abi08h5
There is also some research on new practical methods, for instance, acceleration of DoG is interesting:
https://t.co/VMOdfbL95Z
And I also enjoyed reading this paper by Rodomanov et al. on line-search-inspired stochastic methods:
https://t.co/85uLHGZErQ

3. I also like the direction of getting better assumptions for optimization theory and studying the implications. A good example is the gradient clipping literature:
https://t.co/NMTXzFJScs ((L₀, L₁)-smoothness)
https://t.co/dG8xIoTFPN (same revisited)
https://t.co/goKclD80WG (on heavy-tailed noise)
We need to bridge optimization assumptions with what we know about neural networks, so read about properties of neural networks themselves like this:
https://t.co/6M8l1avBOJ (on scales of layers and how their type affects Lipschitz constants)

4. These days, people are using deep networks of all scales for their tasks, and they have discovered a lot of tricks that haven't been studied thoroughly in optimization literature: quantization, Straight-Through Estimator, (https://t.co/7UK2gsojhm), low-rank techniques such as LoRA, learning-rate warm-up, etc. You should expose yourself to those tricks to get a better understanding of what the current theory is lacking.

If you're considering choosing optimization as the topic for your PhD, here are some extra thoughts. Right now there is less activity than about 5 years ago, most low-hanging fruits seem to have been taken, and the remaining questions seem quite challenging. So if you're looking for a field where it is easy to get publications, it might not be perfect. However, it's still a good field to produce meaningful theory. It's also important who you would work with, i.e. if you can find a good advisor, that often affects one's satisfaction to a larger degree than the topic itself, so make your decision carefully.

As my last word of advice, I definitely encourage testing new methods on neural networks (and preferably not on CIFAR10/CIFAR100, because they give misleading results), at least something like nanoGPT (https://t.co/NTk9KAAqd4). When I was a PhD student, I did a lot of theoretical research testing my methods on logistic regression and that was useful to understand the theory, but I also had the wrong impression about what works and what doesn't because of that. If you can, do both, understand the theory as much as you can, but also learn its limits and failure modes.

[formalproo_f @formalproo_f]

weak Al Jazeera weak idf ramallah bureau raid @Ginger149 @charles_consult

[formalproo_f @formalproo_f]

bloc @Ginger149 @charles_consult

[Al Jazeera English @AJEnglish]

"International law is clear: The supply of weapons to a state engaged in war crimes, crimes against humanity and a plausible genocide - is itself a crime."

— writes @shahdhm for #AJOpinion ⤵️ https://t.co/20blv8lR1Y

[formalproo_f @formalproo_f]

battery no way @Ginger149 @charles_consult

[Arr David 🐦 @da_masse]

@Laurence_in_EU The cctv shows them answering a messaging ringtone. Ensuring hands on and then blown off hand and main body injuries. An explosives expert put it at around 20 grammes. Not much, but with military grade explosives, major trauma. Also, this explosive didn't have scent markers.

[formalproo_f @formalproo_f]

str @Ginger149 @charles_consult

[Peter Dynes @PGDynes]

Parts of #Europe have seen more rainfall in just 3 days than the entire autumn period—this is anything but normal. This is the new climate, with 10% more water vapour in the atmosphere and a jet stream that's increasingly erratic. 🌍🌧️ #ClimateCrisis #ExtremeWeather #Flooding

[formalproo_f @formalproo_f]

weak no friends @Ginger149 @charles_consult houthi thing to

[Dyab Abou Jahjah @Aboujahjah]

This is significant not just from a legal standpoint, but also in terms of the prevailing rules of engagement. Up to this point, both Hezbollah and Israel have maintained that their targets were limited to military personnel on the opposing side. Civilian casualties were typically framed as accidental or as collateral damage. However, with Israel now clearly targeting both Hezbollah’s military and civilian personnel indiscriminately, the nature of the conflict has fundamentally shifted.

This implies that not only IDF soldiers but anyone associated with the Israeli state—civil servants, various employees, and politicians—could now become legitimate targets. The escalation, in this case, would be total. Hezbollah will likely absorb the initial strike, but retaliation seems inevitable. There's also the possibility that Israel might launch an all-out offensive soon, capitalizing on the element of surprise.

[formalproo_f @formalproo_f]

p @Ginger149 @charles_consult

[formalproo_f @formalproo_f]

questions @Ginger149 @charles_consult

[Jörg @kachelmann anderswo: @realkachelmann @Kachelmann]

Wer hat was mit der Erdachse gemacht?

[formalproo_f @formalproo_f]

Software @Ginger149 @charles_consult

[★★★ Patrick ᓚᘏᗢ ★★★ @heroniconic]

Softwarefehler:

Der Fehler bei der Berechnung der Mandate auf Grundlage des vorläufigen Wahlergebnisses sei, "dass ab der Zuteilung des 117. Sitzes die Sitze nicht mehr an den mathematisch höchsten Teiler zugewiesen wurden".

https://t.co/lamhBWG6mf

#Sperrminorität
#Sachsen

[formalproo_f @formalproo_f]

k @Ginger149 @charles_consult

[Democratic Wins Media @DemocraticWins]

BREAKING: Unlike Donald Trump or JD Vance, Kamala Harris and Tim Walz are capable of walking into a restaurant and relating to their peers. Look at the enthusiastic reaction Kamala Harris received today in rural Georgia. Retweet so all Americans see this

[formalproo_f @formalproo_f]

talk to manager @Ginger149 @charles_consult

[Barack Obama @BarackObama]

How it started. How it's going. 

Ten years later, and it's still a good look! https://t.co/NKXRGNgJPv

[formalproo_f @formalproo_f]

trust pandemic @Ginger149 @charles_consult

[Ralf Wittenbrink @RWittenbrink]

Wie Blutgerinnsel bei #COVID19 Gehirn und Körper schädigen

Eine relevante Studie, die unser Wissen über #COVID19 und die Symptome neu formt: Das Blutgerinnungsprotein Fibrin verursacht ungewöhnliche Gerinnung und Entzündung, und unterdrückt gleichzeitig …

#LongCovid #Corona

[formalproo_f @formalproo_f]

Tim @Ginger149 @charles_consult

[formalproo_f @formalproo_f]

m @Ginger149 @charles_consult