Nathan Khadaroo-McCheyne

Forecasting & philosophy of stats @OfficialUoM @opendatamcr. #JuliaLang. Weaselling out of things is what separates man from animals...except the weasel.

[Nathan Khadaroo-McCheyne @NathanKhadaroo]

@ChrisHillidge @psymonbee @LauraTrottMP It's interesting that the people condemning this are all MP's and journalists and the people who think it makes sense are all teachers.

[Nathan Khadaroo-McCheyne @NathanKhadaroo]

@llewelyn20 The thing is, the statistics Gamble is referring to are very obviously indefensible to anyone even slightly versed in the subject. And it's worrying so many senior education professionals cannot see it.

[Jack Scannell @JackScannell13]

At last I understand why it costs 100x more to discover a drug today than in 1950. Our computers have slowed down.

[Nathan Khadaroo-McCheyne @NathanKhadaroo]

@jimfoley Sally Rooney.

[Nathan Khadaroo-McCheyne @NathanKhadaroo]

I think this graph actually somewhat undermines the "funny" narrative about measured vs reported height. Some people just like to round to the nearest big figure ¯\_(ツ)_/¯ .

[Hunter📈🌈📊 @StatisticUrban]

Objectively one of the funnier graphs out there. Measured vs. reported male height.

[Kareem Carr, Ph.D. @kareem_carr]

In my experience, economists are some of the most mathematically literate scientists on the planet, and it's very unlikely that some economist hasn't tried out pretty much every kind of major branch of math out there.

[Nathan Khadaroo-McCheyne @NathanKhadaroo]

@thomasforth Long answer is that equivalence is a very murky concept and French dictionaries often disagree on how to apply it (eg from https://t.co/1NRvLOcKZl)

[Nathan Khadaroo-McCheyne @NathanKhadaroo]

@thomasforth Short answer is when you have a noun being used to qualify another noun, it is typically invariable if there is no equivalence between the words (so des talons aiguilles gets an s because the talons are aiguilles, but des stations service, or des films culte do not).

[Jesús Fernández-Villaverde @JesusFerna7026]

In my post yesterday, I described the double descent phenomenon: in many situations, massively over-parametrized models outperform simpler models out of sample. As I emphasized, this is one of the most surprising and intriguing developments in computer science and statistics in recent decades and it has important implications for economics.

The double descent phenomenon raises many questions, most still unanswered or only partially answered, despite intense work by top researchers. Today I will sketch one key element of the emerging picture: the inductive bias of deep learning. This requires some work, so please stay with me.

When a model is heavily over-parametrized (for example, as I mentioned yesterday, with 12,001 parameters for 12 observations), there are many parameter configurations that interpolate the data, i.e., they fit all 12 observations perfectly.

Which parameters are selected in practice? In high dimensions, gradient-based optimizers that minimize fit loss often converge to minimum-norm (min-norm) interpolants under the relevant function class/parameterization. In linear settings (and several over-parameterized regimes/initializations for neural networks), this can be proved; more broadly, practice shows a strong implicit bias toward min-norm solutions.

What does that mean? Think of the fitted model as a function (the curve that goes exactly through all 12 points). The optimizer effectively selects the curve that is “smoothest” in a well-defined sense: the curve that minimizes a functional seminorm (e.g., a Sobolev seminorm).

What is a functional seminorm? A basic mathematical task is to measure the “size” of an object (like the length of a vector).

A norm is just a concept of “size” that is useful for the task we are dealing with. That is why in high-school math we introduce ideas like the Euclidean norm of a vector: it gives us a very intuitive and useful way to think about the “size” of a vector.

Norms can be too strict for many problems; sometimes we want a size that treats certain nonzero objects as “equivalent to zero.” A seminorm does exactly that: it measures size while allowing some nonzero objects to have size zero.

A functional seminorm applies this idea to functions, giving a way to quantify the size of the whole function, not just its value at a point (though pointwise evaluation can itself be used as a seminorm).

Enter Sobolev seminorms (and their relatives): here, the “aspect” of the function we measure is smoothness—how large its derivatives are. A Sobolev seminorm ignores baseline level and measures only the magnitude of derivatives; it is a seminorm because adding a constant (or, for higher orders, a lower-degree polynomial) does not change it.

A metaphor: imagine rating the difficulty of a bike trail for your weekend excursion. You want to assess the entire trail, not a single point. The Sobolev seminorm “measures” the curvature, wiggles, and twists of the trail, but it does not care whether the trail sits at 100 or 500 meters above sea level.

There are many reasons Sobolev seminorms matter across mathematics. For example, if you’re solving the heat equation for a rod, you care about how steep the temperature gradient is along the rod, not how hot the rod is.

But the key point for us is that they give an intuitive measure of how smooth a curve is.

Now return to the figure from my post yesterday:

https://t.co/wYEbPWBS5v

Of all neural networks with 12,001 parameters that fit the 12 observations perfectly, a gradient-based optimizer tends to pick the smoothest one (in the sense above). That is the inductive bias at work, and it’s remarkable.

The formal statement is in the snapshot from my paper:

https://t.co/AWn6ty6ZPU

included in this post.

We can prove this behavior in a number of cases; in practice, it appears beyond those instances as well. This is why yesterday’s example was not cherry-picked.

Why do we care about “smooth” curves?

1️⃣ Occam’s razor. Smooth curves are typically the simplest, delivering an inductive bias toward simplicity.

2️⃣ Dynamic economic models. The smooth solutions are precisely those that satisfy the transversality condition. I have a full paper on this:

https://t.co/NT9XRMJ4nc
I will explain more another day.

3️⃣ Forecasting. In practice, smoother curves generalize better and forecast new observations more accurately.

Finally, a quick diagnostic from the figure: the ℓ₂ distance between the true function and the interpolated solution is much smaller in panel 4 than in panel 3. That is, the massively over-parametrized solution (panel 4) outperforms the merely overfit solution (panel 3) precisely because the massively over-parametrized solution minimizes the seminorm.

I realize this post likely opens more questions than it answers (and I had to be a bit sloppy with some details), but bear with me—I’ll try to address those next week.

[Nathan Khadaroo-McCheyne @NathanKhadaroo]

The very predictable, widely predicted consequences of passing laws that effectively stopped new hotels from being built.

[Radical Graffiti @GraffitiRadical]

"Your AirBnB was my home"
Seen in Barcelona

[Nathan Khadaroo-McCheyne @NathanKhadaroo]

Contra most comments and quotes on this post, the issue with the paper is not the poor R^2 (which is to be expected in studies with outcome variables like neurodegeneration, which have many subtle causes) but the lack of causal identification mechanism.

[Dr. Dominic Ng @DrDominicNg]

Sitting for hours daily shrinks your brain, even if you exercise.

New 7-year study (n=404):
→ Faster hippocampal atrophy
→ Worse memory performance
→ Slower processing speed

87% met exercise guidelines but STILL declined.

[Nathan Khadaroo-McCheyne @NathanKhadaroo]

@adb0wen @tomhfh Quite maddening to see so many "It's a rare thing to see a journalist who actually understands statistics" type responses.

[Nathan Khadaroo-McCheyne @NathanKhadaroo]

@gavinantonyrice Am I missing something? Why can't we pay for manufactured goods using services (which we are much more competitive at)?

[Nathan Khadaroo-McCheyne @NathanKhadaroo]

@IsaacKing314 I voted prison. Being permanently exiled from the place I grew up, considered my home, and where all my friends and family live would be really terrible! (I'm assuming this is about the Columbia student.)

[Nathan Khadaroo-McCheyne @NathanKhadaroo]

‘Social media is making us all dumb.’ Look, maybe it is, but you cannot get that from an uncontrolled comparison of test scores between cohorts.

[Nathan Khadaroo-McCheyne @NathanKhadaroo]

My timeline is full of people drawing ridiculously strong (and, worse, ridiculously confident) conclusions from this that the data simply does not support.

[John Burn-Murdoch @jburnmurdoch]

NEW 🧵: Is human intelligence starting to decline?

Recent results from major international tests show that the average person’s capacity to process information, use reasoning and solve novel problems has been falling since around the mid 2010s.

What should we make of this?

[Anya Martin @AnyaM8_]

The reaction to some lady getting a literature PhD is underlining the difference between "people who have reasonable critiques of academia" and "people who think they are the former but are too stupid to make those critiques & instead shriek impotently at the concept of research"

[Nathan Khadaroo-McCheyne @NathanKhadaroo]

The main reason I don't trust the government is that nothing ever gets done, in large part because of the endless reviews and consultations which this report recommends we have even more of!

[Miriam Levin @MiriamLevin1]

Excited to publish @Demos Citizens' White Paper today, with @involveUK. It sets out why, when and how the govt should embed public participation in national policy making if it wants to win back trust and make policies that work for people. https://t.co/ORLvTMmYrv
🧵👇

[Nathan Khadaroo-McCheyne @NathanKhadaroo]

@realhansard I think they can, it's more likely that Australia is absent from the dataset used to create this 'heatmap' and so we're seeing extrapolation from the closest data point by an algorithm that assumes that closer = more alike which (genetically) isn't true for Australia.

[Saloni @salonium]

One of the most concerning stories this year.

PEPFAR – an incredibly successful US policy that's made AIDS treatment affordable in Africa, saving 20+ million lives since 2003 – is at risk of not being reauthorised.

It's unimaginable to let this happen.
https://t.co/i4ZJq7HB29