We trained our scientific writing LLM against human writing until it was indistinguishable by human experts. Thanks to @julianharris, we saw that during training the model started looking human to pangram 4.1 too! Very cool to see this emerge
@ProfJAParker@p_ganong For information spread, coeff drop is only at the original sample end but not at the paper circulation. But some learning certainly happens. The second idea is good, but need to think how to implement it. Thank you!
How much of published finance holds up?
I attempted to replicate the main result of every paper in JF, JFE & RFS, 2000β2020.
1,328 in-sample replications; 1,005 out-of-sample. 75% reproduced in sample. But only 42% in later data. Median coeff. : 0.93 β 0.45 of the published.
@jenniferdoleac In finance, most papers reproduce well in-sample from just paper text. But most results decay by half out-of-sample. So coding errors are not the main problem. https://t.co/qApKtJ8TgQ
Published finance mostly replicates in-sample but suffers from out-of-sample decay. The drop is a cliff at the sample end, which is consistent with data snooping.
The stronger the original t-stat, the greater the fraction of the published result is preserved OOS. Behavioral mechanism, publication year, top-5 school and star author don't add additional measurable value.
Does Empirical Finance Replicate? (Muravyev)
On the heels of @OpenAI publishing solutions to hundreds of open problems in mathematics, we are thrilled to announce AGORA, the Agentic Global Open Research Arena, at @SmealCollege:
https://t.co/fWKUmcHpsk
I've been obsessively improving my AI research workflow (instead of doing actual research). I'm sharing the result: EconYoloBox, a container setup for running YOLO AI agents. A short thread about the tool and my workflow:
Putting AI to work! @dmuravyev replicated every empirical paper in the JF, JFE, and RFS from 2000 to 2020, then did out of sample testing.
You can access the tests, replications in a nifty website he has set up:
https://t.co/EdrCYOId84
This is so cool!
https://t.co/jda4igH0PB
After more testing: GPT-6 Pro is the best model ever for Math/ML/Brainstorming. It is less clear cut this time around, though: they must be running agents at High effort (maybe xHigh), which makes Pro reliable, but sometimes loosing to Astra Max. For toughest problems my workflow is to run Pro, 5.1 and Max separately, then paste into Max for the final aggregation.
New episode with @polynoamial
We talk about multi-agent, Navier-Stokes, and what the current explosion of maths progress tells us about what happens once you automate AI research.
And we also discuss how we will know if the models are actually aligned before we kick off RSI.
0:00:00 β Multi-agent and Navier-Stokes
0:15:28 β How will AI firms work?
0:22:02 β What math progress tells us about recursive self improvement
0:40:22 β Hugging Face and alignment
1:01:18 β The internal/external model gap
1:08:34 β Chain of thought is degrading
1:14:12 β How will we know when alignment is solved?
It always bugged me how we use present tense in finance papers. Shouldn't we say "the coefficient *was* economically and statistically significant?", instead of *is*?
Or, we can stick our necks out and say "*was* significant, and we predict it will continue to be so".
@joachim_voth Yes, it's surprisingly hard. I wouldn't claim doing prefect job for every paper. 1. Tables 2 or 3, rarely T5+. 2. Independent audit 3. New models (Astra/Fable 5.1) are much better at this.
Only 42% of empirical finance papers replicate out of sample.
One paper that *does* replicate out of sample is "Anticompetitive Effects of Common Ownership", Journal of Finance, 2018.
@JeanClaudeFox2 Say original paper found 15% long-short return per year and original sample Jul1980-Jul2005. My replication computes LS for each year, say 3% in 2004 and 5% in 2006. Coeff ratio is 3/15 in 2004 and 5/15 in 2006. Then aggregate across papers on each year relative to sample end.
@AshBelur@paulnovosad ~30 minutes per initial replication, and 1-2 hours in valuation checks. 2. For trading strategies - - yes, although discontinuity at sample end suggests some data snooping. But Finance is much broader than just trading strategies. Corporate finance results decay as well.