I started a small Discord for people who build and test systematic strategies
No signals, no courses, nothing for sale. We share back tests, code and ideas and try to break them before the market does
Capped at 100 members and it stays active - quiet accounts get swapped for people on the waitlist
To join answer 3 short questions about what you're building:
https://t.co/S5keAahW2I
Autopsy #2. I ran Thursday's dumb version test on a popular 10 min EMA ribbon reversal setup from YouTube
ES and NQ futures 2021-2026, 870 trades, -0.044R a trade after costs. Against random entry days it sits in the middle of the range, nowhere near top 5%
Then I did what everyone does and searched 1944 variants. Best one made +0.100R a trade on 2021-23. On 2024-26 0 of 10 checks were positive
Same thing as best of 100 MA crossovers on noise, just on real data this time
Which setup should I run through it next?
Validation Lab #1: Beat the dumb version
I've killed way more strategies than I kept, so I'm starting a small weekly series on the checks I run before any bot gets near real money. First one: 100 MA crossovers on pure noise, best one made +56% in sample and -2.7% on data it never saw
The idea is simple. Take your strategy and build a dumb copy of it. Everything stays the same, except the dumb copy enters on random days. If your strategy can't beat that, the profit in your back test isn't coming from your entry
Why does this matter? A back test can make money for three reasons, and the equity curve looks the same for all of them. You might have a real edge. The market might have gone up the whole time while you were long, which on indexes is most of the time. Or you tried 100 settings and kept the lucky one. Looking at the curve alone you can't tell which one you have
To show how it works, I rebuilt a test I posted on Discord a while ago, this time with fixed seeds so anyone can get the same numbers. It's all synthetic data - random walks I generated myself, no costs. I like fake data for this because I know the answer before I start: there is no edge in it
Part 1: pure noise
I made a random walk of about 10 years of daily bars, with no drift at all. Then I ran 100 moving average crossovers on it and kept the first 7 years as in sample and the last 3 aside
The best one was 50/180, with +56% and a Sharpe of 0.48 in sample. If someone posted that, I'd probably take a second look
On the 3 years it had never seen, it made -2.7%
Maybe that was just one weird world, so I repeated the whole thing on 60 different random walks. On average the winner had a Sharpe of 0.50 in sample and -0.01 out of sample. In 30 of the 60 worlds it went negative, basically a coin flip
There was nothing to find, and the scan still found a "winner" every single time
Part 2: a market that goes up
Pure noise is easy to catch with a simple in sample / out of sample split. The trickier case is a market that rises anyway, like an index
So the second world drifts up slowly. The rule is plain dip buying: buy after 3 down days, hold for 5. Result: 143 trades, PF 1.34, 52% winners, +0.23% per trade. I'd probably keep that one if I didn't check further
Then I built the dumb version - same market, same 143 trades, same 5 day hold, only the entry days are random - and ran it 5000 times
91% of those random runs made money too lol. The median random run made +0.21% per trade against +0.23% for the rule, so the rule beats only 56% of them. To make the top 5% it would need around +0.45%, about double
So the PF 1.34 is just the market going up. Any entry held for 5 days picks up the same drift
How to run it on your own strategy
Keep the market, period, number of trades, long/short split, hold, exits and costs exactly the same. Only the entry goes random - random days, or a random time of day if it's intraday. If your real trades can't overlap, don't let the random ones overlap either. Then run it 1000+ times and count how many runs beat you
If fewer than 5% of the random runs beat you, that's interesting. If you're in the middle of the pack, you haven't shown anything yet
One thing that's easy to miss: if you picked your settings out of 100 tries, each random run should also get the best of 100 random tries. Otherwise you're comparing your best of 100 with one average monkey, and of course you win haha
What passing doesn't mean
Passing doesn't mean you found an edge. Even if you pick the best variant in sample and it holds up out of sample, it can still be overfit by chance
Same here. The top 5% happens to random rules 1 time in 20. I ran the dip rule on 60 drifting worlds and it made the top 5% in 1 of them, when pure chance gives about 3. So if you test 20 ideas, one will probably pass on luck alone
That's why this is the first check and not the last. The next ones cover costs, a holdout, nearby parameters and other markets
Next week: the realism ladder - what happens to the same rule when you go from cheap bars to real ticks and real costs
Script on request if anyone wants to play with it. It's all synthetic, so nothing secret in there
Do you run random entries against your strategies, or do you mostly look at the equity curve?
The setup is @satymahajan, rules exactly from his video, checked on his own example day to the cent before any P&L. I also tried 244 filters (VIX, news days, weekday and more) and none did better than a random filter of the same size
https://t.co/kx8Ma3lczz