Anyone can generate a backtest now.
The hard part is knowing whether to trust it.
I share frameworks to test trading ideas, challenge results, and spot look-ahead bias, overfitting, weak execution assumptions, and weak evidence.
Stocks • Forex • Crypto
One distinction I want to keep explicit in this validation process:
Passing a paper forward test does not mean live execution has been verified.
Paper testing is still useful.
If a strategy already breaks at the paper stage, that can help filter it out early.
But even after it survives paper testing, several things remain unresolved:
real slippage
queue priority
partial fills
latency
actual live execution prices
So I don’t want to collapse everything into one “validated” label.
I’m treating:
Backtest → Holdout → Paper → Live
as separate evidence layers.
Each stage can answer different questions — and leave different ones unresolved.
#Backtesting #AlgoTrading #QuantTrading
One principle I’m finding increasingly important in backtest validation:
If the evidence is missing, the answer doesn’t have to be pass or fail. It can simply be unresolved.
For example:
Could the claimed fill actually have occurred?
Was the slippage assumption realistic?
Was the underlying data complete?
Does the same behavior appear in forward testing?
If the evidence needed to verify a claim is missing, forcing a conclusion can be misleading.
Disclosed is not the same as verified.
In this forward test, I’m going to keep those categories separate:
what was verified,
what was not verified,
and what remains unresolved.
#Backtesting #AlgoTrading #QuantTrading
Another thing I don’t want to overlook in backtest validation:
A successful holdout period does not automatically mean the strategy is robust.
The result may depend heavily on whether that period was:
trending
mean-reverting
high volatility
low volatility
So these are not the same claim:
“It worked on unseen data.”
and
“It survived different market regimes.”
In the forward test, I’ll track not only P&L and execution differences, but also how the results change across market conditions.
#Backtesting #AlgoTrading #QuantTrading
That’s a useful point — especially the distinction between transaction-cost sensitivity and regime sensitivity.
The 5 bps test already showed how thin the apparent edge can become once execution costs are introduced.
I also agree that one holdout period is still only one market regime, so I don’t want to treat a successful holdout as proof of robustness.
I’ll add regime dependence to the list of things to track as the forward sample grows.
One thing became very clear in the 5-minute backtest:
small trading costs can completely change the result.
In the holdout test, some symbols were profitable with zero costs, but turned negative after adding just 5 bps per side.
The shorter the timeframe, the harder it is to ignore:
spread
slippage
fees
actual fill prices
That’s why in the forward test I’m not only tracking P&L.
I’ll also compare the backtest’s assumed fill with the paper fill that actually occurs.
For short-horizon strategies, which execution cost do you pay the most attention to?
#Backtesting #AlgoTrading #QuantTrading
I can see that this post has one reply, but the reply itself isn’t visible to me.
If you replied from a protected/private account, I may not be able to see it.
If you’d still like to share your thoughts, feel free to reply again or send me a DM. I’d be interested to hear your feedback.
I’ve frozen the rules.
I tested a simple 5-minute strategy on AAPL, MSFT, NVDA and AMZN using historical data.
Now I’m stopping the historical analysis.
From here, I’ll follow the same rules forward using only new market data as it arrives.
No parameter changes after seeing the results.
The goal isn’t to prove that the strategy works.
It’s to document whether the backtest assumptions still hold in new data, where they begin to diverge, and what remains unresolved.
I’ll review the evidence after 2, 4 and 8 weeks.
https://t.co/f1X0YdcPJi
#Backtesting #AlgoTrading #QuantTrading
A useful distinction I learned recently:
“The assumption was disclosed” is not the same as “the claim was verified.”
A backtest might state its fill assumptions, fees, or data source.
But can those claims actually be reproduced from the underlying evidence?
And if the evidence is missing, maybe the right answer isn’t “pass” or “fail.”
Maybe it’s simply:
Unresolved.
How do you handle missing evidence when reviewing a backtest?
#Backtesting #AlgoTrading #QuantTrading
I’ve published the English version of Shota Note — What to Check Before You Trust a Backtest.
It’s a free PDF for traders and researchers who want to evaluate backtest results for themselves instead of accepting a polished equity curve at face value.
It covers data quality, execution assumptions, fees and slippage, OOS testing, overfitting, drawdown concentration, forward testing, and the gap between backtests and real market conditions.
Free to read here:
https://t.co/Ao02hebLea
#Backtesting #AlgoTrading #QuantTrading #SystematicTrading
Prima di fidarti di un backtest, cosa controlli per prima cosa?
• Esecuzioni poco realistiche
• Costi / slippage
• Drawdown
• OOS / forward testing
• Sensibilità dei parametri
Quale aspetto non dovrebbe mai essere ignorato?
https://t.co/V3lEF257RH
I’ll be completely transparent about why I’m asking.
I’m building a backtest validation tool that I hope to eventually sell as a real business.
But I don’t want to decide what traders “must be struggling with” based on my own assumptions, build a big product around that, and only then discover that nobody actually needs it.
So I’d rather ask directly.
I’m especially interested in people who already take backtest validation seriously — people who don’t just look at a good equity curve and say “good enough.”
If you already check things like:
• out-of-sample results
• fills and execution assumptions
• fees and slippage
• parameter stability
• drawdown concentration
• forward / paper results
• data quality
• multiple-testing / overfitting risk
what part of that process is still frustrating today?
What do you still have to check manually?
What is repetitive?
What takes the most time?
What makes you think, “I understand why I need to do this, but I wish this part were easier”?
That is the kind of problem I want to build around.
I’m not looking for compliments on the idea. If the real problem is different from what I’m currently building, I’d rather find that out now.
#Backtesting #AlgoTrading #QuantTrading #SystematicTrading #TradingResearch