OptiNod Academy

Overfitting: Only Settings With a Plateau Around the Peak Survive Live Trading

The parameter set with the best backtest return is often just a curve fitted to historical noise. In live trading, only plateau-shaped performance curves, where nearby settings also perform well, tend to hold up.

> In live trading, the only backtest peaks that hold up are plateau-shaped ones where the *nearby settings also perform well*.

Overfitting means shaping a strategy's rules and parameters around random movements in historical data. If you run enough combinations on the same data, almost any strategy can produce an impressive backtest curve. The core problem is that backtest returns alone cannot tell you whether that curve captured a real market pattern or simply memorized noise that happened to fit that period.

Most traders use optimization like this: they set a wide parameter range, run hundreds of backtests, then trade the single highest-returning combination live. They compare whether RSI period 14 is better than 13, or whether a stop should be 2% or 2.3%, down to decimal places in backtest return. The selected optimum really was the best-performing setting in that historical window. The problem is that this peak is often the first place to fail in live trading.

The point is simple. In optimization, what matters most is the shape around the single highest point. If performance falls off a cliff when RSI period moves one step from 14 to 13 or 15, then 14 is likely a coincidence fitted to a specific historical fluctuation. If performance remains similarly strong from 12 through 18, that range is a robust area capturing a real market structure. Even with the same backtest return, whether you looked at the neighboring settings can determine live performance.

A Peak Is Dangerous When It Stands at the Edge of a Cliff

Assume two parameter sets both produced a 200% backtest return. One is a sharp peak where the neighboring values drop to 50%. The other sits in the middle of a gentle plateau where the neighboring values still hold 180%. Standard optimization treats them as equivalent because both show 200%. In live trading, their results are likely to diverge completely.

The reason is that future data is never exactly the same as past data. Even if RSI period 14 was the peak in the backtest, live prices will not move exactly like the backtest window. If volatility rises slightly or trend speed changes, the historical optimum may shift one step to the side in the future. If you were sitting on a cliff-like peak, that one-step shift can turn a 200% return into 50%. If you were in the middle of a plateau, the same shift still leaves you inside the strong-performance zone.

The numbers are illustrative, but this is how the difference between a cliff and a plateau shows up in practice. In 2021, BTC started the year around $29,000 and rose to roughly $69,000 on November 10. If you tighten the stop-loss distance and trend-following parameters until they fit that strong uptrend perfectly, the backtest for that period can look spectacular. But in 2022, the market fell into the $17,000 range in June during the Luna and 3AC liquidation cascade, then made a low near $15,500 in November after the FTX collapse. In a regime where both trend direction and volatility changed completely, cliff-like settings tuned to the 2021 peak would start accumulating losses within the first few weeks. The peak was only a peak under that specific market state.

Do not choose settings based only on how high the peak is. Check how flat the area around it is.

More Degrees of Freedom Mean More Memorized Noise

Every time you add another parameter or rule to a strategy, you increase its degrees of freedom. Degrees of freedom are the number of knobs the strategy can use to fit itself to historical data. The more knobs it has, the more precisely it can be tightened around almost any historical curve. That also means it can memorize both real patterns and random noise without knowing the difference.

The principle is the same as basic statistics. Two points can be connected exactly with a straight line. Three points can be connected exactly with a quadratic curve. Add enough parameters and you can draw a curve that passes through every historical trade, but that curve may explain nothing about the next new point. Adding stop-loss rules, take-profit rules, entry filters, and time-of-day conditions to a strategy is like increasing the degree of the curve. More rules generally improve in-sample performance, but within the in-sample data itself, there is no way to know which part is a real pattern and which part is memorized noise.

If a five-rule strategy and a fifteen-rule strategy produce the same return over the same historical period, choose the five-rule strategy. Producing the same result with fewer degrees of freedom means it relied less on coincidence. A performance curve built from fifteen rules is likely to contain a lot of extra machinery fitted to historical noise. This is why simple strategies tend to survive longer in live trading.

Do not ask how much in-sample return improves with each added rule. First ask why that rule must exist.

With a Small Sample, Even a Few Knobs Can Overfit

You cannot judge overfitting by the absolute number of degrees of freedom alone. You also have to compare the number of parameters with the number of trades. The same five parameters may be acceptable in a strategy with 1,000 trades, but in a strategy with only 30 trades, tuning just five can quickly lead to overfitting. The smaller the sample, the larger the role of random variation in total performance, and the easier it is to mistake that randomness for a real signal.

Strategies with few trades are statistically unreliable. It is like flipping a coin ten times, getting heads seven times, and concluding that the coin is biased toward heads. If 30 trades show a 60% win rate, the confidence interval around that 60% is roughly wide enough to run from the low 40% range to the high 70% range. Within such a wide range, a small parameter change can move the win rate sharply. If you choose the optimum by chasing that movement, the next 30 trades are unlikely to reproduce it.

If you backtest one year of BTC daily candles, you only have about 365 bars. For a swing strategy with an average holding period of two weeks, that may mean only around 20 trades in a year. Optimizing five parameters at once on that sample is effectively fitting one knob for every four trades, so it is no surprise when a setting appears to match the past perfectly by chance. To increase the sample, use a longer historical period or apply the same strategy across multiple assets and combine the trade count. If the number of trades is not at least dozens of times larger than the number of parameters, do not trust the optimization result itself.

Before running optimization, count how many trades actually occurred.

The Better the In-Sample Curve Looks, the More Suspicious You Should Be

Counterintuitively, the smoother the backtest curve looks and the more it rises without a single meaningful drawdown, the more suspicious you should be. Real markets always contain drawdown periods that are hard to endure. If a backtest glides through even those periods, it may be a sign that the rules were tuned to avoid every past decline. Future declines will arrive in a different shape, so settings that avoided every historical decline may be fully exposed to the first future one.

This trap often appears when traders keep adding stops, take-profits, and filters to remove only the losing trades they see in the past. They run a backtest, notice a specific losing trade, add a filter to avoid it, then run the test again and add another condition to avoid the next loss. The curve gets smoother each time, but most of the added filters have no basis beyond avoiding that one historical trade. In the end, the strategy has merely memorized a list of specific past losses while failing to understand the structure that creates losses.

Running hundreds of combinations on the same data and selecting the highest peak produces the same result. If you test 1,000 combinations, a few of them will pass through the historical window almost perfectly by pure chance. Selecting the best among them is basically selecting the setting that got luckiest. Repeatedly looking at the same data to choose the best result is commonly called data snooping, and the more combinations you test, the higher the probability of finding a lucky peak. If the curve looks unrealistically perfect, it usually means you have looked at the data too many times.

A backtest with no drawdown is not something to brag about. It is the first thing to inspect.

Neighboring Performance Separates Cliffs From Plateaus

The most direct way to distinguish a robust setting from an overfit one is to examine its neighbors. Do not record only the return of the optimum. Build a table with the returns of the values on both sides and inspect the shape. If the shape is a sharp spike, it is a cliff. If it is a broad hill, it is a plateau. Repeat the same process for every key parameter.

Looking at neighboring performance works because it simulates future market-state changes as movement through parameter space. A future where volatility rises and RSI signals arrive one beat later is similar, in backtest terms, to the optimum shifting from period 14 toward 16. A plateau-shaped setting that still performs well at period 16 can survive that future. A cliff-like setting whose performance drops by more than half at period 16 will break under it. Neighboring performance becomes a proxy for future robustness even without future data.

Apply the following checks to every key parameter.

• [ ] Set a neighbor range: Centered on the optimum, record backtest performance for three steps on each side, for a total of seven values over the same period. For example, if the optimum is RSI period 14, check performance for 11, 12, 13, 14, 15, 16, and 17.
• [ ] Identify a plateau: If the middle five values closest to the optimum, such as 12 through 16, all maintain at least 70% of the best value's performance, treat it as a plateau and accept it.
• [ ] Reject a cliff: If moving just one step from the optimum drops performance below half of the best value, treat that optimum as overfit and discard it. Choose another value closer to the middle of a plateau instead.
• [ ] Check multiple parameters together: If there is more than one parameter, do not vary only one at a time. Check whether the plateau remains on a grid where two parameters move together. A ridge that is flat on only one axis is no better than a cliff.

Stop recording only the best point among seven values. Record all seven and look at the shape.

Splitting Time Periods Filters Out Memorized Curves

If neighboring-performance checks test parameter space, period splits test the time axis. Do not optimize the full historical period all at once. Choose parameters on the earlier section, then validate them on the later section. The earlier section is called in-sample. The later section is called out-of-sample. If a setting looked impressive in-sample but collapses out-of-sample, it memorized the earlier data.

This split detects overfitting because the validation window is completely isolated from the optimization process. If the later data was never seen when the parameters were chosen, its results show honestly how those parameters behave on unseen data. If one split is not enough, extend the process into walk-forward analysis by rolling the in-sample and out-of-sample windows forward through time. By repeatedly checking whether parameters selected in one window work in the next, you can separate settings that worked once by chance from settings that work consistently.

Adding a Monte Carlo check tests robustness one step further. Randomly shuffle the order of the trades from the backtest thousands of times, or randomly shift entry timing by a few days, then examine the distribution of results. If simply changing trade order causes some simulations to fall into drawdowns the account cannot tolerate, the original smooth curve depended on trades happening in a conveniently favorable order. If performance changes sharply when entry timing is shifted by a few days, the strategy depends too heavily on a specific price of a specific candle. In a robust strategy, the center of the distribution should not move much under these random disturbances.

Apply these three checks in order.

• [ ] Period split ratio: Use the first 70% of the historical period as in-sample to choose parameters, then validate on the final 30% as out-of-sample. If out-of-sample performance falls below half of in-sample performance, treat it as overfit and choose the parameters again.
• [ ] Walk-forward repetition: Roll the in-sample and out-of-sample windows forward through time at least five times, and check whether the out-of-sample period avoids a loss in at least four of the five windows.
• [ ] Monte Carlo distribution: Shuffle trade order at least 1,000 times and check whether the drawdown at the worst 5% point remains within the limit the actual account can tolerate.

A result optimized on the entire period without splitting time cannot be called validation. It is closer to grading your own answer with the same data. The stronger the in-sample result looks, the more it must clear out-of-sample testing as well. Only settings that survive both should go live.