OptiNod Academy
The Three Robustness Scores — How to Decide Which of Two Equal-Return Settings to Trust
Between two parameter settings with the same return, three stability scores (neighborhood stability, Monte Carlo, top-N consistency) point to the one that survives live trading. Explained against OptiNod's actual implementation.
> What the leaderboard's number one really is comes down to how closely the cell next to it earned the same amount.
Robustness, in its original statistical sense, refers to the property of an estimate not shifting much even when an assumption is slightly off. Carried over into trading, the definition compresses into a single line: does performance hold up about the same when you nudge a parameter one cell over, or change the data window a little? OptiNod's robustness engine scores this question from three angles and bundles them into one composite score.
The common view treats this concept as a formality you check once after the backtest is done and then move past. You sort the leaderboard by return, pick the number one, and pass it through as long as its stability score is not outright red. The score gets treated as supplementary information tacked on after the return ranking is already set.
That order is backwards. The three scores are not a step you check after picking the number one. They should be the very criterion by which you pick it. When two settings have the same return, the three scores decide which is better. The lower-scoring one is a peak built by curve fitting, so its performance does not follow through in live trading. From then on, you check the stability columns before the return column when you look at a leaderboard.
If the Next Cell Is a Cliff the Top Is Chance; If It Is Flat It Is a Real Peak
The first score, neighborhood stability, looks at how even the performance is across the parameter cells around the optimal combination. OptiNod takes the top 10 results by performance and, on each parameter axis, gathers the performance variance of the neighbors one cell forward and back (±1 step). It takes the square root of the mean of those variances (√mean neighborhood variance) and divides it by the standard deviation of overall performance to get sensitivity. The lower the sensitivity, the higher the score, and this score makes up 35% of the composite.
This score is central because performance built by genuine market structure follows through about the same even when a parameter is off by one cell. Changing an EMA period from 20 to 21 does not make a trend disappear. If the neighboring cell's performance drops off a cliff, that number one is just a result that happened to fit a few specific candles. In live trading those candles do not come again, so the performance of such a setting soon vanishes.
On May 19, 2021, BTC fell from an open of $42,850 to an intraday $30,000 before closing at $36,690. A setting that has tuned its stop distance down to the single tick to fit a one-off crash candle like this easily becomes the number one backtest return. But widen the stop distance by just one cell and performance drops sharply. The neighborhood stability score catches this difference. On the chart, do not look only at where the single stop line sits; look at the spread of performance when you push that stop line one cell to either side. If the performance swing is small, it is a stable region; if large, it is a region leaning on chance.
If Random Draws Produce This Much Performance, It Is Hard to Call It Skill
The second score, the Monte Carlo test, compares the top setting against the distribution of results drawn at random from the same parameter space. OptiNod repeats this 200, 500, or 1000 times depending on the result count, builds the average distribution of random top groups, and computes which percentile the actual top setting sits in within that distribution, along with its statistical significance (pValue). This score is also 35% of the composite.
The pValue band scores are as follows.
| pValue | Score |
|---|---|
| below 0.01 | 95 |
| below 0.05 | 80 |
| below 0.1 | 65 |
| below 0.3 | 45 |
| 0.3 or above | 25 |
The question this test answers is simple. Is my number one at a level you would get picking anything at random from this parameter space, or is it a real peak outside the distribution? A pValue of 0.3 means three out of ten random draws produce performance similar to my number one. That is weak grounds for treating that number one as a result of skill. A pValue of 0.01 means the chance that performance came from luck is below one in a hundred.
A strategy that cannot beat the market average gets caught here. During the FTX collapse in November 2022, BTC dropped from $18,545 on November 9 to an intraday $15,588. Optimize a long strategy on bear-market data that includes a crash stretch like this, and most parameters you pick lose money. If the number one then sits in the middle of the random distribution, the Monte Carlo score comes out low. This is not a problem of parameter selection. It is a signal that applying this strategy to this window was a stretch in the first place. To narrow the gap between backtest and live trading, you have to check this distribution position first.
If the Top Group Clusters in One Place It Is a Real Peak; If It Scatters It Is Chance
The third score, top-N consistency, looks at how narrow a range the parameters of the top 20% of results (a minimum of 3 if results are few) cluster within. For each parameter, OptiNod computes the ratio of the range the top group occupies to the full range, and counts how many distinct parameter combinations there are among the top group as a cluster count. The narrower the range and the fewer the clusters, the higher the score, and it carries 30% of the composite.
The top group clustering in one region means there is a setting range that actually works in the market. If the top group for EMA period all gathers between 18 and 24, that vicinity is a real peak. If the top group is scattered across period 10 and 50 and 90, each result is the product of a different coincidence, and there is no guarantee that whichever you pick will show up again in the next window.
Recall the volatility stretch on March 14, 2024, when BTC printed a new high of $73,777 and then swung more than $5,000 down to $68,555 on the same day. In a stretch like this, settings that happened to catch one or two big candles well rise into the top group in scattered fashion. A high cluster count means the peak failed to consolidate into one and several coincidences got mixed together. Split the window with walk-forward analysis and you can confirm that the position of such a scattered top group shifts to different cells from window to window.
When Returns Are Equal, the Composite Score Decides Which Is Better
The three scores are bundled at 35%, 35%, and 30% weights into the composite score. If there are fewer than 5 results, score calculation is withheld (insufficient sample). Arguing about stability on an inadequate sample is itself another form of overfitting.
The reason to read all three scores together is that each catches a different kind of coincidence. Neighborhood stability catches a cliff one cell over, Monte Carlo catches ordinary performance buried inside the distribution, and top-N consistency catches a scattered peak. A setting where one score is high and the other two are low is curve fitting that passed only one kind of test.
The following is a procedure for filtering two candidates with effectively the same return before putting them live. The cutoff numbers are operating recommendations. They are not values the product enforces, so adjust them to the character of the strategy.
1. First sort: Change the leaderboard sort criterion from return to composite score, descending.
2. Neighborhood stability floor (recommended): Exclude any candidate whose parameter score (a blend of sensitivity and range) for a core parameter (stop distance, entry period) is below 60. For reference, the engine flags below 40 as sensitive and 40 to 60 as moderate.
3. Monte Carlo floor (recommended): If pValue is 0.05 or above (score below 80), the skill grounds are weak, so withhold it.
4. Top-N cluster check (recommended): If the distinct parameter combinations (cluster count) among the top 20% of results exceed half the top sample, treat the peak as scattered and exclude it.
5. Deployment decision: Among the candidates that pass the three conditions above, pick the one with the higher profit factor and the lower maximum drawdown.
Two Patterns of Misreading the Scores
Looking only at the composite score and not checking the three detail scores. A composite of 70 could be all three scores at 70, or one score at 95 and the other two in the 50s. The latter is a dangerous setting that passed only one test. The composite is only a starting point, and you must not skip the step of expanding the three detail scores to see which test is weak.
Trying to raise the score without growing the sample. Even when the score comes out high with only 7 or 8 results, that can be an illusion where the variance was measured as small because the sample is small. In the band where Monte Carlo repetitions drop to 200 (10 or fewer results), it is hard to distinguish pValue finely. To trust the score, first run enough combinations to grow the sample to 30 or more.
Trust It Only When the Three Scores Point the Same Way
Robustness scores do not promise future returns. They are a tool for sorting out whether, within the backtest window, that number one is a result of skill or a result of chance. So the final check is whether the three scores point in the same direction. The criteria below are likewise recommended operating values.
- [ ] Neighborhood stability: is the core parameter score 60 or above?
- [ ] Monte Carlo: is
pValuebelow 0.05 (score 80 or above)? - [ ] Top-N consistency: is the top 20% cluster count gathered within half the sample or fewer?
- [ ] Sample: are there 30 or more results, so the score resolution is sufficient?
- [ ] Did you expand and look at all three detail scores, not just the composite?
A setting where the three scores point the same way has a high probability of holding similar performance even when the data window changes. It is distinct from a chance setting tuned to a single shock candle like the one on August 5, 2024, when the yen carry unwind pushed BTC from $58,161 down to $49,000. Between two settings showing the same return, these three scores decide which to trust.
