How it is tested

Core backtest validation

1. A1

   Massive Grid Search

   We search a wide parameter space and look for a plateau where results hold up, not a single best point.

2. A2

   Parameter-Space Robustness

   We compute beta-adjusted alpha for every grid pattern and report the share of patterns with positive alpha, not just the champion.

3. A3

   In-Sample / Out-of-Sample

   Selection uses only data up to a cutoff; evaluation uses the unseen period after it.

4. A4

   Multi-Window Walk-Forward

   We repeat select-then-evaluate as time moves forward, across several in-sample window lengths, so one lucky window cannot carry the result.

5. A5

   Beta-Adjusted Alpha

   We estimate market beta and evaluate what remains after market exposure is removed; strategies can and do fail this step.

6. A6

   Expanding Window

   We also test with an expanding training window, so results do not depend on one rolling window length.

7. A7

   Regime Robustness

   We break results down by market regime instead of relying on a single average.

8. A8

   Vintage OOS

   We rerun the research as if we stood at an earlier date, using only information available then, and evaluate what followed.

9. A9

   Reject / Adopt

   Strategies that pass early tests are still rejected when later tests fail; rejected research is kept on record.

10. A10

    Real Prospective Forward

    After every backtest check, the researcher still knows the past; only signals published in advance and observed afterward remove that.

DM-Signal robustness tests

1. B1

   Alpha Space Check

   Measures the CAGR positive-rate across the entire parameter space. Broadly positive results across the parameter space reduce the likelihood that performance comes from a single isolated parameter spike, but do not by themselves rule out overfitting.

2. B2

   Decay Analysis

   Directly measures whether a champion selected in the past has eroded over time, using p_det_roll/p_erosion across 6/12/24-month windows.

3. B3

   Vintage Analysis

   Verifies whether out-of-sample performance reproduces across champions selected at different points in time, using selection timelines and IS/OOS comparison.

4. B4

   Parameter Neighborhood Analysis

   Confirms whether a champion is an isolated spike or a broad peak, by comparing the performance drop-off across neighboring parameters.

5. B5

   Stress Test

   Examines whether the apparent edge holds up against the benchmark during extreme periods such as sharp drawdowns or rate-hike shocks.

6. B6

   Fold Percentile Verification

   Confirms whether a fixed champion maintains better-than-random performance within each walk-forward fold's population, via fold percentile.

7. B7

   Metric Top-Stability

   Measures how well the top set is preserved as the calculation window is shortened, using overlap rate / Jaccard similarity.

8. B8

   Regime-Conditional Verification

   Checks whether alpha remains positive across Bull/Bear/Sideways regimes, exposing apparent strength that depends on a single market condition.

9. B9

   Rule Freeze

   Locks selection rules and parameters at a fixed point in time, with no undocumented changes afterward. This freeze point becomes the boundary against the still-unverified future.

10. B10

    Forward Observation

    Observes and recalculates only real-time results after the Rule Freeze, kept distinct from the backtest. Forward metrics are computed using only rows from the forward period.

Research log

The log of adopted, rejected, under-test and retired research is in preparation. It will be linked from here once published.

Read it in the Trust Center →