A signal is a claim. How we test forecasts before anyone sees them
A forecast says something specific about the future. That makes it testable, and it means it can be wrong. MYG Data, our research division, treats every signal as a claim that has to survive three checks before anyone looks at a return: cross-validation that cannot leak, calibration that holds up, and a record of every prediction written down before the outcome is known. This post explains how each one works and why we built it that way.
Why a good backtest proves very little
A positive backtest is easy to produce. The hard part is knowing whether it means anything. In our experience there are four ways a forecasting number misleads, and each needs its own defence.
- Leakage. Information from the test period reaches the model through the training data, usually through labels that overlap in time.
- Selection. Try enough features and one will look good by chance. The more trials, the less a single good result is worth.
- Miscalibration. A model can rank cases in the right order and still state the wrong probabilities. A stated 0.9 that happens half the time is not usable for sizing a decision.
- Hindsight. If predictions can be edited, re-run or back-dated after the fact, the record shows what someone wishes they had said.
The internal tooling behind MYG Data exists to close each of these. None of it is exotic. The methods come from published work by Bailey and López de Prado and from standard forecast verification. What matters is that they run on every candidate, automatically, before a human forms an opinion.
Purged cross-validation with an embargo gap
Ordinary k-fold cross-validation assumes the samples are independent. Time series break that assumption. When a sample's label is only resolved at some later date, its label window overlaps the windows of its neighbours. Put one neighbour in training and the other in test, and the model has already seen part of the answer.
Our validation module orders samples by decision date and cuts them into contiguous, time-ordered test blocks. For each block, the training set is everything outside it, minus two groups:
- The purge. Any training sample whose label window overlaps the test block's time span is dropped.
- The embargo. A buffer of samples immediately after the test block is also dropped, because serial correlation bleeds forward in time.
The defaults are five splits and an embargo of one percent of the samples. If a fold has fewer than ten usable training samples after purging, it is skipped rather than scored, because an information coefficient computed on that little data is noise. Feature preparation follows the same rule: medians for imputation and the mean and spread for standardising are fitted on the training fold only, then applied to the test fold.
The difference is not academic. On our own data, a simple expanding-window walk-forward reported a noticeably better result than the purged, embargoed version of the same test. The gap between the two numbers was leakage, and the purged one is the only one we report.
Charging for every trial
Purging removes leakage. It does nothing about selection. If ten features were tried and one was kept, the kept one has to clear a higher bar than if it had been the only idea. Two statistics handle this.
Deflated Sharpe ratio
The deflated Sharpe ratio corrects an observed Sharpe for the number of trials that were run to find it, and for the skew and fat tails of the series. The benchmark it must beat is the expected maximum Sharpe across that many trials under the null, so more trials raise the bar. The result is a probability that the true Sharpe is above that benchmark. We pass the real trial count in, not a flattering one.
Probability of backtest overfitting
The probability of backtest overfitting uses combinatorially symmetric cross-validation. The history is split into chunks, and for every way of picking half of them as in-sample, we take the configuration that looked best in-sample and check where it lands out of sample. If selection is fitting noise, the winner falls below the out-of-sample median about half the time. If the edge is persistent, it rarely does.
A feature set survives the overfitting check only if all of these hold together: a positive purged out-of-sample information coefficient with p below 0.05, a deflated Sharpe above 0.95, and a probability of overfitting below 0.5. A separate promotion gate then asks whether the candidate adds value over the features already in use, including after neutralising for common factors. Candidates do fail this gate. Each rejection is written down with the reason, so it is on file the next time the idea comes up.
Calibration: does 0.70 mean 70 percent?
Ranking and calibration are different properties. A model can order cases well and still be badly calibrated, with its 0.9 bucket coming true far less often than 90 percent of the time. For anyone using a probability to decide how much to commit, calibration is the property that matters.
The calibration module works on pairs of predicted probability and binary outcome. It reports:
- A reliability curve over ten equal-width bins: mean forecast against observed frequency, with the count per bin.
- The Brier score, the mean squared error of the probabilities.
- Murphy's decomposition into reliability, resolution and uncertainty. The identity is exact against the binned Brier score, and a unit test asserts it.
- Expected calibration error, the count-weighted average gap between forecast and outcome across bins.
Correcting miscalibration has its own leakage trap. An isotonic calibrator fitted on the same fold it is scored on will always look well calibrated. Ours is split into separate fit and transform steps, and the combined helper takes training scores, training outcomes and test scores as three separate arguments, so the test fold cannot accidentally be the data the map was learned from. Because the fit is monotone, it corrects the level of each probability without changing the ranking.
The record is written before the outcome
Validation on history, however careful, is still a backtest. The only evidence of foresight is a prediction that was fixed before the outcome existed. So every live prediction is locked into an append-only ledger ahead of the event it forecasts.
Point-in-time is absolute. A feature for decision date D may only use data whose public availability stamp is on or before D. Each prediction row carries the date it was locked, the period it forecasts, the probability, the commit of the code that produced it and a hash of the configuration, so a change in method is visible in every row that follows.
The ledger is a plain file of JSON lines. Each row commits to the one before it:
row_sha = sha256(prev_sha + "|" + kind + "|" + canonical_json(payload))
{"seq": 3, "kind": "reconciliation", "prev_sha": "cbeefada...",
"payload": {...}, "row_sha": "9c7d8be7..."}A few rules make the chain worth having:
- Never edited. A locked prediction is immutable. Re-running the job skips anything already locked for that case and date. Corrections are new rows.
- Outcomes are rows too. When the result is known, a reconciliation row is appended to the same chain. It recomputes the outcome with the same function that defined the target during validation, so the scoring cannot drift from the framing.
- Anyone can verify it. A verifier re-walks the chain from the genesis row, recomputes each hash, and reports the first row where the sequence, the previous hash or the row hash does not match.
- Backfill is labelled. Rows generated for past dates to demonstrate the pipeline are flagged as backfilled and excluded from live statistics.
That last rule came from a lesson we wrote down early: a backfilled hash chain is not a track record. Tamper-evidence on backfilled rows only proves that a backtest was hashed. Our internal rule is that the live record runs for at least four quarters before it is cited as a record at all.
What we do not claim
MYG Data makes no performance claims for any signal. The division exists to prove or kill a signal on evidence, and the code is built to reflect that. It can report pass or fail on the gates it can check. It cannot declare anything production-ready. The deciding gates are ones code cannot satisfy on its own, such as licensed data for the true label and a multi-quarter live record that was locked before each outcome and reconciled after it.
The data rules are equally plain. Public and licensed data only, no material non-public information, and no personal data. The method and the validation records are shared under NDA with investment firms that want to see how the work is done.
The same discipline travels. The validation suite and the tamper-evident ledger were built for this division first and are now part of the core that client deployments run on. If a system makes predictions that people act on, it should be able to show what it said, when it said it, and how often it was right.