Draws

A draw is a statistical tie, not a claim that two forecasters are equal.

Every head-to-head on this site reports wins, draws and losses. A win or loss is only declared when the difference in forecast quality is large enough to stand out from the noise. Everything else is a draw. This page defines that test.

The test

For a single series, take the two forecasters' per-step log-scores and difference them, step by step:

The mean d is how much better A scores than B on that series, in nats per step. A Diebold–Mariano test asks whether d is far enough from zero to be real, dividing it by its standard error to form

and comparing that to the normal critical value z = 1.96. The verdict is

• win for A when t > 1.96,
• loss for A when t < −1.96,
• draw otherwise, i.e. when d lands inside the band ±1.96 · SE.

That band is the 95% confidence interval for the mean differential. A draw is the case where it straddles zero: the average log-score gap is smaller than its own sampling error, so neither forecaster is distinguishable from the other on this series.

The standard error

Log-score differentials are serially correlated: a series in a hard patch scores poorly for both forecasters several steps in a row. Treating steps as independent would understate the uncertainty and manufacture significant wins. The standard error uses a HAC (Newey–West) long-run variance with a Bartlett kernel and a lag of round(n1/3), which widens the band to account for that autocorrelation. Per-step log-scores are floored at −20 nats so a single catastrophic tail miss cannot dominate the mean. Series with fewer than eight scored steps are called draws automatically, since the test has no power there.

How to read a high draw count

Draws are common here, and on some strata they are the majority. That reflects the power of the test on short, one-step samples, not a claim that the forecasters are identical. Two consequences follow.

First, the bar for a win is deliberately high. A forecaster that is slightly better on average will still draw until the evidence accumulates, so win counts understate small real edges and grow as coverage deepens.

Second, a draw is a good outcome for a challenger only in the sense that it is not a loss. The "never worse than Laplace" property is measured as wins plus draws: the portfolio may not beat Laplace, but the test cannot separate it from Laplace on the series in question. The sidecar and per-model pages read their not-worse rates this way.

Scope

All of these comparisons are one-step-ahead: the horizon is k = 1. Each forecaster predicts the next change, is scored on it, and rolls forward. A multi-horizon sweep (k = 1, 2, 4, 8, 16) is a planned extension and would get its own protocol; the draw definition above carries over unchanged to any fixed horizon. The full method is on the methodology page.