Benchmarks
A new practical benchmark for univariate distributional prediction — simpler and lighter than Prophet, and scored where it counts: held-out likelihood.
Benchmark 1 — the non-price universe
A fair rolling one-step-ahead comparison on the non-price FRED universe —
894 continuous daily change-series (57 correlated families); equity/fx/commodity returns are
excluded and treated separately in the price caveat below. Every method — ours and theirs — is
turned into the same Dist and scored on held-out log-likelihood and CRPS.
Per-series win-rate of laplace, reported raw and family-weighted
(one vote per correlated family):
| Baseline | LL (raw / fam) | CRPS (raw / fam) | mean LL |
|---|---|---|---|
| AutoARIMA | 82 / 90% | 53 / 65% | 2.80 |
| AutoETS | 97 / 99% | 83 / 92% | 2.79 |
| SARIMAX (statsmodels) | 87 / 86% | 53 / 64% | 2.92 |
| ETS (statsmodels) | 96 / 98% | 81 / 90% | 2.89 |
GARCH(1,1)-t (arch) | 68 / 65% | 54 / 45% | 3.03 |
| AutoARIMA + conformal | 86 / 88% | 31 / 31% | 2.27 |
| AutoARIMA + ACI | 88 / 88% | 29 / 29% | 2.51 |
| naive conformal | — (CDF) | 92 / 93% | — |
| NF-StudentT (NeuralForecast) | 100 / 100% | 78 / 76% | 0.82 |
laplace (mean log-likelihood 3.20) wins the likelihood race
against every baseline on non-price series — including GARCH-t, the heavy-tail
SOTA (68% raw / 65% family-weighted; mean LL 3.20 vs 3.03). On CRPS it beats the mean-model
baselines and loses only to the CRPS-specialists (fitted conformal ~30%, GARCH-t 45%
family-weighted) — their home turf. No free lunch on price/returns: on
equity/fx/commodity return series GARCH-t has the higher log-likelihood, and we recommend it
there rather than averaging the two regimes into one number.
The study →
Benchmark 2 — zero-shot foundation models
A different protocol: pretrained models used zero-shot (fixed 256-context, no refit) on 120
change-series (69 continuous). Per-series win-rate of laplace:
| Model | density | LL (all / cont) | CRPS (all / cont) |
|---|---|---|---|
| Moirai (1.1-R-small) | native Student-t | 98 / 97% | 94 / 93% |
| Lag-Llama | native Student-t | 67 / 99% | 65 / 94% |
| Chronos-Bolt (small) | quantile* | 76 / 100% | 67 / 88% |
| TimesFM (2.5-200M) | quantile* | 75 / 100% | 58 / 72% |
On the continuous series, laplace beats all four zero-shot. The native-density
models (Moirai, Lag-Llama) are the closest. Naive per-series fine-tuning doesn't
rescue them — it catastrophically overfits (logpdf +1.5 → −118). A zero-dependency online
ensemble holding its own against 200M-parameter transformers.
The foundation study →
Challengers
Every method that has stepped up, one row each with its tape, lives on its own page now, built for the ones still coming: Challengers.
Reproducibility
Every study's harness and committed results live in skaters/benchmarks, beside the statements filed before results existed. The smaller studies regenerate from committed inputs alone; the paper-scale tables also need the full FRED cache, which is larger than the repository carries, so each study's README says what its rerun requires. The papers behind them are on the papers page.