Sundial vs Laplace

Sundial is a generative time-series model; we draw samples per step and score the empirical predictive. Run raw only in this study; the collaborative arms were built for the three strongest models first.

Resources: GitHub · Model card · Paper (arXiv 2502.00816)

This page measures one thing: how sharp and how well calibrated Sundial's univariate predictive density is, one step ahead on a change series, scored against Laplace on identical windows. It leaves out the covariates, the multivariate inputs, the long horizons and the native quantile or path outputs the model was built for, and its log score passes through a reconstruction of the model's quantiles (the table note says which). A model can trail here and be the better tool for its own job.

Live snapshot  Derived from the week-long round-robin study; counts grow as coverage deepens. Everything scores against Laplace on the identical series and windows.

Standalone

The model's own predictive, run zero-shot, scored per series against Laplace. Win, draw and loss are decided one series at a time. Each series has a rolling test window of 64 one-step-ahead points, and both forecasters are scored on the same points. For each point take the difference of log predictive densities, dt = log pmodel(yt) − log pLaplace(yt), with each log density floored at −20 first. The series' mean d̄ is tested against zero by a Diebold–Mariano test: the standard error is Newey–West with a Bartlett kernel and lag round(n^1/3). The model wins the series if d̄ > 1.96·SE, loses it if d̄ < −1.96·SE, and otherwise the series is a draw; a series with fewer than eight finite differences is a draw. A draw says the 64 points cannot separate the two forecasters at the 5% level. It does not say they are equal. With a window this short both win and loss counts understate the true rates, and the median ΔLL is the better measure of size. Counts are over the series both arms cover.

Sundial issues sample paths rather than quantiles. The arm draws 30 samples per step and scores a kernel density on them. That is also a reconstruction, with a bandwidth and a sample count behind it, and it has not been rescored under a more generous one as the quantile models have, so these numbers are a lower bound on the model.

Median per-series Δ log-likelihood in nats (negative is worse than Laplace); CRPS ratio to Laplace (above 1 is worse); central-90% coverage (0.90 target).

Star plot

Sundial standalone against Laplace, on the same six regime axes as the site's standalone radar. Each radius is the log-likelihood win rate, (wins + ½·ties) / n, divided by one half so an even split with Laplace sits on the dashed 1.0 ring; outward beats Laplace more often, inward less. The quantity is a share of series. The solid polygon is the headline arm and the dashed grey one, where present, is the tight nine-decile reconstruction. The M4-hourly set splits into soft and hard waveforms by corpus order, matching that radar.

Collaborative use

The sandwich arms (+lap, ~lap, @lap, &lap) have not been run on Sundial. The sidecar wraps any per-step predictive, so they can be added without retraining. See the sidecar pattern.

Protocol

Fixed 128-length context, rolling one-step test window, no fitting, each model in its own environment. Strata split the cached FRED universe and the M4-hourly set by frequency and regime. Full method on the sidecar pattern page and in the methodology.

Architecture and methodology

Sundial is a decoder-only transformer trained without discrete tokenisation. It embeds continuous-valued patches and learns a generative TimeFlow head by flow-matching, so instead of a parametric density it produces samples of plausible futures. This study draws those samples (the base-128m model) and scores their empirical predictive; it is pretrained on the trillion-point TimeBench corpus.