David v Goliath

Laplace vs TimesFM: one tiny online forecaster against Google’s 200-million-parameter foundation model, series by series, with the protocol and the losing rows on the table.

The setup

TimesFM 2.5 (200M parameters) is run zero-shot, as intended: a fixed 256-length context window of preceding one-step changes, predict the next change, no fitting, all windows batched into one call. Its quantile output is reconstructed into the same Dist object every method uses, and laplace is re-scored on the identical series and windows. 120 FRED change-series, 150 test steps each, scored on held-out log-likelihood and CRPS. The harness is foundation_study.py and the raw per-series results are checked in.

The headline

On the 69 continuous series, Laplace wins log-likelihood on 69 of 69, with a median gap of 2.26 nats per point (3.02 vs 0.76), and wins CRPS on 50 of 69 with the median CRPS ratio at 0.983. One caveat travels with the likelihood number: the TimesFM density is quantile-reconstructed and tail-limited, so CRPS is the fairer read for it, and Laplace wins that too.

Where TimesFM wins: predicting zeros

TimesFM only beats Laplace on stupid tasks, chiefly predicting zero again. Every one of its largest CRPS wins is a binary recession indicator (CHNRECD, AUSRECD, BRARECD and friends) whose change series is constant over the entire test window. Its quantile head emits a literal atom at zero and scores near machine precision; Laplace concentrates hard but leaves smooth mass and scores near 1e-9. Both are saying “zero again”; one says it bit-exactly, and the ratio looks enormous while the scores differ in the ninth decimal. That is 51 of the 120 series, which is why the continuous split carries the verdict.

This will not necessarily stay true. Foundation models improve quickly, larger checkpoints and native density heads exist, and the comparison here reads TimesFM through a quantile reconstruction. The rerun conditions are tracked in skaters#97, and this page changes when the numbers do.

Every series at once

One bar per series, sorted by the per-point log-likelihood gap, Laplace minus TimesFM. Above the zero line Laplace is better: 90 of 120 overall, and all 69 continuous series (blue). The gray bars below the line are the repeated-value games above.

Footprint

The two artifacts are not the same kind of object, and the size gap is part of the result. These are measured on this machine, single CPU core, except the TimesFM figures, which are the published ones.

Fairness accounting

What favours Laplace here: the target is the change stream, the object it was built for, while TimesFM was chiefly trained on level series; and Laplace has seen each series’ full history where TimesFM sees 256 points. What favours TimesFM: it carries knowledge from a vast pretraining corpus that a from-scratch online method cannot have, and the LL comparison is read through its reconstructed density, which is why CRPS is reported with equal weight. Fine-tuning to a single short series was tried for this model family and catastrophically overfits (the paper’s footnote), so zero-shot is the intended comparison.

Can Laplace teach TimesFM?

Issue #133 tested domain-level distillation on 35 training series, selected on 6 validation series, and scored on 15 disjoint test series (960 causal origins). The first pilot decoded the TimesFM head incorrectly and is invalid. Fresh adapters use channel 5 as the mean and channels 0,1,2,3,4,6,7,8,9 as q10–q90.

The committed sidecars preserve those native channels before ordering repair. They cross on almost every test row; canonical density scores apply a declared row-wise PAVA isotonic projection before the validation-frozen mixture reconstruction.

Against zero-shot TimesFM, the corrected adapter has −0.155385 median dLL, a 1.002855 CRPS ratio, 0/15 LL wins, and 7/15 CRPS wins. Its LL Diebold–Mariano record is 0 wins, 10 draws, and 5 losses. Context 256 does not reverse the result.

Matched-context Laplace also wins: the corrected adapter is −0.510919 median dLL behind when both see 128 values. The validation-frozen teacher q-grid and exact GMM body both remain close to complete Laplace, so the corrected experiment identifies failed student transfer rather than a dominant q-grid reconstruction loss.

Both corrected adapters are negative research evidence, not deployment candidates. The technical report includes chronology, 128/256 results, frozen source data, fresh adapters, raw quantiles, canonical predictions, checksums, limitations, and exact commands.

What has not been run

Thoroughness includes the ledger of what is missing. Point accuracy (MAE of the median) has not been scored, though CRPS bounds how different it can be. The level-series protocol, the TimesFM home turf, has not been run on this universe. The original 120-series zero-shot protocol has not tested contexts other than 256, larger checkpoints, or per-series compute cost. The separate issue-133 study uses a 128 context and a post-hoc 256-context sensitivity on 15 disjoint test series. The zero-shot harness needs an environment with timesfm installed and a FRED key; everything else is one command.

Run this study yourself

The whole protocol is packaged as a copyable skill: hand it to Claude with your own series and it reproduces the study, the splits, the fairness accounting and the report format. Get it from the skills page or on GitHub. The reference harness is foundation_study.py; it needs an environment with timesfm installed and a FRED key, and everything else is one command.

The context for these numbers

This is one cell of a broader study: the same protocol covers Chronos-Bolt, Moirai and Lag-Llama, and Laplace beats all four on the continuous split, with the native-density models closest. The rolling eight-baseline study, the price caveat where GARCH-t wins, and the full tables are on the papers page; slice the non-price benchmark yourself in the robustness explorer.