David v Goliath

Laplace vs TimesFM: one tiny online forecaster against Google’s 200-million-parameter foundation model, series by series, with the protocol and the losing rows on the table.

The setup

TimesFM 2.5 (200M parameters) is run zero-shot, as intended: a fixed 256-length context window of preceding one-step changes, predict the next change, no fitting, all windows batched into one call. Its quantile output is reconstructed into the same Dist object every method uses, and laplace is re-scored on the identical series and windows. 120 FRED change-series, 150 test steps each, scored on held-out log-likelihood and CRPS. The harness is foundation_study.py and the raw per-series results are checked in.

The headline

On the 69 continuous series, Laplace wins log-likelihood on 69 of 69, with a median gap of 2.26 nats per point (3.02 vs 0.76), and wins CRPS on 50 of 69 with the median CRPS ratio at 0.983. One caveat travels with the likelihood number: the TimesFM density is quantile-reconstructed and tail-limited, so CRPS is the fairer read for it, and Laplace wins that too.

Where TimesFM wins: predicting zeros

TimesFM only beats Laplace on stupid tasks, chiefly predicting zero again. Every one of its largest CRPS wins is a binary recession indicator (CHNRECD, AUSRECD, BRARECD and friends) whose change series is constant over the entire test window. Its quantile head emits a literal atom at zero and scores near machine precision; Laplace concentrates hard but leaves smooth mass and scores near 1e-9. Both are saying “zero again”; one says it bit-exactly, and the ratio looks enormous while the scores differ in the ninth decimal. That is 51 of the 120 series, which is why the continuous split carries the verdict.

This will not necessarily stay true. Foundation models improve quickly, larger checkpoints and native density heads exist, and the comparison here reads TimesFM through a quantile reconstruction. The rerun conditions are tracked in skaters#97, and this page changes when the numbers do.

Every series at once

One bar per series, sorted by the per-point log-likelihood gap, Laplace minus TimesFM. Above the zero line Laplace is better: 90 of 120 overall, and all 69 continuous series (blue). The gray bars below the line are the repeated-value games above.

Footprint

The two artifacts are not the same kind of object, and the size gap is part of the result. These are measured on this machine, single CPU core, except the TimesFM figures, which are the published ones.

Fairness accounting

What favours Laplace here: the target is the change stream, the object it was built for, while TimesFM was chiefly trained on level series; and Laplace has seen each series’ full history where TimesFM sees 256 points. What favours TimesFM: it carries knowledge from a vast pretraining corpus that a from-scratch online method cannot have, and the LL comparison is read through its reconstructed density, which is why CRPS is reported with equal weight. Fine-tuning to a single short series was tried for this model family and catastrophically overfits (the paper’s footnote), so zero-shot is the intended comparison.

What has not been run

Thoroughness includes the ledger of what is missing. Point accuracy (MAE of the median) has not been scored, though CRPS bounds how different it can be. The level-series protocol, the TimesFM home turf, has not been run on this universe. Context lengths other than 256, larger TimesFM checkpoints, and per-series compute cost are unrun. None of these should reverse a 69-of-69 likelihood split, and this page will be updated if any of them is run and says otherwise. The harness needs an environment with timesfm installed and a FRED key; everything else is one command.

Run this study yourself

The whole protocol is packaged as a copyable skill: hand it to Claude with your own series and it reproduces the study, the splits, the fairness accounting and the report format. Get it from the skills page or on GitHub. The reference harness is foundation_study.py; it needs an environment with timesfm installed and a FRED key, and everything else is one command.

The context for these numbers

This is one cell of a broader study: the same protocol covers Chronos-Bolt, Moirai and Lag-Llama, and Laplace beats all four on the continuous split, with the native-density models closest. The rolling eight-baseline study, the price caveat where GARCH-t wins, and the full tables are on the papers page; slice the non-price benchmark yourself in the robustness explorer.