What is the Impermanent benchmark?

Impermanent is a live benchmark for time series foundation models that tests temporal generalization. It uses a prequential protocol on continuously updating GitHub activity data, scoring models on forecasts they made before outcomes existed.

Why is Impermanent listed as an emerging benchmark?

The benchmark methodology and dataset are established, but the public machine-readable leaderboard feed is still stabilizing. TSFM.ai will publish automated rankings once the upstream source exposes a durable feed.

What is temporal contamination?

Temporal contamination occurs when benchmark test data exists at the time of model training, allowing models to overfit — intentionally or not — to the specific test split. Impermanent avoids this because future observations do not exist at evaluation time.

What data does Impermanent use?

Impermanent uses GitHub activity data from 400 repositories across 4 event types (issues, pull requests, pushes, stargazers) at 4 frequencies, producing 6,400 total time series.

Live·18 models · 13 cutoffs · 24 slices·Window 2026-05-31-00 → 2026-08-23-00

Impermanent Benchmark

Name: Impermanent Benchmark (TSFM.ai snapshot)
Creator: TimeCopilot

Impermanent is designed to test whether benchmark wins survive once real time passes. It scores forecasts sequentially on a continuously updating GitHub activity stream, so the benchmark reflects temporal drift and the fact that future observations do not exist at training time. This page scrapes the live Impermanent dashboard on every refresh so you can see how each model’s rank moves across the prequential window — including per-subdataset, per-frequency, and per-sparsity drift.

ZeroModel leads with an average MASE rank of 3.49 across the prequential window.

What this benchmark answers

Does model performance hold up as real time passes and the data distribution shifts?

Methodology

The benchmark uses a prequential protocol: models forecast before outcomes exist, scores accumulate over time, and rankings reflect sustained performance under live temporal change rather than one-off wins on a frozen split.

Prequential leaderboard

Models ranked by their sustained MASE rank across 13 weekly cutoffs from May 31 to Aug 23.

#	Model	MASE rank	CRPS rank	Scaled MASE	Scaled CRPS	Hosted
🥇	ZeroModel	3.494	1.185	0.247	0.246	—
🥈	AutoNHITS	6.488	4.595	1.142	2.178	—
🥉	AutoNBEATS	6.798	5.113	1.448	2.453	—
4	Toto	6.851	5.756	0.668	2.387	—
5	AutoPatchTST	7.649	5.107	1.324	2.151	—
6	TiRex	8.008	5.500	0.773	2.009	View model
7	Moirai	6.589	7.446	0.760	5.723	View family
8	Chronos	9.518	5.738	1.008	2.095	View family
9	TimesFM	7.738	8.315	0.683	3.718	View family
10	AutoTFT	9.321	6.982	7.982	14.76	—
11	SeasonalNaive	3.512	17.63	0.371	369.3	—
12	FlowState	12.27	9.232	2.613	6.350	View model
13	DynamicOptimizedTheta	8.077	13.55	0.796	167.2	—
14	AutoCES	9.571	12.56	0.971	149.4	—
15	AutoETS	13.75	14.17	10.35	168.8	—
16	AutoARIMA	14.79	14.51	11.92	163.6	—
17	Prophet	16.66	14.18	31.02	163.0	—
18	HistoricAverage	17.77	16.76	69.44	191.8	—

Per-slice performance over time

Pick a subdataset, frequency, and sparsity bucket to see how each model scored at every prequential cutoff. Lower is better for both metrics.

Metric

Subdataset

Frequency

Sparsity

Model	May 31	Jun 7	Jun 14	Jun 21	Jun 28	Jul 5	Jul 12	Jul 19	Jul 26	Aug 2	Aug 9	Aug 16	Aug 23
SeasonalNaive	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000
ZeroModel	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000
AutoPatchTST	0.267	0.175	0.036	0.153	0.028	0.044	0.009	0.067	0.007	0.027	0.037	0.028	0.011
Moirai	0.037	0.050	0.029	0.039	0.036	0.038	0.031	0.029	0.030	0.012	0.030	0.012	0.016
Toto	0.066	0.089	0.052	0.061	0.039	0.062	0.036	0.040	0.047	0.022	0.060	0.026	0.029
TimesFM	0.188	0.205	0.142	0.182	0.087	0.113	0.063	0.061	0.075	0.030	0.059	0.031	0.035
AutoNHITS	0.048	0.028	0.057	0.032	0.038	0.025	0.022	0.032	0.028	0.008	0.037	0.016	0.048
AutoNBEATS	0.161	0.021	0.038	0.066	0.029	0.024	0.012	0.033	0.058	0.021	0.054	0.025	0.061
TiRex	0.126	—	—	0.166	0.119	0.153	0.107	0.113	0.136	0.064	0.155	0.076	0.084
AutoTFT	0.112	0.218	0.016	0.038	0.605	0.198	0.446	0.196	0.174	0.163	0.107	0.388	0.217
DynamicOptimizedTheta	2.064	1.747	1.358	1.088	0.457	0.495	0.314	0.205	0.303	0.124	0.316	0.217	0.243
AutoCES	2.210	2.093	1.797	1.634	0.960	1.395	0.737	0.525	0.548	0.214	0.488	0.260	0.268
Chronos	0.443	0.482	0.463	0.678	0.400	0.699	0.452	0.485	0.568	0.221	0.561	0.254	0.341
FlowState	1.180	1.267	0.832	1.060	0.711	1.181	0.694	0.863	1.106	0.508	1.329	0.629	0.679
AutoETS	2.107	2.148	2.067	2.193	1.397	2.156	1.242	0.979	1.262	0.550	1.188	0.618	0.717
AutoARIMA	2.332	2.253	2.015	2.719	2.346	4.276	2.686	3.154	4.707	2.129	6.174	2.573	3.379
Prophet	4.478	4.450	5.651	8.844	6.082	11.20	8.095	8.920	11.59	5.244	14.28	6.855	8.236
HistoricAverage	15.64	17.06	24.81	37.55	25.62	45.95	32.37	36.00	45.63	19.05	51.63	22.19	27.65

13 prequential cutoffs for Issues opened · Daily · Low sparsity. Values are scaled MASE against the seasonal naive baseline; lower is better.

18 models · 4 subdatasets · 2 frequencies · 3 sparsity buckets · Scraped live from impermanent.timecopilot.dev. Refreshes every 12 hours.Last refreshed Sep 11, 2026, 5:07 PM

The temporal contamination problem

Static benchmarks freeze a test split at one point in time. Once that split is public, model authors can overfit to it — intentionally or not — and benchmark scores stop reflecting real-world performance. Impermanent sidesteps this by scoring forecasts before outcomes exist: the future data literally does not exist at evaluation time, so there is no split to leak.

How the prequential protocol works

Models submit forecasts on a rolling basis against a continuously updating stream of GitHub activity data (issues opened, PRs merged, pushes, stargazers) across 400 repositories and 4 frequencies. Scores accumulate over time, so a model that performs well early but degrades under distribution shift will see its aggregate ranking fall. This makes Impermanent a measure of sustained forecasting ability, not one-off performance.

How to interpret it

—Impermanent is the right benchmark when you care about temporal drift, not just a frozen snapshot.
—A strong live benchmark should tell you whether early leaderboard wins persist under new data.
—Because the public interface is still settling, treat the methodology as the durable asset and the rows as an upcoming integration.

Frequently asked questions

What is the Impermanent benchmark?: Impermanent is a live benchmark for time series foundation models that tests temporal generalization. It uses a prequential protocol on continuously updating GitHub activity data, scoring models on forecasts they made before outcomes existed.
Why is Impermanent listed as an emerging benchmark?: The benchmark methodology and dataset are established, but the public machine-readable leaderboard feed is still stabilizing. TSFM.ai will publish automated rankings once the upstream source exposes a durable feed.
What is temporal contamination?: Temporal contamination occurs when benchmark test data exists at the time of model training, allowing models to overfit — intentionally or not — to the specific test split. Impermanent avoids this because future observations do not exist at evaluation time.
What data does Impermanent use?: Impermanent uses GitHub activity data from 400 repositories across 4 event types (issues, pull requests, pushes, stargazers) at 4 frequencies, producing 6,400 total time series.

Title	Date
Impermanent: The First Live Benchmark for Temporal Generalization in TSFMs	2026-03-14	Read →
The Challenges of Benchmarking TSFMs	2024-11-05	Read →
Zero-Shot Forecasting: Why It Matters	2024-04-12	Read →