What is the Impermanent benchmark?

Impermanent is a live benchmark for time series foundation models that tests temporal generalization. It uses a prequential protocol on continuously updating GitHub activity data, scoring models on forecasts they made before outcomes existed.

Why is Impermanent listed as an emerging benchmark?

The benchmark methodology and dataset are established, but the public machine-readable leaderboard feed is still stabilizing. TSFM.ai will publish automated rankings once the upstream source exposes a durable feed.

What is temporal contamination?

Temporal contamination occurs when benchmark test data exists at the time of model training, allowing models to overfit — intentionally or not — to the specific test split. Impermanent avoids this because future observations do not exist at evaluation time.

What data does Impermanent use?

Impermanent uses GitHub activity data from 400 repositories across 4 event types (issues, pull requests, pushes, stargazers) at 4 frequencies, producing 6,400 total time series.

Upstream results·18 models · 13 cutoffs · 12 slices·Window 2026-07-12-00 → 2026-10-04-00

Impermanent Benchmark

Name: Impermanent Benchmark (TSFM.ai snapshot)
Creator: TimeCopilot

Impermanent is designed to test whether benchmark wins survive once real time passes. It scores forecasts sequentially on a continuously updating GitHub activity stream, so the benchmark reflects temporal drift and the fact that future observations do not exist at training time. This page scrapes the live Impermanent dashboard on every refresh so you can see how each model’s rank moves across the prequential window — including per-subdataset, per-frequency, and per-sparsity drift.

ZeroModel leads with an average MASE rank of 3.24 across the prequential window.

What this benchmark answers

Does model performance hold up as real time passes and the data distribution shifts?

Methodology

The benchmark uses a prequential protocol: models forecast before outcomes exist, scores accumulate over time, and rankings reflect sustained performance under live temporal change rather than one-off wins on a frozen split.

Prequential leaderboard

Models ranked by their sustained MASE rank across 13 weekly cutoffs from Jul 12 to Oct 4.

#	Model	MASE rank	CRPS rank	Scaled MASE	Scaled CRPS	Directory
🥇	ZeroModel	3.237	1.186	0.267	0.267	—
🥈	Moirai	5.641	6.891	0.576	5.369	View family
🥉	AutoNHITS	7.391	5.218	1.283	2.067	—
4	TiRex	7.372	5.494	0.666	1.731	View model
5	AutoNBEATS	7.551	5.942	1.495	2.420	—
6	Toto	7.391	6.455	0.634	2.066	—
7	AutoPatchTST	8.635	5.372	1.704	9.172	—
8	Chronos	9.455	5.788	0.909	1.809	View family
9	TimesFM	7.603	7.923	0.604	3.068	View family
10	AutoTFT	9.994	7.404	8.935	14.60	—
11	FlowState	12.13	8.442	2.319	5.411	View model
12	SeasonalNaive	3.667	17.93	0.349	343.3	—
13	DynamicOptimizedTheta	8.096	13.76	0.772	150.2	—
14	AutoCES	9.590	13.00	0.860	137.2	—
15	AutoETS	13.82	14.47	10.58	151.3	—
16	AutoARIMA	14.62	14.68	11.97	147.5	—
17	Prophet	16.85	14.21	29.32	146.9	—
18	HistoricAverage	17.95	16.83	63.45	173.2	—

Per-slice performance over time

Pick a subdataset, frequency, and sparsity bucket to see how each model scored at every prequential cutoff. Lower is better for both metrics.

Metric

Subdataset

Frequency

Sparsity

Model	Jul 12	Jul 19	Jul 26	Aug 2	Aug 9	Aug 16	Aug 23	Aug 30	Sep 6	Sep 13	Sep 20	Sep 27	Oct 4
SeasonalNaive	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.961	1.425	0.278	0.000	0.000	0.000
ZeroModel	0.000	0.000	0.000	0.000	0.000	0.000	0.000	1.000	1.000	0.000	0.000	0.000	0.000
AutoNHITS	0.022	0.032	0.028	0.008	0.037	0.016	0.048	0.984	2.510	0.732	0.020	0.019	0.005
Moirai	0.031	0.029	0.030	0.012	0.030	0.012	0.016	0.986	1.023	0.008	0.007	0.007	0.009
Toto	0.036	0.040	0.047	0.022	0.060	0.026	0.029	0.972	1.072	0.045	0.028	0.022	0.033
TiRex	0.107	0.113	0.136	0.064	0.155	0.076	0.084	0.983	1.040	0.048	0.043	0.037	0.059
AutoNBEATS	0.012	0.033	0.058	0.021	0.054	0.025	0.061	0.970	1.079	0.518	0.006	0.024	0.071
AutoPatchTST	0.009	0.067	0.007	0.027	0.037	0.028	0.011	1.012	1.339	0.759	0.024	0.048	0.097
AutoTFT	0.446	0.196	0.174	0.163	0.107	0.388	0.217	0.993	1.130	0.175	0.149	0.188	0.168
TimesFM	0.063	0.061	0.075	0.030	0.059	0.031	0.035	0.998	1.106	0.171	0.105	0.108	0.169
Chronos	0.452	0.485	0.568	0.221	0.561	0.254	0.341	1.025	1.060	0.221	0.176	0.132	0.185
FlowState	0.694	0.863	1.106	0.508	1.329	0.629	0.679	1.085	1.360	0.894	0.200	0.145	0.205
AutoCES	0.737	0.525	0.548	0.214	0.488	0.260	0.268	1.051	1.415	1.598	1.256	0.912	1.403
DynamicOptimizedTheta	0.314	0.205	0.303	0.124	0.316	0.217	0.243	1.014	1.574	1.937	1.665	0.960	1.465
AutoETS	1.242	0.979	1.262	0.550	1.188	0.618	0.717	1.213	1.429	1.897	1.893	1.192	1.989
AutoARIMA	2.686	3.154	4.707	2.129	6.174	2.573	3.379	1.653	1.687	1.927	1.859	1.464	2.379
Prophet	8.095	8.920	11.59	5.244	14.28	6.855	8.236	2.294	2.430	2.613	3.453	2.776	4.835
HistoricAverage	32.37	36.00	45.63	19.05	51.63	22.19	27.65	4.011	4.556	7.097	10.45	7.235	14.07

13 prequential cutoffs for Issues opened · Daily · Low sparsity. Values are scaled MASE against the seasonal naive baseline; lower is better.

18 models · 4 subdatasets · 1 frequencies · 3 sparsity buckets · Scraped live from impermanent.timecopilot.dev. Sources cached up to 12 hours; refreshed on visits.Last refreshed Oct 27, 2026, 7:08 PM

The temporal contamination problem

Static benchmarks freeze a test split at one point in time. Once that split is public, model authors can overfit to it — intentionally or not — and benchmark scores stop reflecting real-world performance. Impermanent sidesteps this by scoring forecasts before outcomes exist: the future data literally does not exist at evaluation time, so there is no split to leak.

How the prequential protocol works

Models submit forecasts on a rolling basis against a continuously updating stream of GitHub activity data (issues opened, PRs merged, pushes, stargazers) across 400 repositories and 4 frequencies. Scores accumulate over time, so a model that performs well early but degrades under distribution shift will see its aggregate ranking fall. This makes Impermanent a measure of sustained forecasting ability, not one-off performance.

How to interpret it

—Impermanent is the right benchmark when you care about temporal drift, not just a frozen snapshot.
—A strong live benchmark should tell you whether early leaderboard wins persist under new data.
—Overview ordering follows the upstream summary’s mean of MASE and CRPS ranks. Raw scaled errors and per-cell ranks answer different questions.

Frequently asked questions

What is the Impermanent benchmark?: Impermanent is a live benchmark for time series foundation models that tests temporal generalization. It uses a prequential protocol on continuously updating GitHub activity data, scoring models on forecasts they made before outcomes existed.
Why is Impermanent listed as an emerging benchmark?: The benchmark methodology and dataset are established, but the public machine-readable leaderboard feed is still stabilizing. TSFM.ai will publish automated rankings once the upstream source exposes a durable feed.
What is temporal contamination?: Temporal contamination occurs when benchmark test data exists at the time of model training, allowing models to overfit — intentionally or not — to the specific test split. Impermanent avoids this because future observations do not exist at evaluation time.
What data does Impermanent use?: Impermanent uses GitHub activity data from 400 repositories across 4 event types (issues, pull requests, pushes, stargazers) at 4 frequencies, producing 6,400 total time series.

Title	Date
Impermanent: The First Live Benchmark for Temporal Generalization in TSFMs	2026-03-14	Read →
The Challenges of Benchmarking TSFMs	2024-11-05	Read →
Zero-Shot Forecasting: Why It Matters	2024-04-12	Read →