What is the Impermanent benchmark?

Impermanent is a live benchmark for time series foundation models that tests temporal generalization. It uses a prequential protocol on continuously updating GitHub activity data, scoring models on forecasts they made before outcomes existed.

Why is Impermanent listed as an emerging benchmark?

The benchmark methodology and dataset are established, but the public machine-readable leaderboard feed is still stabilizing. TSFM.ai will publish automated rankings once the upstream source exposes a durable feed.

What is temporal contamination?

Temporal contamination occurs when benchmark test data exists at the time of model training, allowing models to overfit — intentionally or not — to the specific test split. Impermanent avoids this because future observations do not exist at evaluation time.

What data does Impermanent use?

Impermanent uses GitHub activity data from 400 repositories across 4 event types (issues, pull requests, pushes, stargazers) at 4 frequencies, producing 6,400 total time series.

Live·18 models · 13 cutoffs · 24 slices·Window 2026-04-19-00 → 2026-07-12-00

Impermanent Benchmark

Name: Impermanent Benchmark (TSFM.ai snapshot)
Creator: TimeCopilot

Impermanent is designed to test whether benchmark wins survive once real time passes. It scores forecasts sequentially on a continuously updating GitHub activity stream, so the benchmark reflects temporal drift and the fact that future observations do not exist at training time. This page scrapes the live Impermanent dashboard on every refresh so you can see how each model’s rank moves across the prequential window — including per-subdataset, per-frequency, and per-sparsity drift.

ZeroModel leads with an average MASE rank of 5.46 across the prequential window.

What this benchmark answers

Does model performance hold up as real time passes and the data distribution shifts?

Methodology

The benchmark uses a prequential protocol: models forecast before outcomes exist, scores accumulate over time, and rankings reflect sustained performance under live temporal change rather than one-off wins on a frozen split.

Prequential leaderboard

Models ranked by their sustained MASE rank across 13 weekly cutoffs from Apr 19 to Jul 12.

#	Model	MASE rank	CRPS rank	Scaled MASE	Scaled CRPS	Hosted
🥇	ZeroModel	5.463	1.750	0.411	0.410	—
🥈	AutoNBEATS	5.900	4.987	0.459	1.252	—
🥉	AutoNHITS	6.225	5.138	0.456	1.241	—
4	AutoPatchTST	7.287	5.213	0.574	1.589	—
5	TiRex	6.907	5.971	0.531	1.500	View model
6	Chronos	7.750	5.333	0.600	1.438	View family
7	Toto	7.479	5.675	0.714	1.655	—
8	AutoTFT	7.150	6.287	1.571	3.041	—
9	Moirai	7.246	7.550	0.657	2.018	View family
10	TimesFM	7.521	8.279	0.586	2.119	View family
11	FlowState	11.08	9.454	0.998	2.704	View model
12	SeasonalNaive	5.075	17.38	0.477	71.31	—
13	AutoCES	10.91	12.95	0.854	32.83	—
14	DynamicOptimizedTheta	10.34	13.58	0.837	39.68	—
15	AutoETS	14.05	13.82	2.420	39.67	—
16	AutoARIMA	14.65	14.11	3.334	38.10	—
17	Prophet	16.54	14.69	7.747	38.47	—
18	HistoricAverage	17.75	17.02	18.18	46.38	—

Per-slice performance over time

Pick a subdataset, frequency, and sparsity bucket to see how each model scored at every prequential cutoff. Lower is better for both metrics.

Metric

Subdataset

Frequency

Sparsity

Model	Apr 19	Apr 26	May 3	May 10	May 17	May 24	May 31	Jun 7	Jun 14	Jun 21	Jun 28	Jul 5	Jul 12
SeasonalNaive	1.276	1.521	1.363	1.698	1.170	1.283	0.000	0.000	0.000	0.000	0.000	0.000	0.000
ZeroModel	1.000	1.000	1.000	1.000	0.880	0.259	0.000	0.000	0.000	0.000	0.000	0.000	0.000
AutoPatchTST	1.063	1.377	1.592	1.736	1.171	1.402	0.267	0.175	0.036	0.153	0.028	0.044	0.009
AutoNBEATS	1.171	1.222	1.211	1.223	1.033	0.959	0.161	0.021	0.038	0.066	0.029	0.024	0.012
AutoNHITS	1.110	1.316	1.316	1.219	1.053	0.593	0.048	0.028	0.057	0.032	0.038	0.025	0.022
Moirai	1.203	1.066	1.304	1.453	1.116	0.874	0.037	0.050	0.029	0.039	0.036	0.038	0.031
Toto	3.763	4.227	1.075	1.224	0.934	0.662	0.066	0.089	0.052	0.061	0.039	0.062	0.036
TimesFM	1.294	1.204	1.333	1.417	1.114	0.891	0.188	0.205	0.142	0.182	0.087	0.113	0.063
TiRex	1.248	1.152	1.286	1.304	1.005	0.798	0.126	—	—	0.166	0.119	0.153	0.107
DynamicOptimizedTheta	1.783	1.844	2.161	2.424	2.078	1.765	2.064	1.747	1.358	1.088	0.457	0.495	0.314
AutoTFT	1.027	1.069	1.028	1.029	0.930	0.442	0.112	0.218	0.016	0.038	0.605	0.198	0.446
Chronos	1.254	1.236	1.313	1.350	1.076	0.970	0.443	0.482	0.463	0.678	0.400	0.699	0.452
FlowState	1.571	1.536	1.765	1.975	1.837	1.496	1.180	1.267	0.832	1.060	0.711	1.181	0.694
AutoCES	1.868	1.961	2.183	2.404	2.247	1.957	2.210	2.093	1.797	1.634	0.960	1.395	0.737
AutoETS	1.962	1.933	2.307	2.385	2.276	1.940	2.107	2.148	2.067	2.193	1.397	2.156	1.242
AutoARIMA	1.912	2.011	2.334	2.408	2.321	2.042	2.332	2.253	2.015	2.719	2.346	4.276	2.686
Prophet	2.888	2.962	3.616	4.035	3.999	3.460	4.478	4.450	5.651	8.844	6.082	11.20	8.095
HistoricAverage	7.513	7.851	10.25	11.08	12.77	10.96	15.64	17.06	24.81	37.55	25.62	45.95	32.37

13 prequential cutoffs for Issues opened · Daily · Low sparsity. Values are scaled MASE against the seasonal naive baseline; lower is better.

18 models · 4 subdatasets · 2 frequencies · 3 sparsity buckets · Scraped live from impermanent.timecopilot.dev. Refreshes every 12 hours.Last refreshed Jul 29, 2026, 6:05 AM

The temporal contamination problem

Static benchmarks freeze a test split at one point in time. Once that split is public, model authors can overfit to it — intentionally or not — and benchmark scores stop reflecting real-world performance. Impermanent sidesteps this by scoring forecasts before outcomes exist: the future data literally does not exist at evaluation time, so there is no split to leak.

How the prequential protocol works

Models submit forecasts on a rolling basis against a continuously updating stream of GitHub activity data (issues opened, PRs merged, pushes, stargazers) across 400 repositories and 4 frequencies. Scores accumulate over time, so a model that performs well early but degrades under distribution shift will see its aggregate ranking fall. This makes Impermanent a measure of sustained forecasting ability, not one-off performance.

How to interpret it

—Impermanent is the right benchmark when you care about temporal drift, not just a frozen snapshot.
—A strong live benchmark should tell you whether early leaderboard wins persist under new data.
—Because the public interface is still settling, treat the methodology as the durable asset and the rows as an upcoming integration.

Frequently asked questions

What is the Impermanent benchmark?: Impermanent is a live benchmark for time series foundation models that tests temporal generalization. It uses a prequential protocol on continuously updating GitHub activity data, scoring models on forecasts they made before outcomes existed.
Why is Impermanent listed as an emerging benchmark?: The benchmark methodology and dataset are established, but the public machine-readable leaderboard feed is still stabilizing. TSFM.ai will publish automated rankings once the upstream source exposes a durable feed.
What is temporal contamination?: Temporal contamination occurs when benchmark test data exists at the time of model training, allowing models to overfit — intentionally or not — to the specific test split. Impermanent avoids this because future observations do not exist at evaluation time.
What data does Impermanent use?: Impermanent uses GitHub activity data from 400 repositories across 4 event types (issues, pull requests, pushes, stargazers) at 4 frequencies, producing 6,400 total time series.

Title	Date
Impermanent: The First Live Benchmark for Temporal Generalization in TSFMs	2026-03-14	Read →
The Challenges of Benchmarking TSFMs	2024-11-05	Read →
Zero-Shot Forecasting: Why It Matters	2024-04-12	Read →