How well does a model generalize to unseen real-world forecasting tasks?
Pretrained models run zero-shot under the benchmark's leakage policy. Task-specific models (classical baselines, fine-tuned models) may use each task's training split. All models are ranked by MASE-based Skill Score relative to a seasonal naive baseline; Win Rate measures pairwise wins against all included models, with half credit for ties.
Which models stay strong across heterogeneous datasets and probabilistic settings?
Models are scored on grouped benchmark slices and ranked by average rank, with Weighted Quantile Loss providing a secondary read on probabilistic accuracy.
How do models behave on observability telemetry instead of academic datasets?
Models are ranked on real observability time series using CRPS for probabilistic quality and MASE for point accuracy, aggregated across production telemetry collected from real infrastructure workloads.
Does model performance hold up as real time passes and the data distribution shifts?
The benchmark uses a prequential protocol: models forecast before outcomes exist, scores accumulate over time, and rankings reflect sustained performance under live temporal change rather than one-off wins on a frozen split.
Which multimodal models can reason about anomalies, timing, magnitude, and cross-series structure in production telemetry?
750 multiple-choice QA pairs built from 142 real Datadog observability time series and 63 production incidents, spanning eight question categories grouped into three difficulty tiers. Models receive a templated question, a metric description, and the plotted time series, and are scored on Accuracy and macro-F1 against expert-reviewed answers.
TIMESee official metric selection How do models behave across realistic tasks and different temporal patterns?
The benchmark evaluates 12 foundation models across 98 tasks, combining dataset quality checks with forecasting-relevant pattern analysis. Consult the official leaderboard for current submissions and selected metrics.
Does a model remain strong across easy and difficult forecasting regimes?
The study compares statistical, deep learning and foundation-model approaches across eight regimes and multiple context lengths. Results are published study comparisons, rather than a continuously updated competition ranking.