Benchmarks
Independent leaderboards for time series foundation models, refreshed automatically from upstream sources. Each benchmark answers a different forecasting question — pick the one that matches your workload.
FEV Bench: Failed to fetch benchmark source (404)
How well does a model generalize to unseen real-world forecasting tasks?
Pretrained models run zero-shot under the benchmark's leakage policy. Task-specific models (classical baselines, fine-tuned models) may use each task's training split. All models are ranked by MASE-based Skill Score relative to a seasonal naive baseline; Win Rate shows how often a model leads individual task rankings.
Which models stay strong across heterogeneous datasets and probabilistic settings?
Models are scored on grouped benchmark slices and ranked by average rank, with Weighted Quantile Loss providing a secondary read on probabilistic accuracy.
How do models behave on observability telemetry instead of academic datasets?
Models are ranked on real observability time series using CRPS for probabilistic quality and MASE for point accuracy, aggregated across production telemetry collected from real infrastructure workloads.
Does model performance hold up as real time passes and the data distribution shifts?
The benchmark uses a prequential protocol: models forecast before outcomes exist, scores accumulate over time, and rankings reflect sustained performance under live temporal change rather than one-off wins on a frozen split.
Which multimodal models can reason about anomalies, timing, magnitude, and cross-series structure in production telemetry?
750 multiple-choice QA pairs built from 142 real Datadog observability time series and 63 production incidents, spanning eight question categories grouped into three difficulty tiers. Models receive a templated question, a metric description, and the plotted time series, and are scored on Accuracy and macro-F1 against expert-reviewed answers.
2026-02-10
2024-04-12
2024-11-05
2026-02-15
2025-05-08
2026-02-20