How TSFMs work

Quantizes continuous time series values into discrete token bins and processes them through a T5-style encoder-decoder transformer. The encoder reads the full context window; the decoder autoregressively generates future tokens which are mapped back to continuous values. Naturally produces probabilistic outputs by sampling multiple trajectories.

Groups consecutive time points into patches (similar to Vision Transformer patches) and feeds them as tokens to a decoder-only transformer. The model autoregressively predicts the next patch of values. Patching reduces sequence length and computational cost while preserving local structure.

Instead of generating one token at a time, predicts the entire forecast horizon in a single forward pass. The model maps the input context directly to all future steps simultaneously. This eliminates the autoregressive bottleneck and achieves 3-5x faster inference.

Uses a masked encoder architecture with Any-Variate Attention — a mechanism that handles arbitrary numbers of input variates without assuming channel independence. Each variate is treated as a separate token sequence, and cross-variate attention captures dependencies between them.

Routes inputs through a sparse set of expert subnetworks. Only a fraction of the total parameters are active for any given input, selected by a learned routing mechanism. Different experts specialize in different temporal patterns or domains, enabling strong transfer without proportional compute cost.

Replaces attention with MLP-based mixing layers that alternate between mixing across time steps and mixing across features. Extremely parameter-efficient — achieves competitive accuracy with roughly 1M parameters compared to hundreds of millions for transformer-based models.

Generates forecasts by iteratively denoising from random noise into a plausible future trajectory. Flow-matching variants use a continuous-time framework that requires fewer denoising steps than standard diffusion. Produces full predictive distributions by generating multiple samples.

Takes a pre-trained large language model (e.g., LLaMA-7B) and reprograms it for time series by converting series data into text-like token sequences via Prompt-as-Prefix. The LLM's pre-trained reasoning capabilities are repurposed for temporal pattern recognition without retraining most parameters.