Foundation models for time series applications: An overview of the leading technologies

What are foundation models for time series applications?

Foundation models are advanced machine learning models that are trained on large and diverse data sets and can then be applied to a variety of specific tasks. In the field of time series analysis, this means that such a model is trained on extensive time series data and can then be used for prediction, classification or pattern recognition in specific time series applications. Foundation models are an answer to the challenges that traditional models often cannot overcome in time series analysis. These models aim to reduce the need for extensive customisation and fine-tuning for each individual application. Through prior training on large and diverse data sets, these models develop a deep understanding of the underlying patterns and structures in the data so that they can be applied to new, never-before-seen data with a minimum of additional training.

The advantages of these models for time series analysis are their flexibility, scalability and customisability. They can recognise complex patterns in a variety of data and efficiently process both small and large data sets. Their ability to continuously learn from new data and adapt to changing conditions makes them particularly valuable in dynamic environments. However, LLMs also have disadvantages: they are considered "black boxes", which means that their internal functioning is often not as transparent and interpretable as that of classic statistical models. This can limit the traceability and explainability of the results, which can be problematic in some areas of application.

A concrete example of the application of foundation models is the "MOMENT" model (see MOMENT: A Family of Open Time-series Foundation Models(arxiv.org)). This model was trained with a variety of time series data from different domains such as health, finance and technology. The goal was to create a model that can be used effectively for various tasks, including long- and short-term forecasting, anomaly detection and classification.

Variety of foundation models

The variety of foundation models for time series applications is impressive. There are a large number of models based on different data types and methodologies. A comprehensive categorisation system shows how these models can be classified:

• Transformer-based models: these models use self-learning mechanisms to recognise complex patterns.
• Non-transformer-based models: These include models such as MLPs, RNNs and CNNs, which are also suitable for time series analysis.
• Diffusion-based models: These use diffusion techniques for modelling and predicting time series.
• Space-time models: These combine spatial and temporal data for prediction and analysis.
• Other specialised models: These include models for specific applications such as mobility, climate, events and more.

This variety shows how comprehensive and adaptable foundation models can be for different use cases in time series analysis. Each model offers specific advantages and specialisations that are tailored to the respective application.

What is behind it?

Foundation models often use complex neural networks such as transformer architectures or large language models (LLMs) that were originally developed for text or image processing. These models are reprogrammed or adapted to meet the characteristics and challenges of time series analysis. A central concept here is the ability of the models to learn new tasks on the basis of prior knowledge and small amounts of data (few-shot learning) or even without explicit examples (zero-shot learning).

Technical details and customisations

• Transformer architectures: These models are particularly good at capturing long-term dependencies in data. They use self-learning mechanisms to determine which parts of the input data are relevant and can therefore recognise both short- and long-term patterns. One example is the "TimesFM" model (see A decoder-only foundation model for time-series forecasting (arxiv.org)), which was developed specifically for time-series forecasting and uses techniques such as patch-masking and spectral normalisation to pre-process the data and improve prediction accuracy.
• Large language models (LLMs): These models, such as GPT-3 or GPT-4, were originally developed for processing text data, but have proven to be very flexible and can also be used for time series data with appropriate adaptations. A good example of this is the "Time-LLM" (see 2310.01728 (arxiv.org)), which uses a reprogramming framework to transform time series data into a form that can be processed by language models. Using techniques such as "Prompt-as-Prefix", the model can improve predictive ability while keeping the model structure intact.

Adaptive and multilevel training

Various techniques are used to increase the versatility and efficiency of these models:

• Adaptive patching: this involves dynamically adjusting the length of data segments presented to the model to better deal with different resolutions and lengths of time series data. This technique has been successfully used in the "TinyTimeMixer" model (see2401.03955 (arxiv.org)), which uses adaptive patching and data augmentation to improve its prediction accuracy while minimising model size and computational effort.
• Multilevel modelling: This technique is used to capture and integrate different levels of data structure. In "TinyTimeMixer", this is achieved by splitting the model into several levels, each of which processes and combines specific aspects of the data to provide a comprehensive prediction.

Comparison of the models

Conclusion

Foundation models are revolutionising the way we perform time series analyses. Their flexibility, accuracy and efficiency make them an indispensable tool in modern data analysis. The models presented show how powerful and versatile these approaches are. There are still many open questions and challenges to be explored in this exciting field. We invite you to get to grips with these technologies and develop innovative solutions.

If you would like to learn more about the application of foundation models in time series analysis or need support in implementing these technologies, please contact us. Together we can realise the potential of these advanced models and take your data analysis to the next level.

Get in touch now

Would you like to find out more about exciting topics from the world of adesso? Then take a look at our previous blog posts.