DeformTime: Capturing Variable Dependencies with Deformable Attention for Time Series Forecasting

Abstract

In multivariate time series (MTS) forecasting, existing state-of-the-art deep learning approaches tend to focus on autoregressive formulations and overlook the information within exogenous indicators. To address this limitation, we present DeformTime, a neural network architecture that attempts to capture correlated temporal patterns from the input space, and hence, improve forecasting accuracy. It deploys two core operations performed by deformable attention blocks (DABs): learning dependencies across variables from different time steps (variable DAB), and preserving temporal dependencies in data from previous time steps (temporal DAB). Input data transformation is explicitly designed to enhance learning from the deformed series of information while passing through a DAB. We conduct extensive experiments on 6 MTS data sets, using previously established benchmarks as well as challenging infectious disease modelling tasks with more exogenous variables. The results demonstrate that DeformTime improves accuracy against previous competitive methods across the vast majority of MTS forecasting tasks, reducing the mean absolute error by 10% on average. Notably, performance gains remain consistent across longer forecasting horizons.

Highlights

---

We propose DeformTime, a novel MTS forecasting model that better captures inter- and intra-variate dependencies at different temporal granularities. It comprises two Deformable Attention Blocks (DAB) which allow the model to adaptively focus on more informative neighbouring attributes. The below figure shows how different dependencies are established:

(a) The inter-variable dependency is established across different variables over time. (b) The intra-variable dependency focuses on the important information of the specific variable across time. Both dependencies are adaptively established w.r.t. the input.

Datasets

We conduct experiments on 6 real-world data sets. These include 3 established benchmarks from previously published papers, and 3 self-collected data sets focusing on disease rate modelling tasks. We provide further details in obtaining the datasets in our GitHub repository.

Citation

---

    
    @article{shu2024deformtime,
      author    = {Yuxuan Shu and Vasileios Lampos},
      title     = {{\textsc{DeformTime}: Capturing Variable Dependencies with 
                  Deformable Attention for Time Series Forecasting}},
      year      = {2024},
      journal   = {Preprint under review}
    }