DESTformer: A Transformer Based on Explicit Seasonal–Trend Decomposition for Long-Term Series Forecasting
Source abstract
Abstract
Seasonal–trend-decomposed transformer has empowered long-term time series forecasting via capturing global temporal dependencies (e.g., period-based dependencies) in disentangled temporal patterns. However, existing methods design various auto-correlation or attention mechanisms in the seasonal view while ignoring the fine-grained temporal patterns in the trend view in the series decomposition component, which causes an information utilization bottleneck. To this end, a Transformer-based seasonal–trend decomposition methodology with a multi-scale attention mechanism in the trend view and a multi-view attention mechanism in the seasonal view is proposed, called DESTformer. Specifically, rather than utilizing the moving average operation in obtaining trend data, a frequency domain transform is first applied to extract seasonal (high-frequency) and trend (low-frequency) components, explicitly capturing different temporal patterns in both seasonal and trend views. For the trend component, a multi-scale attention mechanism is designed to capture fine-grained sub-trends under different receptive fields. For the seasonal component, instead of the frequency-only attention mechanism, a multi-view frequency domain (i.e., frequency, amplitude, and phase) attention mechanism is designed to enhance the ability to capture the complex periodic changes. Extensive experiments are conducted on six benchmark datasets covering five practical applications: energy, transportation, economics, weather, and disease. Compared to the state-of-the-art FEDformer, our model shows reduced MSE and MAE by averages of 6.5% and 3.7%, respectively. Such experimental results verify the effectiveness of our method and point out a new way towards handling trends and seasonal patterns in long-term time series forecasting tasks.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Time Series Analysis and Forecasting · Stock Market Forecasting Methods · Forecasting Techniques and Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Mechanical Engineering, Dalian University of Technology, Dalian 116024, China
School of Computer Science and Technology, Dalian University of Technology, Dalian 116024, China
Funding
National Defense Basic Scientific Research Program of China
References (39)
Show source-provided references
1. Zhou, H., Zhang, S., Peng, J., Zhang, S., Li, J., Xiong, H., and Zhang, W. (2021, January 2–9). Informer: Beyond efficient transformer for long sequence time-series forecasting. Proceedings of the AAAI Conference on Artificial Intelligence, Vancouver, BC, Canada.10.1609/aaai.v35i12.17325 ↗
2. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting
3. Zhou, T., Ma, Z., Wen, Q., Wang, X., Sun, L., and Jin, R. (2022, January 17–23). Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. Proceedings of the International Conference on Machine Learning, PMLR, Baltimore, MD, USA.
4. Guo, C., Li, D., and Chen, X. (2023). Unequal Interval Dynamic Traffic Flow Prediction with Singular Point Detection. Appl. Sci., 13.10.3390/app13158973 ↗
5. Han, L., Du, B., Sun, L., Fu, Y., Lv, Y., and Xiong, H. (April, January 13). Dynamic and multi-faceted spatio-temporal deep learning for traffic speed forecasting. Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, Virtual.
6. He, Z., Zhao, C., and Huang, Y. (2022). Multivariate Time Series Deep Spatiotemporal Forecasting with Graph Neural Network. Appl. Sci., 12.10.3390/app12115731 ↗
7. Qin, H., Ke, S., Yang, X., Xu, H., Zhan, X., and Zheng, Y. (2021, January 2–9). Robust spatio-temporal purchase prediction via deep meta learning. Proceedings of the AAAI Conference on Artificial Intelligence, Vancouver, BC, Canada.10.1609/aaai.v35i5.16556 ↗
8. Prediction of treatment medicines with dual adaptive sequential networks10.1109/TKDE.2021.3052992 ↗
9. Zhu, J., Tang, H., Zhang, L., Jin, B., Xu, Y., and Wei, X. (2023, January 25–28). A Global View-Guided Autoregressive Residual Network for Irregular Time Series Classification. Proceedings of the Pacific-Asia Conference on Knowledge Discovery and Data Mining, Osaka, Japan.10.1007/978-3-031-33383-5_23 ↗
10. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (2017). Attention is all you need. Adv. Neural Inf. Process. Syst., 30.
11. Kitaev, N., Kaiser, Ł., and Levskaya, A. (2020). Reformer: The efficient transformer. arXiv.
12. Du, D., Su, B., and Wei, Z. (2022). Preformer: Predictive transformer with multi-scale segment-wise correlations for long-term time series forecasting. arXiv.10.1109/ICASSP49357.2023.10096881 ↗
13. Wang, Z., Xu, X., Zhang, W., Trajcevski, G., Zhong, T., and Zhou, F. (December, January 28). Learning Latent Seasonal-Trend Representations for Time Series Forecasting. Proceedings of the Advances in Neural Information Processing Systems, New Orleans, LA, USA.
14. Some recent advances in forecasting and control
15. Box, G.E., Jenkins, G.M., Reinsel, G.C., and Ljung, G.M. (2015). Time Series Analysis: Forecasting and Control, John Wiley & Sons.
16. Woo, G., Liu, C., Sahoo, D., Kumar, A., and Hoi, S. (2022). Etsformer: Exponential smoothing transformers for time-series forecasting. arXiv.
17. Employing long short-term memory and Facebook prophet model in air temperature forecasting10.1080/03610918.2020.1854302 ↗
18. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv.
19. SpeechFormer++: A Hierarchical Efficient Framework for Paralinguistic Speech Processing10.1109/TASLP.2023.3235194 ↗
20. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2020). An image is worth 16 × 16 words: Transformers for image recognition at scale. arXiv.
21. Li, S., Jin, X., Xuan, Y., Zhou, X., Chen, W., Wang, Y.X., and Yan, X. (2019). Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. Adv. Neural Inf. Process. Syst., 32.
22. STL: A seasonal-trend decomposition
23. Jarrah, M., and Derbali, M. (2023). Predicting Saudi Stock Market Index by Using Multivariate Time Series Based on Deep Learning. Appl. Sci., 13.10.20944/preprints202306.1537.v1 ↗
24. A spatio-temporal decomposition based deep neural network for time series forecasting10.1016/j.asoc.2019.105963 ↗
25. Ju, J., and Liu, F.A. (2021). Multivariate time series data prediction based on att-lstm network. Appl. Sci., 11.10.3390/app11209373 ↗
26. Forecasting at scale10.1080/00031305.2017.1380080 ↗
27. Oreshkin, B.N., Carpov, D., Chapados, N., and Bengio, Y. (2019). N-BEATS: Neural basis expansion analysis for interpretable time series forecasting. arXiv.
28. Sen, R., Yu, H.F., and Dhillon, I.S. (2019). Think globally, act locally: A deep neural network approach to high-dimensional time series forecasting. Adv. Neural Inf. Process. Syst., 32.
29. Woo, G., Liu, C., Sahoo, D., Kumar, A., and Hoi, S. (2022). CoST: Contrastive learning of disentangled seasonal-trend representations for time series forecasting. arXiv.
30. Denoising nonlinear time series by adaptive filtering and wavelet shrinkage: A comparison10.1109/LSP.2009.2037773 ↗
31. Gao, J., Hu, J., and Tung, W.w. (2011). Facilitating joint chaos and fractal analysis of biosignals through nonlinear adaptive filtering. PLoS ONE, 6.10.1371/journal.pone.0024331 ↗
32. Generalized harmonic analysis10.1007/BF02546511 ↗
33. Automatic time series forecasting: The forecast package for R10.18637/jss.v027.i03 ↗
34. Bahdanau, D., Cho, K., and Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. arXiv.
35. Van der Maaten, L., and Hinton, G. (2008). Visualizing data using t-SNE. J. Mach. Learn. Res., 9.
36. Lai, G., Chang, W.C., Yang, Y., and Liu, H. (2018, January 8–12). Modeling long-and short-term temporal patterns with deep neural networks. Proceedings of the 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, Ann Arbor, MI, USA.10.1145/3209978.3210006 ↗
37. Ariyo, A.A., Adewumi, A.O., and Ayo, C.K. (2014, January 26–28). Stock Price Prediction Using the ARIMA Model. Proceedings of the 2014 UKSim-AMSS 16th International Conference on Computer Modelling and Simulation, Cambridge, UK.10.1109/UKSim.2014.67 ↗
38. Kingma, D.P., and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv.
39. Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., and Antiga, L. (2019). Pytorch: An imperative style, high-performance deep learning library. Adv. Neural Inf. Process. Syst., 32.