Risk-Aware Multimodal Sensing Network with Asynchronous Temporal Alignment and Predictive Uncertainty Estimation
Source abstract
Abstract
Financial markets are increasingly shaped by heterogeneous information sources, including trading behaviors, order-book dynamics, textual events, investor sentiment, and macroeconomic conditions. These signals are often asynchronous, noisy, partially missing, and associated with different degrees of reliability, which makes trustworthy financial risk early warning challenging. To address these issues, this study proposes an uncertainty-aware multimodal financial sensing network, termed UAMF-Net. The model treats price series, trading volume, order books, news texts, investor sentiment, and macroeconomic variables as financial sensing signals and integrates them through three task-oriented modules. First, the asynchronous multimodal temporal alignment module uses absolute time encoding, relative interval modeling, event-lag representation, temporal gating, and target-time-guided cross-scale attention to align cross-frequency financial signals according to their relevance to the prediction time. Second, the risk-aware multimodal soft fusion module estimates modality-level risk contribution and signal reliability by combining fuzzy risk membership, modality confidence weights, and cross-modal consistency constraints. Third, the uncertainty-aware risk early warning module adopts evidential learning to generate nonnegative class evidence, derive risk-category probabilities from Dirichlet parameters, estimate predictive uncertainty from total evidence strength, and jointly predict continuous risk intensity. Experimental results show that UAMF-Net achieves the best overall performance, with Accuracy, Precision, Recall, F1-score, Macro-F1, ROC-AUC, and PR-AUC reaching 0.882, 0.864, 0.849, 0.856, 0.839, 0.941, and 0.824, respectively, while ECE and Brier score are reduced to 0.037 and 0.096. Under severe temporal asynchrony, UAMF-Net maintains an Accuracy of 0.849, a Macro-F1 of 0.797, and a PR-AUC of 0.774. Under the missing multiple modalities setting, it achieves an Accuracy of 0.842 and a Macro-F1 of 0.788. The uncertainty analysis further shows that Risk Precision@90% reaches 0.889. Validation on FNSPID, Daily News, and StockEmotions also confirms its generalization ability across public financial benchmarks. These results indicate that UAMF-Net improves financial risk early warning by jointly modeling temporal asynchrony, modality reliability, and predictive uncertainty.
Source: crossref · Quality: baseline
Affiliations
National School of Development, Peking University, Beijing 100871, China
China Agricultural University, Beijing 100084, China
China University of Political Science and Law, Beijing 102249, China
Renmin University of China, Beijing 100872, China
Funding
National Natural Science Foundation of China
References (74)
Show source-provided references
1. Ang, G., and Lim, E.P. (2022). Guided attention multimodal multitask financial forecasting with inter-company relationships and global and local news. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics.10.18653/v1/2022.acl-long.437 ↗
2. Pei, Y., Cartlidge, J., Mandal, A., Gold, D., Marcilio, E., and Mazzon, R. (2025). Cross-modal temporal fusion for financial market forecasting. arXiv.10.3233/FAIA251474 ↗
3. Cao, Y., Chen, Z., Kumar, P., Pei, Q., Yu, Y., Li, H., Dimino, F., Ausiello, L., Subbalakshmi, K., and Ndiaye, P.M. (2025). RiskLabs: Predicting financial risk using large language model based on multimodal and multi-sources data. Proceedings of the 2025 IEEE International Conference on Data Mining Workshops (ICDMW), IEEE.10.1109/ICDMW69685.2025.00107 ↗
4. Luo, J., Kou, Z., Yang, L., Luo, X., Huang, J., Xiao, Z., Peng, J., Liu, C., Ji, J., and Liu, X. (2025). Finmme: Benchmark dataset for financial multi-modal reasoning evaluation. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics.10.18653/v1/2025.acl-long.1426 ↗
5. Jiang, Y., Ning, K., Pan, Z., Shen, X., Ni, J., Yu, W., and Song, D. (2025). Multi-Modal Time Series Analysis: A Tutorial and Survey. Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining, ACM.10.1145/3711896.3736567 ↗
6. Disentangling multi-modal fusion for financial distress prediction with multi-level attention mechanisms10.1016/j.asoc.2026.115085 ↗
7. Liu, Z., Zhang, X., Yang, K., Xie, Q., Huang, J., and Ananiadou, S. (2025). FMDLlama: Financial misinformation detection based on large language models. WWW ’25: Companion Proceedings of the ACM on Web Conference 2025, ACM.10.1145/3701716.3715599 ↗
8. Large language models: A survey of their development, capabilities, and applications10.1007/s10115-024-02310-4 ↗
9. Self-attention fusion and adaptive continual updating for multimodal federated learning with heterogeneous data10.1016/j.neunet.2025.107345 ↗
10. Fraud detection by integrating multisource heterogeneous presence-only data10.1287/ijoc.2023.0366 ↗
11. Kong, X., Shuai, S., Wang, H., Shen, G., and Xia, F. (2025). Dual-view anomaly detection in heterogeneous information networks with hierarchical neighborhood fusion. ACM Trans. Knowl. Discov. Data, 19.10.1145/3767156 ↗
12. Drinkall, F., Pierrehumbert, J., and Zohren, S. (2025). Forecasting credit ratings: A case study where traditional methods outperform generative llms. Proceedings of the Joint Workshop of the 9th Financial Technology and Natural Language Processing (FinNLP), the 6th Financial Narrative Processing (FNP), and the 1st Workshop on Large Language Models for Finance and Legal (LLMFinLegal), Association for Computational Linguistics.
13. EmoKEG: Knowledge-enhanced heterogeneous graph for emotion recognition in conversation10.1016/j.patcog.2026.113500 ↗
14. Interpreting Fedspeak with Confidence: A LLM-Based Uncertainty-Aware Framework Guided by Monetary Policy Transmission Paths
15. Uncertainty-Aware Long-Term Multi-Worker Behavior Anticipation for Proactive Human-Robot Collaboration in Construction10.1016/j.aei.2025.104203 ↗
16. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting
17. Nie, Y., Nguyen, N.H., Sinthong, P., and Kalagnanam, J. (2022). A time series is worth 64 words: Long-term forecasting with transformers. arXiv.
18. iTransformer: Inverted transformers are effective for time series forecasting
19. Time-LLM: Time series forecasting by reprogramming large language models
20. A representation learning framework for stock movement prediction10.1016/j.asoc.2023.110409 ↗
21. Recurrent ensemble random vector functional link neural network for financial time series forecasting10.1016/j.asoc.2024.111759 ↗
22. Enhancing financial time series forecasting with hybrid Deep Learning: CEEMDAN-Informer-LSTM model10.1016/j.asoc.2025.113241 ↗
23. A dynamic hypergraph attention network: Capturing market-wide spatiotemporal dependencies for stock selection10.1016/j.asoc.2024.112524 ↗
24. Model-Guided Stock Movements Prediction with Homogeneous–Heterogeneous pattern learning10.1016/j.asoc.2025.113519 ↗
25. Multi-modal deep learning for credit rating prediction using text and numerical data streams10.1016/j.asoc.2025.112771 ↗
26. A multi-scale multi-head attention network for stock trend prediction considering textual factors10.1016/j.asoc.2024.112388 ↗
27. Hierarchical multi-modal fusion architecture search for stock market forecasting10.1016/j.asoc.2025.113581 ↗
28. Deep Learning-Based Feature Fusion and Prediction Approach For Stock Market Forecasting10.1016/j.asoc.2025.113623 ↗
29. Liang, P.P., Lyu, Y., Fan, X., Wu, Z., Cheng, Y., Wu, J., Chen, L., Wu, P., Lee, M.A., and Zhu, Y. (2021). Multibench: Multiscale benchmarks for multimodal representation learning. Adv. Neural Inf. Process. Syst., 2021.
30. Wang, H., Chen, Y., Ma, C., Avery, J., Hull, L., and Carneiro, G. (2023). Multi-modal learning with missing modality via shared-specific feature modelling. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE.10.1109/CVPR52729.2023.01524 ↗
31. Shukla, S.N., and Marlin, B.M. (2021). Multi-time attention networks for irregularly sampled time series. arXiv.
32. Zhang, X., Zeman, M., Tsiligkaridis, T., and Zitnik, M. (2021). Graph-guided network for irregularly sampled multivariate time series. arXiv.
33. Zhang, W., Yin, C., Liu, H., Zhou, X., and Xiong, H. (2024, January 21–27). Irregular multivariate time series forecasting: A transformable patching graph neural networks approach. Proceedings of the Forty-First International Conference on Machine Learning, Vienna, Austria.
34. Embracing data irregularities in multivariate time series10.1016/j.asoc.2025.114039 ↗
35. CausalStock: Deep end-to-end causal discovery for news-driven multi-stock movement prediction10.52202/079017-1504 ↗
36. Stock trend prediction based on dynamic hypergraph spatio-temporal network10.1016/j.asoc.2024.111329 ↗
37. Sun, H., Zhang, J., Hong, W., Fang, Y., Ma, M., Shi, K., and Li, M. (2026). A Cross-Modal Temporal Alignment Framework for Artificial Intelligence-Driven Sensing in Multilingual Risk Monitoring. Sensors, 26.10.3390/s26082319 ↗
38. Modeling of limit order book data with ordered fuzzy numbers10.1016/j.asoc.2024.111555 ↗
39. Learning and processing framework using Fuzzy Deep Neural Network for trading and portfolio rebalancing10.1016/j.asoc.2024.111233 ↗
40. A system of trading in the foreign exchange market based on multi-criteria optimization under Belief-Plausibility uncertainty10.1016/j.asoc.2024.112573 ↗
41. Risk measurement of aggregation approaches in multiple attribute decision making under uncertain information10.1016/j.asoc.2024.111568 ↗
42. Revisiting the calibration of modern neural networks
43. Rethinking calibration of deep neural networks: Do not be afraid of overconfidence
44. Charpentier, B., Borchert, O., Zügner, D., Geisler, S., and Günnemann, S. (2021). Natural posterior network: Deep bayesian uncertainty for exponential family distributions. arXiv.
45. Mukhoti, J., Kirsch, A., Van Amersfoort, J., Torr, P.H., and Gal, Y. (2023). Deep deterministic uncertainty: A new simple baseline. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE.10.1109/CVPR52729.2023.02336 ↗
46. Board efficiency index, government ownership and financial performance of Shanghai stock exchange companies10.1108/AJAR-03-2025-0099 ↗
47. Social media, interaction information and stock market efficiency: Evidence from the Shenzhen stock exchange easy interaction platform in China
48. Caldeirinha, V., Felício, J.A., Pinho, T., and Rodrigues, R. (2024). Fuzzy-set QCA on performance and sustainability determinants of ports supporting floating offshore wind farms. Sustainability, 16.10.3390/su16072947 ↗
49. A stock price prediction model based on investor sentiment and optimized deep learning10.1109/ACCESS.2023.3278790 ↗
50. Knowledge representation framework of accounting event in corpus-based financial report text10.1007/s10586-018-2153-8 ↗
51. What drives users’ knowledge payment behaviour? The moderating role of knowledge expertise and social identity10.1080/0144929X.2024.2320213 ↗
52. A novel stock recommendation system using Guba sentiment analysis10.1007/s00779-018-1121-x ↗
53. Deconstructing China’s GDP statistics10.1016/j.chieco.2003.11.002 ↗
54. Allen, F., Gu, X., and Qian, J. (2017). People’s bank of China: History, current operations and future outlook. SSRN.10.2139/ssrn.3018506 ↗
55. Logistic regression10.1161/CIRCULATIONAHA.106.682658 ↗
56. Random forest10.17849/insm-47-01-31-39.1 ↗
57. Chen, T., and Guestrin, C. (2016). Xgboost: A scalable tree boosting system. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ACM.10.1145/2939672.2939785 ↗
58. Lightgbm: A highly efficient gradient boosting decision tree
59. LSTM: A search space odyssey10.1109/TNNLS.2016.2582924 ↗
60. Dey, R., and Salem, F.M. (2017). Gate-variants of gated recurrent unit (GRU) neural networks. Proceedings of the 2017 IEEE 60th International Midwest Symposium on Circuits and Systems (MWSCAS), IEEE.10.1109/MWSCAS.2017.8053243 ↗
61. Zhu, C., Chen, Z., Zhao, R., Wang, J., and Yan, R. (2021). Decoupled feature-temporal CNN: Explaining deep learning-based machine health monitoring. IEEE Trans. Instrum. Meas., 70.10.1109/TIM.2021.3084310 ↗
62. A survey on vision transformer10.1109/TPAMI.2022.3152247 ↗
63. Temporal fusion transformers for interpretable multi-horizon time series forecasting10.1016/j.ijforecast.2021.03.012 ↗
64. Barnum, G., Talukder, S., and Yue, Y. (2020). On the benefits of early fusion in multimodal representation learning. arXiv.
65. Late fusion incomplete multi-view clustering10.1109/TPAMI.2018.2879108 ↗
66. Tsai, Y.H.H., Bai, S., Liang, P.P., Kolter, J.Z., Morency, L.P., and Salakhutdinov, R. (2019). Multimodal transformer for unaligned multimodal language sequences. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics.10.18653/v1/P19-1656 ↗
67. Cross-modal attention network for temporal inconsistent audio-visual event localization
68. Arevalo, J., Solorio, T., Montes-y Gómez, M., and González, F.A. (2017). Gated multimodal units for information fusion. arXiv.
69. Bayesian LSTM with stochastic variational inference for estimating model uncertainty in process-based hydrological models10.1029/2021WR029772 ↗
70. Multidimensional uncertainty-aware evidential neural networks
71. Fuzzy neural networks and neuro-fuzzy networks: A review the main techniques and applications used in the literature10.1016/j.asoc.2020.106275 ↗
72. Dong, Z., Fan, X., and Peng, Z. (2024). Fnspid: A comprehensive financial news dataset in time series. Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, ACM.10.1145/3637528.3671629 ↗
73. Aaron7Sun (2026, June 02). Daily News for Stock Market Prediction. Available online: https://www.kaggle.com/datasets/aaron7sun/stocknews.
74. Lee, J., Youn, H.L., Poon, J., and Han, S.C. (2023). StockEmotions: Discover Investor Emotions for Financial Sentiment Analysis and Multivariate Time Series. arXiv.