AFT-SAM: Adaptive Fusion Transformer with a Sparse Attention Mechanism for Audio–Visual Speech Recognition
Source abstract
Abstract
Aiming at the problems of serious information redundancy, complex inter-modal information interaction, and difficult multimodal fusion faced by the audio–visual speech recognition system when dealing with complex multimodal information, this paper proposes an adaptive fusion transformer algorithm (AFT-SAM) based on a sparse attention mechanism. The algorithm adopts the sparse attention mechanism in the feature-encoding process to reduce excessive attention to non-important regions and dynamically adjusts the attention weights through adaptive fusion to capture and integrate the multimodal information more effectively and reduce the impact of redundant information on the model performance. Experiments are conducted on the audio–visual speech recognition dataset LRS2 and compared with other algorithms, and the experimental results show that the proposed algorithm in this paper has significantly lower WERs in the audio-only, visual-only, and audio–visual bimodal cases.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Speech and Audio Processing · Speech Recognition and Synthesis
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
College of Computer Science and Technology, Changchun University, Changchun 130022, China
Ministry of Education Key Laboratory of Intelligent Rehabilitation and Barrier-Free Access for the Disabled, Changchun 130022, China
Jilin Provincial Key Laboratory of Human Health State Identification and Function Enhancement, Changchun 130022, China
Funding
Science and Technology Research Project of the Education Department of Jilin Province · Key Project of the Jilin Special Education Society · Jilin Province Science and Technology Development Plan Project · Changchun Science and Technology Development Plan Project · Science and Technology Development Plan Project of the Jilin Provincial Science and Technology Department
References (44)
Show source-provided references
1. Sterpu, G., Saam, C., and Harte, N. (2018, January 16–20). Attention-based Audio-Visual Fusion for Robust Automatic Speech Recognition. Proceedings of the International Conference on Multimodal Interaction, Boulder, CO, USA.10.1145/3242969.3243014 ↗
2. Ma, P., Haliassos, A., Fernandez-Lopez, A., Chen, H., Petridis, S., and Pantic, M. (2023, January 4–10). Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labels. Proceedings of the ICASSP 2023—2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Rhodes Island, Greece.10.1109/ICASSP49357.2023.10096889 ↗
3. Hu, Y., Li, R., Chen, C., Zou, H., Zhu, Q., and Chng, E.S. (2023, January 19–25). Cross-modal global interaction and local alignment for audio-visual speech recognition. Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence {IJCAI-23}, Macao, China.10.24963/ijcai.2023/564 ↗
4. George, S., Christian, S., and Naomi, H. (2020, January 25–29). Should we hard-code the recurrence concept or learn it instead ? Exploring the Transformer architecture for Audio-Visual Speech Recognition. Proceedings of the Conference of the International Speech Communication Association (INTERSPEECH 2020), Shanghai, China.
5. Yang, W., Li, P., Yang, W., Liu, Y., He, Y., Petrosian, O., and Davydenko, A. (2023). Research on Robust Audio-Visual Speech Recognition Algorithms. Mathematics, 11.10.3390/math11071733 ↗
6. Hong, J., Kim, M., Choi, J., and Ro, Y.M. (2023, January 17–24). Watch or Listen: Robust Audio-Visual Speech Recognition With Visual Corruption Modeling and Reliability Scoring. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada.10.1109/CVPR52729.2023.01801 ↗
7. Wang, D., Wang, X.D., and Lv, S.H. (2019). An overview of end-to-end automatic speech recognition. Symmetry, 11.10.3390/sym11081018 ↗
8. Bengio, S. (2002, January 9–14). An Asynchronous Hidden Markov Model for Audio-Visual Speech Recognition. Proceedings of the Conference on Neural Information Processing Systems, Vancouver, BC, Canada.
9. Bhatnagar, M., Shinghal, K., and Gurjar, S. (2008, January 8–11). AVSR through HMM and lip tracking method. Proceedings of the 2008 International Symposium on Intelligent Signal Processing and Communications Systems, Bangkok, Thailand.
10. Makino, T., Liao, H., Assael, Y.M., Shillingford, B., Garcia, B., Braga, O., and Siohan, O. (2019, January 14–18). Recurrent Neural Network Transducer for Audio-Visual Speech Recognition. Proceedings of the 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), Sentosa, Singapore.10.1109/ASRU46091.2019.9004036 ↗
11. Yang, X., Molchanov, P., and Kautz, J. (2018, January 18–23). Making Convolutional Networks Recurrent for Visual Sequence Learning. Proceedings of the Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00677 ↗
12. Burchi, M., Puvvada, K.C., Balam, J., Ginsburg, B., and Timofte, R. (2024, January 14–19). Multilingual Audio-Visual Speech Recognition with Hybrid CTC/RNN-T Fast Conformer. Proceedings of the ICASSP 2024—2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Seoul, Republic of Korea.10.1109/ICASSP48485.2024.10445891 ↗
13. Feng, W., Guan, N., Li, Y., Zhang, X., and Luo, Z. (2017, January 14–19). Audio Visual Speech Recognition With Multimodal Recurrent Neural Networks. Proceedings of the IEEE International Joint Conference on Neural Network, Anchorage, Alaska, USA.10.1109/IJCNN.2017.7965918 ↗
14. Petridis, S., Li, Z., and Pantic, M. (2017, January 5–9). End-to-end visual speech recognition with LSTMs. Proceedings of the 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Los Angeles, America.10.1109/ICASSP.2017.7952625 ↗
15. Subakan, C., Ravanelli, M., Cornell, S., Bronzi, M., and Zhong, J. (2021, January 6–11). Attention is All You Need in Speech Separation. Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing, Toronto, ON, Canada.10.1109/ICASSP39728.2021.9413901 ↗
16. Vladimir, I., and Esa, R. (2020, January 13–19). Multi-modal Dense Video Captioning. Proceedings of the Computer Vision and Pattern Recognition, Seattle, WA, USA.
17. Serdyuk, D., Braga, O., and Siohan, O. (2022, January 18–22). Transformer-based video front-ends for audio-visual speech recognition for single and multi-person video. Proceedings of the INTERSPEECH 2022, Incheon, Republic of Korea.10.21437/Interspeech.2022-10920 ↗
18. Audio-visual speech recognition based on regulated transformer and spatio-temporal fusion strategy for driver assistive systems10.1016/j.eswa.2024.124159 ↗
19. Deep audio-visual speech recognition10.1109/TPAMI.2018.2889052 ↗
20. Li, D.S., Gao, Y., Zhu, C.Y., Wang, Q.R., and Wang, R.X. (2023). Improving speech recognition performance in noisy environments by enhancing lip reading accuracy. Sensors, 23.10.3390/s23042053 ↗
21. Xue, J., Zheng, T., and Han, J. (2020, January 4–8). Structured Sparse Attention for End-to-End Automatic Speech Recognition. Proceedings of the ICASSP 2020—2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Barcelona, Spain.10.1109/ICASSP40776.2020.9054245 ↗
22. Sun, T.W., and Wu, A.Y. (2019, January 18–20). Sparse Autoencoder with Attention Mechanism for Speech Emotion Recognition. Proceedings of the 2019 IEEE International Conference on Artificial Intelligence Circuits and Systems (AICAS), Taiwan.10.1109/AICAS.2019.8771593 ↗
23. Zhao, C., Wang, J., Wei, W., Qu, X., Wang, H., and Xiao, J. (2022, January 3–16). Adaptive Sparse and Monotonic Attention for Transformer-based Automatic Speech Recognition. Proceedings of the 2022 IEEE 9th International Conference on Data Science and Advanced Analytics (DSAA), Shenzhen, China.10.1109/DSAA54385.2022.10032360 ↗
24. Woo, B.J., Kim, H.Y., Kim, J., and Kim, N.S. (2021, January 17–19). Speech Separation Based on DPTNet with Sparse Attention. Proceedings of the 2021 7th IEEE International Conference on Network Intelligence and Digital Content (IC-NIDC), Beijing, China.10.1109/IC-NIDC54101.2021.9660488 ↗
25. Bhasi K., C., Rajan, R., and Noumida, A. (2024, January 1–5). Attention-augmented X-vectors for the Evaluation of Mimicked Speech Using Sparse Autoencoder-LSTM Framework. Proceedings of the Interspeech 2024, Kos Island, Greece.10.21437/Interspeech.2024-2036 ↗
26. Wang, X., Sun, S., Xie, L., and Ma, L. (September, January 30). Efficient Conformer with Prob-Sparse Attention Mechanism for End-to-EndSpeech Recognition. Proceedings of the Interspeech 2021, Brno, Czechia.10.21437/Interspeech.2021-415 ↗
27. DAA: Dual LSTMs with Adaptive Attention for Image Captioning10.1016/j.neucom.2019.06.085 ↗
28. Parisae, V., and Bhavanam, S.N. (2024). Adaptive Attention Mechanism for Single Channel Speech Enhancement. Multimedia Tools and Applications, Springer.10.1007/s11042-024-19076-0 ↗
29. Adaptive Selection of Local and Non-Local Attention Mechanisms for Speech Enhancement10.1016/j.neunet.2024.106236 ↗
30. Theory of communication
31. Afouras, T., Chung, J.S., and Zisserman, A. (2018, January 2–6). Deep Lip Reading: A comparison of models and an online application. Proceedings of the Conference of the International Speech Communication Association, Hyderabad, India.10.21437/Interspeech.2018-1943 ↗
32. Paraskevopoulos, G., Parthasarathy, S., Khare, A., and Sundaram, S. (2020, January 5–10). Multiresolution and Multimodal Speech Recognition with Transformers. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL 2020), Online.10.18653/v1/2020.acl-main.216 ↗
33. Graves, A., and Jaitly, N. (2014, January 21–26). Towards End-to-end Speech Recognition with Recurrent Neural Networks. Proceedings of the International Conference on Machine Learning (ICML), Beijing, China.
34. Vidhya, B., O, R.M., and Yazhini, K.S. (2024, January 21–23). Deep Learning Model Efficient Lip Reading by Using Connectionist Temporal Classification Algorithm. Proceedings of the 2024 IEEE Students Conference on Engineering and Systems (SCES), Prayagraj, India.10.1109/SCES61914.2024.10652291 ↗
35. Training a Singing Transcription Model Using Connectionist Temporal Classification Loss and Cross-Entropy Loss10.1109/TASLP.2022.3224297 ↗
36. Graves, A., Fernández, S., Gomez, F., and Schmidhuber, J. (2006, January 25–29). Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks. Proceedings of the 23rd International Conference on Machine Learning, Pittsburgh, PA, USA.10.1145/1143844.1143891 ↗
37. Sequence Modeling with CTC10.23915/distill.00008 ↗
38. PyTorch: An Imperative Style, High-Performance Deep Learning Library
39. Kingma, D., and Ba, J. (2014). Adam: A Method for Stochastic Optimization. Comput. Sci.
40. Understanding the Difficulty of Training Deep Feedforward Neural Networks
41. A Survey on Curriculum Learning
42. Speech recognition in noisy environments: A survey10.1016/0167-6393(94)00059-J ↗
43. Lee, Y.-H., Jang, D.-W., Kim, J.-B., Park, R.-H., and Park, H.-M. (2020). Audio-Visual Speech Recognition Based On Dual Cross-Modality Attentions With The Transformer Model. Appl. Sci., 10.10.3390/app10207263 ↗
44. Multimodal sparse Transformer network for audio-visual speech recognition10.1109/TNNLS.2022.3163771 ↗