Improving Mandarin ASR Performance Through Multimodality
Source abstract
Abstract
In the context of Internet of Things (IoT) applications, accurate and efficient speech recognition is essential for enabling seamless voice-based interactions and control. Mandarin ASR, in particular, presents unique challenges due to the ideographic nature of the Chinese language, where recognition results are not directly correlated with pronunciation. Pinyin, as a representation of Chinese character pronunciation, has an intrinsic connection with Chinese characters, making it a valuable tool for enhancing ASR performance. This paper proposes a multimodal ASR neural network that combines pinyin data from the text modality and speech data from the audio modality as shared inputs to the ASR model. Specifically, the system processes the speech input through a preprocessed WeNet to generate pinyin text, which is then enhanced using a label denoising algorithm to improve its accuracy. The proposed text-acoustic multimodal ASR model improves the overall speech recognition performance by approximately 4%, making it more suitable for IoT applications that require high accuracy in voice commands and interactions.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Speech Recognition and Synthesis · Speech and Audio Processing · Emotion and Mood Recognition
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Faculty of Electronic and Information Engineering, Xi’an Jiaotong University, Xi’an 710049, China
Funding
National Key R&D Program of China · NSFC
References (39)
Show source-provided references
1. Speech recognition system; challenges and techniques
2. Zaidi, S.F.N., Shukla, V.K., Mishra, V.P., and Singh, B. (2021). Redefining home automation through voice recognition system. Emerging Technologies in Data Mining and Information Security: Proceedings of IEMIS 2020, Volume 2, Springer.10.1007/978-981-33-4367-2_16 ↗
3. Zhang, S., Lei, M., Liu, Y., and Li, W. (2019, January 12–17). Investigation of modeling units for mandarin speech recognition using dfsmn-ctc-smbr. Proceedings of the ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brighton, UK.10.1109/ICASSP.2019.8683859 ↗
4. WeNet: Production Oriented Streaming and Non-Streaming End-to-End Speech Recognition Toolkit
5. Dong, L., Xu, S., and Xu, B. (2018, January 15–20). Speech-transformer: A no-recurrence sequence-to-sequence model for speech recognition. Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, AB, Canada.10.1109/ICASSP.2018.8462506 ↗
6. Boosting Character-based Mandarin ASR via Chinese Pinyin Representation10.1007/s10772-023-10050-z ↗
7. Yang, Z., Ng, D., Fu, X., Han, L., Xi, W., Wang, R., Jiang, R., and Zhao, J. (2022). On the Effectiveness of Pinyin-Character Dual-Decoding for End-to-End Mandarin Chinese ASR. arXiv.
8. Chinese Pinyin10.1353/aad.2021.0038 ↗
9. Effendi, J., Tjandra, A., Sakti, S., and Nakamura, S. (2019, January 14–18). Listening while speaking and visualizing: Improving ASR through multimodal chain. Proceedings of the 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), Singapore.10.1109/ASRU46091.2019.9003899 ↗
10. Ma, Y. (2024, January 1–3). Design and Implementation of Smart Home Air Monitoring System. Proceedings of the 4th International Conference on Computer, Internet of Things and Control Engineering, Wuhan, China.10.1145/3705677.3705703 ↗
11. Kumar, M., Mukherjee, P., Verma, S., Kaur, M., Singh, S., Kobielnik, M., Woźniak, M., Shafi, J., and Ijaz, M.F. (2022). BBNSF: Blockchain-based novel secure framework using RP2-RSA and ASR-ANN technique for IoT enabled healthcare systems. Sensors, 22.10.3390/s22239448 ↗
12. Speech-oriented sparse attention denoising for voice user interface toward industry 5.010.1109/TII.2022.3206872 ↗
13. Zhang, B., Wu, D., Peng, Z., Song, X., Yao, Z., Lv, H., Xie, L., Yang, C., Pan, F., and Niu, J. (2022). Wenet 2.0: More productive end-to-end speech recognition toolkit. arXiv.10.21437/Interspeech.2022-483 ↗
14. Bai, Y., Chen, J., Chen, J., Chen, W., Chen, Z., Ding, C., Dong, L., Dong, Q., Du, Y., and Gao, K. (2024). Seed-ASR: Understanding Diverse Speech and Contexts with LLM-based Speech Recognition. arXiv.
15. He, J., Shi, X., Li, X., and Toda, T. (2024, January 14–19). MF-AED-AEC: Speech Emotion Recognition by Leveraging Multimodal Fusion, ASR Error Detection, and ASR Error Correction. Proceedings of the ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Seoul, Republic of Korea.10.1109/ICASSP48485.2024.10446548 ↗
16. Oneață, D., and Cucu, H. (2022, January 19–20). Improving multimodal speech recognition by data augmentation and speech representations. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPRW56347.2022.00504 ↗
17. Afouras, T., Chung, J.S., and Zisserman, A. (2020, January 4–8). Asr is all you need: Cross-modal distillation for lip reading. Proceedings of the ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Barcelona, Spain.10.1109/ICASSP40776.2020.9054253 ↗
18. Reconstruction of speech from whispers10.1016/S1350-4533(02)00060-7 ↗
19. Ao, J., Wang, R., Zhou, L., Wang, C., Ren, S., Wu, Y., Liu, S., Ko, T., Li, Q., and Zhang, Y. (2022, January 22–27). SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Dublin, Ireland.10.18653/v1/2022.acl-long.393 ↗
20. Semantic context enhances the early auditory encoding of natural speech10.1523/JNEUROSCI.0584-19.2019 ↗
21. Ji, X., Zhu, Z., Xi, W., Gadyatskaya, O., Song, Z., Cai, Y., and Liu, Y. (2024, January 20–27). FedFixer: Mitigating heterogeneous label noise in federated learning. Proceedings of the AAAI Conference on Artificial Intelligence, Vancouver, BC, Canada.10.1609/aaai.v38i11.29179 ↗
22. Lukasik, M., Bhojanapalli, S., Menon, A., and Kumar, S. (2020, January 13–18). Does label smoothing mitigate label noise?. Proceedings of the International Conference on Machine Learning, PMLR, Online.
23. Hao, Y., Madani, S., Guan, J., Alloulah, M., Gupta, S., and Hassanieh, H. (2024, January 16–22). Bootstrapping autonomous driving radars with self-supervised learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR52733.2024.01422 ↗
24. Xu, P., Xiang, Z., Qiao, C., Fu, J., and Pu, T. (2024, January 16–22). Adaptive multi-modal cross-entropy loss for stereo matching. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR52733.2024.00491 ↗
25. Focal loss for dense object detection
26. Wang, J., Huang, D., Wu, X., Tang, Y., and Lan, L. (2024, January 14–19). Continuous review and timely correction: Enhancing the resistance to noisy labels via self-not-true distillation. Proceedings of the ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Seoul, Republic of Korea.10.1109/ICASSP48485.2024.10446765 ↗
27. Berthelot, D., Carlini, N., Goodfellow, I., Papernot, N., Oliver, A., and Raffel, C.A. (2019). Mixmatch: A holistic approach to semi-supervised learning. Adv. Neural Inf. Process. Syst., 32.
28. Radhakrishnan, A., Davis, J., Rabin, Z., Lewis, B., Scherreik, M., and Ilin, R. (2024, January 3–8). Design choices for enhancing noisy student self-training. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, Waikoloa, HI, USA.10.1109/WACV57701.2024.00193 ↗
29. Han, B., Yao, Q., Yu, X., Niu, G., Xu, M., Hu, W., Tsang, I., and Sugiyama, M. (2018). Co-teaching: Robust training of deep neural networks with extremely noisy labels. Adv. Neural Inf. Process. Syst., 31.
30. Li, J., Socher, R., and Hoi, S.C. (2020). Dividemix: Learning with noisy labels as semi-supervised learning. arXiv.
31. Jiang, L., Zhou, Z., Leung, T., Li, L.J., and Li, F.F. (2018, January 10–15). Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. Proceedings of the International Conference on Machine Learning, PMLR, Stockholm, Sweden.
32. Wang, D., and Zhang, X. (2025, August 01). THCHS-30: A Free Chinese Speech Corpus. arXiv e-Prints, Available online: http://index.cslt.org/mediawiki/images/f/fe/Thchs30.pdf.
33. Pseudo Labeling Methods for Semi-Supervised Semantic Segmentation: A Review and Future Perspectives10.1109/TCSVT.2024.3508768 ↗
34. Liu, S., Cao, W., Fu, R., Yang, K., and Yu, Z. (2024, January 20–27). RPSC: Robust pseudo-labeling for semantic clustering. Proceedings of the AAAI Conference on Artificial Intelligence, Vancouver, BC, Canada.10.1609/aaai.v38i12.29309 ↗
35. Pei, H., Xiong, Y., Wang, P., Tao, J., Liu, J., Deng, H., Ma, J., and Guan, X. (2024, January 13–17). Memory disagreement: A pseudo-labeling measure from training dynamics for semi-supervised graph learning. Proceedings of the ACM Web Conference 2024, Singapore.10.1145/3589334.3645398 ↗
36. Yu, X., Han, B., Yao, J., Niu, G., Tsang, I., and Sugiyama, M. (, January 9–15). How does disagreement help generalization against label corruption?. Proceedings of the International Conference on Machine Learning, PMLR, Long Beach, CA, USA.
37. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (2017). Attention is all you need. Adv. Neural Inf. Process. Syst., 30.
38. Gulati, A., Qin, J., Chiu, C.C., Parmar, N., Zhang, Y., Yu, J., Han, W., Wang, S., Zhang, Z., and Wu, Y. (2020). Conformer: Convolution-augmented transformer for speech recognition. arXiv.10.21437/Interspeech.2020-3015 ↗
39. Bu, H., Du, J., Na, X., Wu, B., and Zheng, H. (2017, January 1–3). AIShell-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baseline. Proceedings of the Oriental COCOSDA 2017, Seoul, Republic of Korea. Submitted.10.1109/ICSDA.2017.8384449 ↗