OS-Denseformer: A Lightweight End-to-End Noise-Robust Method for Chinese Speech Recognition
Source abstract
Abstract
Automatic speech recognition (ASR) technology faces the dual challenges of model complexity and noise robustness when deployed on terminal devices (e.g., mobile devices, embedded systems). To meet the demand for lightweight and high-performance models in terminal devices, we propose a lightweight end-to-end speech recognition model, OS-Denseformer (Omni-Scale-Denseformer). The core of this model lies in its lightweight design and noise adaptability: multi-scale acoustic features are efficiently extracted through a multi-sampling structure to enhance noise robustness; the proposed OS-Conv module improves local feature extraction capability while significantly reducing the number of parameters, enhancing computational efficiency, and lowering model complexity; the proposed normalization function, ExpNorm, normalizes the model output, facilitating more accurate parameter optimization during model training. Finally, we employ distinct loss functions across different training stages, using Minimum Bayes Risk (MBR) joint optimization to determine the optimal weighting scheme that directly minimizes the character error rate (CER). Experimental results on public datasets such as AISHELL-1 demonstrate that, under a high-noise environment of −15 dB, the CER of the OS-Denseformer model is reduced by 9.95%, 7.97%, and 4.85% compared to the benchmark models Squeezeformer, Conformer, and Zipformer, respectively. Additionally, the model parameter count is reduced by 53.35%, 10.27%, and 27.66%, while the giga floating-point operations per second (GFLOPs) are decreased by 67.51%, 66.51%, and 13.82%, respectively. Deployment on resource-constrained mobile devices demonstrates that, compared to Conformer, OS-Denseformer reduced memory usage by 10.79% and decreased inference latency by 61.62%.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Speech Recognition and Synthesis · Speech and Audio Processing · Music and Audio Processing
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
The Institute of Cyberspace Security, Zhejiang University of Technology, Hangzhou 310023, China
Funding
National Natural Science Foundation of China
References (34)
Show source-provided references
1. Deep Learning for Steganalysis of Diverse Data Types: A Review of Methods, Taxonomy, Challenges and Future Directions10.1016/j.neucom.2024.127528 ↗
2. Automatic Speech Recognition of Portuguese Phonemes Using Neural Networks Ensemble10.1016/j.eswa.2023.120378 ↗
3. Trends and Developments in Automatic Speech Recognition Research10.1016/j.csl.2023.101538 ↗
4. Deep Transfer Learning for Automatic Speech Recognition: Towards Better Generalization10.1016/j.knosys.2023.110851 ↗
5. Automatic Speech Recognition Using Advanced Deep Learning Approaches: A Survey10.1016/j.inffus.2024.102422 ↗
6. Radford, A., Kim, J.W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, I. (2023, January 23–29). Robust Speech Recognition via Large-Scale Weak Supervision. Proceedings of the International Conference on Machine Learning, Honolulu, HI, USA.
7. Zhang, W., Zhai, M., Huang, Z., Liu, C., Li, W., and Cao, Y. (2019, January 8–11). Towards End-to-End Speech Recognition with Deep Multipath Convolutional Neural Networks. Proceedings of the International Conference on Intelligent Robotics and Applications, Shenyang, China.10.1007/978-3-030-27529-7_29 ↗
8. Miyazaki, K., Murata, M., and Koriyama, T. (2023, January 4–10). Structured State Space Decoder for Speech Recognition and Synthesis. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Rhodes Island, Greece.10.1109/ICASSP49357.2023.10096135 ↗
9. Karita, S., Chen, N., Hayashi, T., Hori, T., Inaguma, H., Jiang, Z., Someki, M., Soplin, N.E.Y., Yamamoto, R., and Wang, X. (2019, January 15–18). A Comparative Study on Transformer vs RNN in Speech Applications. Proceedings of the IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), Taipei, China.10.1109/ASRU46091.2019.9003750 ↗
10. Zhang, Q., Lu, H., Sak, H., Tripathi, A., McDermott, E., Koo, S., and Kumar, S. (2020, January 4–8). Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Barcelona, Spain.10.1109/ICASSP40776.2020.9053896 ↗
11. Wang, Y., Mohamed, A., Le, D., Liu, C., Xiao, A., Mahadeokar, J., Huang, H., Tjandra, A., Zhang, X., and Zhang, F. (2020, January 4–8). Transformer-Based Acoustic Modeling for Hybrid Speech Recognition. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Barcelona, Spain.10.1109/ICASSP40776.2020.9054345 ↗
12. Gulati, A., Qin, J., Chiu, C.-C., Parmar, N., Zhang, Y., Yu, J., Han, W., Wang, S., Zhang, Z., and Wu, Y. (2020, January 25–29). Conformer: Convolution-Augmented Transformer for Speech Recognition. Proceedings of the Interspeech 2020, Shanghai, China.10.21437/Interspeech.2020-3015 ↗
13. Peng, Y., Dalmia, S., Lane, I., and Watanabe, S. (2022, January 18–21). Branchformer: Parallel MLP-Attention Architectures to Capture Local and Global Context for Speech Recognition and Understanding. Proceedings of the International Conference on Machine Learning, Baltimore, MD, USA.
14. Kim, K., Wu, F., Peng, Y., Pan, J., Sridhar, P., Han, K.J., and Watanabe, S. (2023, January 9–12). E-Branchformer: Branchformer with Enhanced Merging for Speech Recognition. Proceedings of the IEEE Spoken Language Technology Workshop (SLT), Macau, China.10.1109/SLT54892.2023.10022656 ↗
15. Kim, S., Gholami, A., Shaw, A., Lee, N., Mangalam, K., Malik, J., Mahoney, M.W., and Keutzer, K. (December, January 28). Squeezeformer: An Efficient Transformer for Automatic Speech Recognition. Proceedings of the Advances in Neural Information Processing Systems, New Orleans, LA, USA.
16. Yao, Z., Guo, L., Yang, X., Kang, W., Kuang, F., Yang, Y., Jin, Z., Lin, L., and Povey, D. (2023). Zipformer: A Faster and Better Encoder for Automatic Speech Recognition. arXiv.
17. Gao, X., and Chen, N.F. (2024, January 2–5). Speech-Mamba: Long-Context Speech Recognition with Selective State Spaces Models. Proceedings of the IEEE Spoken Language Technology Workshop (SLT), Macau, China.10.1109/SLT61566.2024.10832137 ↗
18. Gao, Z., Zhang, S., McLoughlin, I., and Yan, Z. (2022). Paraformer: Fast and Accurate Parallel Transformer for Non-Autoregressive End-to-End Speech Recognition. arXiv.10.21437/Interspeech.2022-9996 ↗
19. Xu, K.-T., Xie, F.-L., Tang, X., and Hu, Y. (2025). FireRedASR: Open-Source Industrial-Grade Mandarin Speech Recognition Models from Encoder-Decoder to LLM Integration. arXiv.
20. Yu, W., Tang, C., Sun, G., Chen, X., Tan, T., Li, W., Lu, L., Ma, Z., and Zhang, C. (2024, January 14–19). Connecting Speech Encoder and Large Language Model for ASR. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Seoul, Republic of Korea.10.1109/ICASSP48485.2024.10445874 ↗
21. A Deep Denoising Sound Coding Strategy for Cochlear Implants10.1109/TBME.2023.3262677 ↗
22. Spectral Subtraction-Based Speech Enhancement for Cochlear Implant Patients in Background Noise10.1121/1.1852873 ↗
23. The Voice Track Multiband Single-Channel Modified Wiener-Filter Noise Reduction System for Cochlear Implants: Patients’ Outcomes and Subjective Appraisal10.3109/14992027.2016.1172267 ↗
24. Sun, X., Xia, R., Li, J., and Yan, Y. (2019, January 12–17). A Deep Learning Based Binaural Speech Enhancement Approach with Spatial Cues Preservation. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brighton, UK.10.1109/ICASSP.2019.8683589 ↗
25. Nian, Z., Tu, Y.-H., Du, J., and Lee, C.-H. (2021, January 6–11). A Progressive Learning Approach to Adaptive Noise and Speech Estimation for Speech Enhancement and Noisy Speech Recognition. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Toronto, ON, Canada.10.1109/ICASSP39728.2021.9413395 ↗
26. Modeling Speech Structure to Improve TF Masks for Speech Enhancement and Recognition10.1109/TASLP.2022.3196168 ↗
27. Pizarro, M., Kolossa, D., and Fischer, A. (2021, January 10–12). Robustifying Automatic Speech Recognition by Extracting Slowly Varying Features. Proceedings of the SPSC 2021, Virtual.10.21437/SPSC.2021-8 ↗
28. Burchi, M., and Timofte, R. (2023, January 2–7). Audio-Visual Efficient Conformer for Robust Speech Recognition. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, Waikoloa, HI, USA.10.1109/WACV56688.2023.00229 ↗
29. Tang, W., Long, G., Liu, L., Zhou, T., Blumenstein, M., and Jiang, J. (2022, January 25–29). Omni-Scale CNNs: A Simple and Effective Kernel Size Configuration for Time Series Classification. Proceedings of the International Conference on Learning Representations, Virtual.
30. Huang, G., Liu, Z., van der Maaten, L., and Weinberger, K.Q. (2017, January 21–26). Densely Connected Convolutional Networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.243 ↗
31. Weng, C., Yu, C., Cui, J., Zhang, C., and Yu, D. (2020, January 25–29). Minimum Bayes Risk Training of RNN-Transducer for End-to-End Speech Recognition. Proceedings of the Interspeech 2020, Shanghai, China.10.21437/Interspeech.2020-1221 ↗
32. Bu, H., Du, J., Na, X., Wu, B., and Zheng, H. (2017, January 17–19). AISHELL-1: An Open-Source Mandarin Speech Corpus and a Speech Recognition Baseline. Proceedings of the 20th Conference of the Oriental Chapter of the International Coordinating Committee on Speech Databases and Speech I/O Systems and Assessment (O-COCOSDA), Hsinchu, China.10.1109/ICSDA.2017.8384449 ↗
33. Dean, D., Sridharan, S., Vogt, R., and Mason, M. (2010, January 26–30). The QUT-NOISE-TIMIT Corpus for Evaluation of Voice Activity Detection Algorithms. Proceedings of the 11th Annual Conference of the International Speech Communication Association (INTERSPEECH), Makuhari, Japan.10.21437/Interspeech.2010-774 ↗
34. Assessment for Automatic Speech Recognition: II. NOISEX-92: A Database and an Experiment to Study the Effect of Additive Noise on Speech Recognition Systems10.1016/0167-6393(93)90095-3 ↗