LWMD: A Comprehensive Compression Platform for End-to-End Automatic Speech Recognition Models
Source abstract
Abstract
Recently end-to-end (E2E) automatic speech recognition (ASR) models have achieved promising performance. However, existing models tend to adopt increasing model sizes and suffer from expensive resource consumption for real-world applications. To compress E2E ASR models and obtain smaller model sizes, we propose a comprehensive compression platform named LWMD (light-weight model designing), which consists of two essential parts: a light-weight architecture search (LWAS) framework and a differentiable structured pruning (DSP) algorithm. On the one hand, the LWAS framework adopts the neural architecture search (NAS) technique to automatically search light-weight architectures for E2E ASR models. By integrating different architecture topologies of existing models together, LWAS designs a topology-fused search space. Furthermore, combined with the E2E ASR training criterion, LWAS develops a resource-aware search algorithm to select light-weight architectures from the search space. On the other hand, given the searched architectures, the DSP algorithm performs structured pruning to reduce parameter numbers further. With a Gumbel re-parameter trick, DSP builds a stronger correlation between the pruning criterion and the model performance than conventional pruning methods. And an attention-similarity loss function is further developed for better performance. On two mandarin datasets, Aishell-1 and HKUST, the compression results are well evaluated and analyzed to demonstrate the effectiveness of the LWMD platform.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Speech Recognition and Synthesis · Music and Audio Processing · Speech and Audio Processing
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Key Laboratory of Speech Acoustics and Content Understanding, Institute of Acoustic, Chinese Academy of Sciences, No. 21 North 4th Ring Road, Haidian District, Beijing 100190, China
University of Chinese Academy of Sciences, Beijing 101408, China
Xinjiang Laboratory of Minority Speech and Language Information Processing, Xinjiang Technical Institute of Physics and Chemistry, Chinese Academy of Sciences, 40-1 South Beijing Road Urumqi, Urumqi 830011, China
Funding
National Key Research and Development Program of China · Goal-Oriented Project Independently Deployed by Institute of Acoustics, Chinese Academy of Sciences
References (35)
Show source-provided references
1. Unsupervised and supervised VAD systems using combination of time and frequency domain features10.1016/j.bspc.2020.102044 ↗
2. milVAD: A bag-level MNIST modelling of voice activity detection using deep multiple instance learning10.1016/j.bspc.2022.103520 ↗
3. Hybrid voice activity detection system based on LSTM and auditory speech features10.1016/j.bspc.2022.104408 ↗
4. Hori, T., Watanabe, S., and Hershey, J.R. (August, January 30). Joint CTC/attention decoding for end-to-end speech recognition. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vancouver, BC, Canada.
5. Chiu, C.C., Sainath, T.N., Wu, Y., Prabhavalkar, R., Nguyen, P., Chen, Z., Kannan, A., Weiss, R.J., Rao, K., and Gonina, E. (2018, January 15–20). State-of-the-art speech recognition with sequence-to-sequence models. Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, AB, Canada.10.1109/ICASSP.2018.8462105 ↗
6. Amodei, D., Ananthanarayanan, S., Anubhai, R., Bai, J., Battenberg, E., Case, C., Casper, J., Catanzaro, B., Cheng, Q., and Chen, G. (2016, January 20–22). Deep speech 2: End-to-end speech recognition in english and mandarin. Proceedings of the International Conference on Machine Learning, New York, NY, USA.
7. Graves, A., Fernández, S., Gomez, F., and Schmidhuber, J. (2006, January 25–29). Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks. Proceedings of the 23rd International Conference on Machine Learning, Pittsburgh, PA, USA.10.1145/1143844.1143891 ↗
8. Online hybrid CTC/attention end-to-end automatic speech recognition architecture10.1109/TASLP.2020.2987752 ↗
9. ETEH: Unified Attention-Based End-to-End ASR and KWS Architecture10.1109/TASLP.2022.3161159 ↗
10. Gulati, A., Qin, J., Chiu, C.C., Parmar, N., Zhang, Y., Yu, J., Han, W., Wang, S., Zhang, Z., and Wu, Y. (2020). Conformer: Convolution-augmented transformer for speech recognition. arXiv.10.21437/Interspeech.2020-3015 ↗
11. Wu, F., Fan, A., Baevski, A., Dauphin, Y.N., and Auli, M. (2019). Pay less attention with lightweight and dynamic convolutions. arXiv.
12. Wu, Z., Liu, Z., Lin, J., Lin, Y., and Han, S. (2020). Lite transformer with long-short range attention. arXiv.
13. NAS-SCAE: Searching Compact Attention-based Encoders For End-to-end Automatic Speech Recognition10.21437/Interspeech.2022-748 ↗
14. Gordon, M.A., Duh, K., and Andrews, N. (2020). Compressing bert: Studying the effects of weight pruning on transfer learning. arXiv.10.18653/v1/2020.repl4nlp-1.18 ↗
15. Mao, Y., Wang, Y., Wu, C., Zhang, C., Wang, Y., Yang, Y., Zhang, Q., Tong, Y., and Bai, J. (2020). Ladabert: Lightweight adaptation of bert through hybrid model compression. arXiv.10.18653/v1/2020.coling-main.287 ↗
16. Michel, P., Levy, O., and Neubig, G. (2019). Are sixteen heads really better than one?. Adv. Neural Inf. Process. Syst., 32, Available online: https://proceedings.neurips.cc/paper/2019/hash/2c601ad9d2ff9bc8b282670cdd54f69f-Abstract.html.
17. Xue, J., Li, J., Yu, D., Seltzer, M., and Gong, Y. (2014, January 4–9). Singular value decomposition based low-footprint speaker adaptation and personalization for deep neural network. Proceedings of the 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Florence, Italy.10.1109/ICASSP.2014.6854828 ↗
18. Dudziak, L., Abdelfattah, M.S., Vipperla, R., Laskaridis, S., and Lane, N.D. (2019). Shrinkml: End-to-end asr model compression using reinforcement learning. arXiv.10.21437/Interspeech.2019-2811 ↗
19. Zafrir, O., Boudoukh, G., Izsak, P., and Wasserblat, M. (2019, January 13). Q8bert: Quantized 8bit bert. Proceedings of the 2019 Fifth IEEE Workshop on Energy Efficient Machine Learning and Cognitive Computing-NeurIPS Edition (EMC2-NIPS), Vancouver, BC, Canada.10.1109/EMC2-NIPS53020.2019.00016 ↗
20. Zadeh, A.H., Edo, I., Awad, O.M., and Moshovos, A. (2020, January 17–21). Gobo: Quantizing attention-based nlp models for low latency and energy efficient inference. Proceedings of the 2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), Athens, Greece.10.1109/MICRO50266.2020.00071 ↗
21. Bondarenko, Y., Nagel, M., and Blankevoort, T. (2021). Understanding and overcoming the challenges of efficient transformer quantization. arXiv.10.18653/v1/2021.emnlp-main.627 ↗
22. Bhandare, A., Sripathi, V., Karkada, D., Menon, V., Choi, S., Datta, K., and Saletore, V. (2019). Efficient 8-bit quantization of transformer neural machine language translation model. arXiv.
23. Liu, H., Simonyan, K., and Yang, Y. (2018). Darts: Differentiable architecture search. arXiv.
24. Xie, S., Zheng, H., Liu, C., and Lin, L. (2018). SNAS: Stochastic neural architecture search. arXiv.
25. Improving speech recognition models with small samples for air traffic control systems10.1016/j.neucom.2020.08.092 ↗
26. Xue, J., Zheng, T., and Han, J. (2019, January 12–15). Convolutional Grid Long Short-Term Memory Recurrent Neural Network for Automatic Speech Recognition. Proceedings of the International Conference on Neural Information Processing, Sydney, Australia.10.1007/978-3-030-36802-9_76 ↗
27. Bahdanau, D., Cho, K., and Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. arXiv.
28. Chorowski, J., Bahdanau, D., Serdyuk, D., Cho, K., and Bengio, Y. (2015). Attention-based models for speech recognition. arXiv.
29. Exploring attention mechanisms based on summary information for end-to-end automatic speech recognition10.1016/j.neucom.2021.09.017 ↗
30. Billa, J. (2017). Improving LSTM-CTC based ASR performance in domains with limited training data. arXiv.
31. Dynabert: Dynamic bert with adaptive width and depth
32. Knowledge Distillation For CTC-based Speech Recognition Via Consistent Acoustic Representation Learning10.21437/Interspeech.2022-775 ↗
33. Winata, G.I., Cahyawijaya, S., Lin, Z., Liu, Z., and Fung, P. (2020, January 4–8). Lightweight and efficient end-to-end speech recognition using low-rank transformer. Proceedings of the ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Barcelona, Spain.10.1109/ICASSP40776.2020.9053878 ↗
34. A model compression method with matrix product operators for speech enhancement10.1109/TASLP.2020.3030495 ↗
35. Watanabe, S., Hori, T., Karita, S., Hayashi, T., and Ochiai, T. (2018, January 2–6). ESPnet: End-to-End Speech Processing Toolkit. Proceedings of the Interspeech 2018, Hyderabad, India.10.21437/Interspeech.2018-1456 ↗