Joint On-Demand Pruning and Online Distillation in Automatic Speech Recognition Language Model Optimization
Source abstract
Abstract
Automatic speech recognition (ASR) systems have emerged as indispensable tools across a wide spectrum of applications, ranging from transcription services to voice-activated assistants. To enhance the performance of these systems, it is important to deploy efficient models capable of adapting to diverse deployment conditions. In recent years, on-demand pruning methods have obtained significant attention within the ASR domain due to their adaptability in various deployment scenarios. However, these methods often confront substantial tradeoffs, particularly in terms of unstable accuracy when reducing the model size. To address challenges, this study introduces two crucial empirical findings. Firstly, it proposes the incorporation of an online distillation mechanism during on-demand pruning training, which holds the promise of maintaining more consistent accuracy levels. Secondly, it proposes the utilization of the Mogrifier long short-term memory (LSTM) language model (LM), an advanced iteration of the conventional LSTM LM, as an effective alternative for pruning targets within the ASR framework. Through rigorous experimentation on the ASR system, employing the Mogrifier LSTM LM and training it using the suggested joint on-demand pruning and online distillation method, this study provides compelling evidence. The results exhibit that the proposed methods significantly outperform a benchmark model trained solely with on-demand pruning methods. Impressively, the proposed strategic configuration successfully reduces the parameter count by approximately 39%, all the while minimizing trade-offs.
Source: openalex_inverted_index · Quality: baseline
Research context
OpenAlex topics: Speech Recognition and Synthesis · Speech and dialogue systems · Topic Modeling
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
References (65)
Show source-provided references
1. Wav2vec 2.0: A framework for self-supervised learning of speech representations
2. Transformer transducer: A streamable speech recognition model with transformer encoders and RNN-T loss
3. Conformer: Convolution-augmented transformer for speech recognition
4. Residual language model for end-to-end speech recognition
5. WavLM: Large-scale self-supervised pre-training for full stack speech processing10.1109/JSTSP.2022.3188113 ↗
6. Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks
7. Sequence transduction with recurrent neural networks
8. Listen, attend and spell: A neural network for large vocabulary conversational speech recognition
9. Speech-transformer: A no-recurrence sequence-to-sequence model for speech recognition
10. Towards end-to-end speech recognition with recurrent neural networks
11. A. Hannun, C. Case, J. Casper, B. Catanzaro, G. Diamos et al., “Deep speech: Scaling up end-to-end speech recognition,” arXiv:1412.5567, 2014.
12. Shallow-fusion end-to-end contextual biasing
13. Long short-term memory10.1162/neco.1997.9.8.1735 ↗
14. RDNN: Rumor detection neural network for veracity analysis in social media text
15. Multivariate congestion prediction using stacked LSTM autoencoder based bidirectional LSTM model
16. Attention is all you need
17. Transformer-XL: Attentive language models beyond a fixed-length context
18. On-device neural net inference with mobile GPUs
19. AI benchmark: All about deep learning on smartphones in 2019
20. Conformer-based on-device streaming speech recognition with KD compression and two-pass architecture
21. Slimmable neural networks
22. Universally slimmable networks and improved training techniques
23. Dynamic slimmable network
24. Deep networks with stochastic depth
25. Reducing transformer depth on demand with structured dropout
26. Layer pruning on demand with intermediate CTC
27. On-demand compute reduction with stochastic wav2vec 2.0
28. Learning both weights and connections for efficient neural network
29. Blockdrop: Dynamic inference paths in residual networks
30. Rethinking the value of network pruning
31. Filter contribution recycle: Boosting model pruning with small norm filters
32. Multi-classification sensitive image detection method based on lightweight convolutional neural network
33. G. Melis, “Circling back to recurrent models of language,” arXiv:2211.01848, 2022.
34. Mogrifier LSTM
35. Improvements to n-gram language model using text generated from neural language model
36. A neural probabilistic language model
37. Learning continuous word embedding with metadata for question retrieval in community question answering
38. Recurrent neural network based language model
39. Investigating bidirectional recurrent neural network language models for speech recognition
40. On the difficulty of training recurrent neural networks
41. Bidirectional recurrent neural network language models for automatic speech recognition
42. An improved Elman neural network with piecewise weighted gradient for time series prediction10.1016/j.neucom.2019.06.001 ↗
43. Sharp nearby, fuzzy far away: How neural language models use context
44. Show, attend and tell: Neural image caption generation with visual attention
45. Character-level language modeling with deeper self-attention
46. Syntax-aware transformer encoder for neural machine translation
47. BERT: Pre-training of deep bidirectional transformers for language understanding
48. A text generation and prediction system: Pre-training on new corpora using BERT and GPT-2
49. Distilling the knowledge in a neural network
50. Patient knowledge distillation for bert model compression
51. TinyBERT: Distilling bert for natural language understanding
52. Lightweight single image super-resolution convolution neural network in portable device
53. Knowledge distillation by on-the-fly native ensemble
54. Deep mutual learning
55. Online knowledge distillation via collaborative learning
56. Joint structured pruning and dense knowledge distillation for efficient transformer model compression10.1016/j.neucom.2021.05.084 ↗
57. Progressive multi-level distillation learning for pruning network10.1007/s40747-023-01036-0 ↗
58. PADA: Pruning assisted domain adaptation for self-supervised speech representations
59. Structured pruning of self-supervised pre-trained models for speech recognition and understanding
60. DPHuBERT: Joint distillation and pruning of self-supervised speech models
61. S. Merity, “Single headed attention RNN: Stop thinking with your head,” arXiv:1911.11423, 2019.
62. LibriSpeech: An ASR corpus based on public domain audio books
63. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing
64. D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv:1412.6980, 2014.
65. M. Ott, S. Edunov, A. Baevski, A. Fan, S. Gross et al., “Fairseq: A fast, extensible toolkit for sequence modeling,” arXiv:1904.01038, 2019.10.18653/v1/N19-4009 ↗