Audio-Text Multimodal Speech Recognition via Dual-Tower Architecture for Mandarin Air Traffic Control Communications
Source abstract
Abstract
In air traffic control communications (ATCC), misunderstandings between pilots and controllers could result in fatal aviation accidents. Fortunately, advanced automatic speech recognition technology has emerged as a promising means of preventing miscommunications and enhancing aviation safety. However, most existing speech recognition methods merely incorporate external language models on the decoder side, leading to insufficient semantic alignment between speech and text modalities during the encoding phase. Furthermore, it is challenging to model acoustic context dependencies over long distances due to the longer speech sequences than text, especially for the extended ATCC data. To address these issues, we propose a speech-text multimodal dual-tower architecture for speech recognition. It employs cross-modal interactions to achieve close semantic alignment during the encoding stage and strengthen its capabilities in modeling auditory long-distance context dependencies. In addition, a two-stage training strategy is elaborately devised to derive semantics-aware acoustic representations effectively. The first stage focuses on pre-training the speech-text multimodal encoding module to enhance inter-modal semantic alignment and aural long-distance context dependencies. The second stage fine-tunes the entire network to bridge the input modality variation gap between the training and inference phases and boost generalization performance. Extensive experiments demonstrate the effectiveness of the proposed speech-text multimodal speech recognition method on the ATCC and AISHELL-1 datasets. It reduces the character error rate to 6.54% and 8.73%, respectively, and exhibits substantial performance gains of 28.76% and 23.82% compared with the best baseline model. The case studies indicate that the obtained semantics-aware acoustic representations aid in accurately recognizing terms with similar pronunciations but distinctive semantics. The research provides a novel modeling paradigm for semantics-aware speech recognition in air traffic control communications, which could contribute to the advancement of intelligent and efficient aviation safety management.
Source: openalex_inverted_index · Quality: baseline
Research context
OpenAlex topics: Speech Recognition and Synthesis · Speech and Audio Processing · Natural Language Processing Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
References (78)
Show source-provided references
1. Spoken instruction understanding in air traffic control: Challenge, technique, and application10.3390/aerospace8030065 ↗
2. Aviation communication: The effect of critical factors on the rate of misunderstandings10.1108/AEAT-02-2022-0052 ↗
3. Subjective measures of communication errors between pilots and air traffic controllers10.1016/j.jairtraman.2023.102461 ↗
4. Effect of fatigue in air traffic controllers’ workload, situation awareness, and control strategy
5. Issues and solutions in air-traffic infrastructure and flow management for sustainable aviation growth: A literature review10.1504/WRITR.2019.103286 ↗
6. BERTraffic: BERT-based joint speaker role and speaker change detection for air traffic control communications
7. Prediction and extraction of tower controller commands for speech recognition applications10.1016/j.jairtraman.2021.102089 ↗
8. Automatic speech recognition and understanding for radar label maintenance support increases safety and reduces air traffic controllers’ workload
9. Validating automatic speech recognition and understanding for pre-filling radar labels—increasing safety while reducing air traffic controllers’ workload10.3390/aerospace10060538 ↗
10. Improving CTC-based ASR models with gated interlayer collaboration
11. ETEH: Unified attention-based end-to-end ASR and KWS architecture10.1109/TASLP.2022.3161159 ↗
12. Recent advances in end-to-end automatic speech recognition10.1561/116.00000050 ↗
13. Streaming end-to-end speech recognition for hybrid RNN-T/attention architecture
14. Boosting low-resource speech recognition in air traffic communication via pretrained feature aggregation and multi-task learning
15. Improving speech recognition models with small samples for air traffic control systems10.1016/j.neucom.2020.08.092 ↗
16. A unified framework for multilingual speech recognition in air traffic control systems10.1109/TNNLS.2020.3015830 ↗
17. Towards multilingual end-to-end speech recognition for air traffic control10.1049/itr2.12094 ↗
18. P. Fan, D. Guo, J. Zhang, B. Yang, and Y. Lin, “Enhancing multilingual speech recognition in air traffic control by sentence-level language identification,” arXiv preprint arXiv:2305.00170, 2023.10.1016/j.apacoust.2024.110123 ↗
19. Noise-robust speech recognition with 10 minutes unparalleled in-domain data
20. Gradient remedy for multi-task learning in end-to-end noise-robust speech recognition
21. Adversarial multi-task learning with inverse mapping for speech enhancement10.1016/j.asoc.2022.108568 ↗
22. Robust data2vec: Noise-robust speech representation learning for ASR by combining regression and improved contrastive learning
23. A joint speech enhancement and self-supervised representation learning framework for noise-robust speech recognition10.1109/TASLP.2023.3275033 ↗
24. L. Cheng et al., “Improving speaker diarization using semantic information: Joint pairwise constraints propagation,” arXiv preprint arXiv:2309.10456, 2023.
25. Using large language models to shape social robots’ speech
26. A context-aware language model to improve the speech recognition in air traffic control10.3390/aerospace8110348 ↗
27. R. Cabrera, X. Liu, M. Ghodsi, Z. Matteson, E. Weinstein, and A. Kannan, “Language model fusion for streaming end to end speech recognition,” arXiv preprint arXiv:2104.04487, 2021.
28. On the similarities of representations in artificial and brain neural networks for speech recognition10.3389/fncom.2022.1057439 ↗
29. Disentangling syntax and semantics in the brain with deep networks
30. A review on multimodal zero-shot learning
31. Self-supervised speech representation learning: A review10.1109/JSTSP.2022.3207050 ↗
32. Multimodal co-learning: Challenges, applications with datasets, recent advances and future directions10.1016/j.inffus.2021.12.003 ↗
33. JOIST: A joint speech and text streaming model for ASR
34. Fast end-to-end speech recognition via non-autoregressive models and cross-modal knowledge transferring from BERT10.1109/TASLP.2021.3082299 ↗
35. Context-and knowledge-aware graph convolutional network for multimodal emotion recognition10.1109/MMUL.2022.3173430 ↗
36. CTNet: Conversational transformer network for emotion recognition10.1109/TASLP.2021.3049898 ↗
37. SPLAT: Speech-language joint pre-training for spoken language understanding
38. Pre-training for spoken language understanding with joint textual and phonetic representation learning
39. STEMM: Self-learning with speech-text manifold mixup for speech translation
40. SpeechBERT: An audio-and-text jointly learned language model for end-to-end spoken question answering
41. Wav-BERT: Cooperative acoustic and linguistic representation learning for low-resource speech recognition
42. Multimodal cross- and self-attention network for speech emotion recognition
43. SpeechT5: Unified-modal encoder-decoder pre-training for spoken language processing
44. C. Meng, J. Ao, T. Ko, M. Wang, and H. Li, “CoBERT: Self-supervised speech representation learning through code representation learning,” arXiv preprint arXiv:2210.04062, 2022.10.21437/Interspeech.2023-1390 ↗
45. SpeechUT: Bridging speech and text with hidden-unit for encoder-decoder based speech-text pre-training
46. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups10.1109/MSP.2012.2205597 ↗
47. Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks
48. The speechtransformer for large-scale Mandarin Chinese speech recognition
49. Attention-based models for speech recognition
50. Vision transformers are parameter-efficient audio-visual learners
51. Deep audio-visual learning: A survey10.1007/s11633-021-1293-0 ↗
52. X. Zhou et al., “MMSpeech: Multi-modal multi-task encoder-decoder pre-training for speech recognition,” arXiv preprint arXiv:2212.00500, 2022.10.21437/Interspeech.2023-791 ↗
53. St-BERT: Cross-modal language model pre-training for end-to-end spoken language understanding
54. Multimodal learning with transformers: A survey10.1109/TPAMI.2023.3275156 ↗
55. What makes multi-modal learning better than single (provably)
56. Efficiently fusing pretrained acoustic and linguistic encoders for low-resource speech recognition10.1109/LSP.2021.3071668 ↗
57. MAESTRO: Matched speech text representations through modality matching
58. MAESTRO-U: Leveraging joint speech-text representation learning for zero supervised speech ASR
59. A general multi-task learning framework to leverage text data for speech to text tasks
60. Unified speech-text pre-training for speech translation and recognition
61. A. Bapna et al., “SLAM: A unified encoder for speech and language modeling via speech-text joint pre-training,” arXiv preprint arXiv:2110.10329, 2021.
62. Automatic speech recognition benchmark for air-traffic communications
63. How does pre-trained wav2vec 2.0 perform on domain-shifted ASR? An extensive benchmark on air traffic control communications
64. ATCSpeechNet: A multilingual end-to-end speech recognition framework for air traffic control systems10.1016/j.asoc.2021.107847 ↗
65. Improved CTC-attention based end-to-end speech recognition on air traffic control
66. Speech GAU: A single head attention for Mandarin speech recognition for air traffic control10.3390/aerospace9080395 ↗
67. Contextual semi-supervised learning: An approach to leverage air-surveillance and untranscribed ATC data in ASR systems
68. I. Nigmatulina, R. Braun, J. Zuluaga-Gomez, and P. Motlicek, “Improving callsign recognition with air-surveillance data in air-traffic communication,” arXiv preprint arXiv:2108.12156, 2021.
69. Boosting of contextual information in ASR for air-traffic call-sign recognition
70. A comparative study of speaker role identification in air traffic communication using deep learning approaches10.1145/3572792 ↗
71. Attention is all you need
72. Neural machine translation by jointly learning to align and translate
73. Y. Liu et al., “RoBERTa: A robustly optimized BERT pretraining approach,” arXiv preprint arXiv:1907.11692, 2019.
74. AISHELL-1: An open-source mandarin speech corpus and a speech recognition baseline
75. Adam: A method for stochastic optimization
76. BERT: Pre-training of deep bidirectional transformers for language understanding
77. Decoupled weight decay regularization
78. SGDR: Stochastic gradient descent with warm restarts