Code-Switching in Automatic Speech Recognition: The Issues and Future Directions
Source abstract
Abstract
Code-switching (CS) in spoken language is where the speech has two or more languages within an utterance. It is an unsolved issue in automatic speech recognition (ASR) research as ASR needs to recognise speech in bilingual and multilingual settings, where the accuracy of ASR systems declines with CS due to pronunciation variation. There are very few reviews carried out on CS, with none conducted on bilingual and multilingual CS ASR systems. This study investigates the importance of CS in bilingual and multilingual speech recognition systems. To meet the objective of this study, two research questions were formulated, which cover both the current issues and the direction of the research. Our review focuses on databases, acoustic and language modelling, and evaluation metrics. Using selected keywords, this research has identified 274 papers and selected 42 experimental papers for review, of which 24 (representing 57%) have discussed CS, while the rest look at multilingual ASR research. The selected papers cover many well-resourced and under-resourced languages, and novel techniques to manage CS in ASR systems, which are mapping, combining and merging the phone sets of the languages experimented with in the research. Our review also examines the performance of those methods. This review found a significant variation in the performance of CS in terms of word error rates, indicating an inconsistency in the ability of ASRs to handle CS. In the conclusion, we suggest several future directions that address the issues identified in this review.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Speech Recognition and Synthesis · Speech and Audio Processing · Speech and dialogue systems
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Faculty of Computer Science and Information Technology, Universiti Malaya, Kuala Lumpur 50603, Malaysia
Faculty of Business Finance and Information Technology, MAHSA University, Jenjarom 42610, Malaysia
Cardiff School of Technologies, Cardiff Metropolitan University, Llandaff Campus, Western Avenue, Cardiff CF5 2YB, UK
Department of Biomedical Engineering, Faculty of Engineering, Universiti Malaya, Kuala Lumpur 50603, Malaysia
Faculty of Management, BR1018, Multimedia University, Persiaran Multimedia, Cyberjaya 63100, Malaysia
Funding
Ministry of Higher Education under the Fundamental Research Grant Scheme
References (66)
Show source-provided references
1. Classification of Dysarthric Speech According to the Severity of Impairment: An Analysis of Acoustic Features10.1109/ACCESS.2021.3053335 ↗
2. Modipa, T.I., and Davel, M.H. (2015, January 26–27). Predicting vowel substitution in code-switched speech. Proceedings of the Pattern Recognition Association of South Africa and Robotics and Mechatronics International Conference (PRASA-RobMech), Port Elizabeth, South Africa.10.1109/RoboMech.2015.7359515 ↗
3. Shen, H.-P., Wu, C.-H., Yang, Y.-T., and Hsu, C.-S. (2011, January 26–28). CECOS: A Chinese-English code-switching speech database. Proceedings of the International Conference on Speech Database and Assessments (Oriental COCOSDA), Hsinchu, Taiwan.10.1109/ICSDA.2011.6085992 ↗
4. Yılmaz, E., van den Heuvel, H., and Van Leeuwen, D. (2016, January 13–16). Code-switching detection using multilingual DNNs. Proceedings of the IEEE Spoken Language Technology Workshop (SLT), San Diego, CA, USA.10.1109/SLT.2016.7846326 ↗
5. Yue, X., Lee, G., Yılmaz, E., Deng, F., and Li, H. (2019, January 14–18). End-to-end code-switching ASR for low-resourced language pairs. Proceedings of the IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), Singapore.10.1109/ASRU46091.2019.9004035 ↗
6. Nakayama, S., Tjandra, A., Sakti, S., and Nakamura, S. (2018, January 18–21). Speech chain for semi-supervised learning of Japanese-English code-switching ASR and TTS. Proceedings of the IEEE Spoken Language Technology Workshop (SLT), Athens, Greece.10.1109/SLT.2018.8639674 ↗
7. Automatic speech recognition: Systematic literature review10.1109/ACCESS.2021.3112535 ↗
8. Adaptation algorithms for neural network-based speech recognition: An overview10.1109/OJSP.2020.3045349 ↗
9. Feature extraction and classification techniques for speech recognition: A review
10. Sarma, M., and Sarma, K.K. (2015). Acoustic modeling of speech signal using artificial neural network: A review of techniques and current trends. Intelligent Applications for Heterogeneous System Modeling and Design, IGI Global.10.4018/978-1-4666-8493-5.ch012 ↗
11. Automatic speech recognition errors detection and correction: A review10.1016/j.procs.2018.03.005 ↗
12. Machine learning in automatic speech recognition: A survey10.1080/02564602.2015.1010611 ↗
13. Machine learning paradigms for speech recognition: An overview10.1109/TASL.2013.2244083 ↗
14. An overview of noise-robust automatic speech recognition10.1109/TASLP.2014.2304637 ↗
15. Wang, D., Wang, X., and Lv, S. (2019). An overview of end-to-end automatic speech recognition. Symmetry, 11.10.3390/sym11081018 ↗
16. A survey on automatic speech recognition systems for Portuguese language and its variations10.1016/j.csl.2019.101055 ↗
17. ASRoIL: A comprehensive survey for automatic speech recognition of Indian languages10.1007/s10462-019-09775-8 ↗
18. Literature review on automatic speech recognition
19. Unsupervised automatic speech recognition: A review10.1016/j.specom.2022.02.005 ↗
20. Front end analysis of speech recognition: A review10.1007/s10772-010-9088-7 ↗
21. Automatic speech recognition: A review
22. Comparative study of automatic speech recognition techniques10.1049/iet-spr.2012.0151 ↗
23. A review on automatic speech recognition architecture and approaches
24. Difficulties in automatic speech recognition of dysarthric speakers and implications for speech-based applications used by the elderly: A literature review10.1080/10400435.2010.483646 ↗
25. Sitaram, S., Chandu, K.R., Rallabandi, S.K., and Black, A.W. (2019). A survey of code-switched speech and language processing. arXiv.
26. Nakayama, S., Tjandra, A., Sakti, S., and Nakamura, S. (2019, January 14–18). Zero-shot code-switching ASR and TTS with multilingual machine speech chain. Proceedings of the IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), Singapore.10.1109/ASRU46091.2019.9003926 ↗
27. Chen, Y.-C., Hsu, J.-Y., Lee, C.-K., and Lee, H.-Y. (2020). DARTS-ASR: Differentiable architecture search for multilingual speech recognition and adaptation. arXiv.10.21437/Interspeech.2020-1315 ↗
28. Biswas, A., Yılmaz, E., De Wet, F., van der Westhuizen, E., and Niesler, T. (2020). Semi-supervised development of ASR systems for multilingual code-switched speech in under-resourced languages. arXiv.
29. Du, C., Li, H., Lu, Y., Wang, L., and Qian, Y. (2021, January 19–22). Data augmentation for end-to-end code-switching speech recognition. Proceedings of the IEEE Spoken Language Technology Workshop (SLT), Shenzhen, China.10.1109/SLT48900.2021.9383620 ↗
30. Emond, J., Ramabhadran, B., Roark, B., Moreno, P., and Ma, M. (2018, January 18–21). Transliteration-based approaches to improve code-switched speech recognition performance. Proceedings of the IEEE Spoken Language Technology Workshop (SLT), Athens, Greece.10.1109/SLT.2018.8639699 ↗
31. Hara, S., and Nishizaki, H. (2017, January 12–15). Acoustic modeling with a shared phoneme set for multilingual speech recognition without code-switching. Proceedings of the Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), Kuala Lumpur, Malaysia.10.1109/APSIPA.2017.8282284 ↗
32. Huang, Z., Li, P., Xu, J., Zhang, P., and Yan, Y. (2021, January 24–27). Context-dependent Label Smoothing Regularization for Attention-based End-to-End Code-Switching Speech Recognition. Proceedings of the 12th International Symposium on Chinese Spoken Language Processing (ISCSLP), Hong Kong, China.10.1109/ISCSLP49672.2021.9362080 ↗
33. Imseng, D., Bourlard, H., and Garner, P.N. (2012, January 25–30). Using KL-divergence and multilingual information to improve ASR for under-resourced languages. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Kyoto, Japan.10.1109/ICASSP.2012.6289010 ↗
34. Kannan, A., Datta, A., Sainath, T.N., Weinstein, E., Ramabhadran, B., Wu, Y., Bapna, A., Chen, Z., and Lee, S. (2019). Large-scale multilingual speech recognition with a streaming end-to-end model. arXiv.10.21437/Interspeech.2019-2858 ↗
35. Lin, H., Deng, L., Yu, D., Gong, Y.-f., Acero, A., and Lee, C.-H. (2009, January 19–24). A study on multilingual acoustic modeling for large vocabulary ASR. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, Taipei, Taiwan.10.1109/ICASSP.2009.4960588 ↗
36. Liu, C., Zhang, Q., Zhang, X., Singh, K., Saraf, Y., and Zweig, G. (2019). Multilingual graphemic hybrid ASR with massive data augmentation. arXiv.
37. Pratap, V., Sriram, A., Tomasello, P., Hannun, A., Liptchinsky, V., Synnaeve, G., and Collobert, R. (2020). Massively multilingual asr: 50 languages, 1 model, 1 billion parameters. arXiv.10.21437/Interspeech.2020-2831 ↗
38. Xiao, Y., Gong, K., Zhou, P., Zheng, G., Liang, X., and Lin, L. (2020). Adversarial meta sampling for multilingual low-resource speech recognition. arXiv.10.1609/aaai.v35i16.17661 ↗
39. Yılmaz, E., Biswas, A., van der Westhuizen, E., de Wet, F., and Niesler, T. (2018). Building a unified code-switching ASR system for South African languages. arXiv.10.21437/Interspeech.2018-1966 ↗
40. Adel, H., Vu, N.T., Kraus, F., Schlippe, T., Li, H., and Schultz, T. (2013, January 26–31). Recurrent neural network language modeling for code switching conversational speech. Proceedings of the 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, Vancouver, BC, Canada.10.1109/ICASSP.2013.6639306 ↗
41. Exploration of end-to-end framework for code-switching speech recognition task: Challenges and enhancements10.1109/ACCESS.2020.2986255 ↗
42. Chinese-English phone set construction for code-switching ASR using acoustic and DNN-extracted articulatory features10.1109/TASLP.2014.2310353 ↗
43. Guidelines for conducting systematic mapping studies in software engineering: An update10.1016/j.infsof.2015.03.007 ↗
44. Tong, S., Garner, P.N., and Bourlard, H. (2017). Multilingual training and cross-lingual adaptation on CTC-based acoustic model. arXiv.10.1016/j.specom.2018.09.001 ↗
45. Tüske, Z., Schlüter, R., and Ney, H. (2013, January 25–29). Multilingual hierarchical MRASTA features for ASR. Proceedings of the Interspeech, Lyon, France.10.21437/Interspeech.2013-523 ↗
46. Zhou, S., Xu, S., and Xu, B. (2018). Multilingual end-to-end speech recognition with a single transformer on low-resource languages. arXiv.
47. Yilmaz, E., McLaren, M., van den Heuvel, H., and van Leeuwen, D.A. (2017, January 16–20). Language diarization for semi-supervised bilingual acoustic model training. Proceedings of the IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), Okinawa, Japan.10.1109/ASRU.2017.8268921 ↗
48. Barik, R.E., and Lestari, D.P. (2019, January 20–21). Text corpus and acoustic model addition for Indonesian-Arabic code-switching in automatic speech recognition system. Proceedings of the International Conference of Advanced Informatics: Concepts, Theory and Applications (ICAICTA), Yogyakarta, Indonesia.10.1109/ICAICTA.2019.8904183 ↗
49. Masekwameng, M.S., Mokgonyane, T.B., Modipa, T.I., Manamela, M.J., and Mogale, M.M. (2020, January 6–7). Effects of Language Modelling for Sepedi-English Code-Switched Speech in Automatic Speech Recognition System. Proceedings of the International Conference on Artificial Intelligence, Big Data, Computing and Data Communication Systems (icABCD), Durban, South Africa.10.1109/icABCD49160.2020.9183865 ↗
50. Shah, S., and Sitaram, S. (2019, January 8–11). Using monolingual speech recognition for spoken term detection in code-switched hindi-english speech. Proceedings of the 2019 International Conference on Data Mining Workshops (ICDMW), Beijing, China.10.1109/ICDMW48858.2019.9024755 ↗
51. Li, K., Li, J., Ye, G., Zhao, R., and Gong, Y. (2019, January 12–17). Towards code-switching ASR for end-to-end CTC models. Proceedings of the ICASSP 2019—2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brighton, UK.10.1109/ICASSP.2019.8683223 ↗
52. Seki, H., Watanabe, S., Hori, T., Le Roux, J., and Hershey, J.R. (2018, January 15–20). An end-to-end language-tracking speech recognizer for mixed-language speech. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, AB, Canada.10.1109/ICASSP.2018.8462180 ↗
53. Shan, C., Weng, C., Wang, G., Su, D., Luo, M., Yu, D., and Xie, L. (2019, January 12–17). Investigating end-to-end speech recognition for mandarin-english code-switching. Proceedings of the ICASSP 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brighton, UK.10.1109/ICASSP.2019.8682850 ↗
54. Vu, N.T., Lyu, D.-C., Weiner, J., Telaar, D., Schlippe, T., Blaicher, F., Chng, E.-S., Schultz, T., and Li, H. (2012, January 25–30). A first speech recognition system for Mandarin-English code-switch conversational speech. Proceedings of the 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Kyoto, Japan.10.1109/ICASSP.2012.6289015 ↗
55. Lee, D., Kim, D., Yun, S., and Kim, S. (2021). Phonetic Variation Modeling and a Language Model Adaptation for Korean English Code-Switching Speech Recognition. Appl. Sci., 11.10.3390/app11062866 ↗
56. Mabokela, K.R. (2019, January 21–22). A multilingual ASR of Sepedi-English code-switched speech for automatic language identification. Proceedings of the 2019 International Multidisciplinary Information Technology and Engineering Conference (IMITEC), Vanderbijlpark, South Africa.10.1109/IMITEC45504.2019.9015903 ↗
57. Lin, H., Deng, L., Droppo, J., Yu, D., and Acero, A. (2008, January 12–13). Learning methods in multilingual speech recognition. Proceedings of the Proc. NIPS, Vancouver, BC, Canada.
58. Song, X., Zou, Y., Huang, S., Chen, S., and Liu, Y. (2017, January 5–7). Investigating multi-task learning for automatic speech recognition with code-switching between Mandarin and English. Proceedings of the 2017 International Conference on Asian Language Processing (IALP), Singapore.10.1109/IALP.2017.8300538 ↗
59. Biswas, A., de Wet, F., van der Westhuizen, E., Yilmaz, E., and Niesler, T. (2018, January 2–6). Multilingual Neural Network Acoustic Modelling for ASR of Under-Resourced English-isiZulu Code-Switched Speech. Proceedings of the Interspeech, Hyderabad, India.10.21437/Interspeech.2018-1711 ↗
60. Tong, S., Garner, P.N., and Bourlard, H. (2019, January 12–17). An investigation of multilingual ASR using end-to-end LF-MMI. Proceedings of the ICASSP 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brighton, UK.10.1109/ICASSP.2019.8683338 ↗
61. Toshniwal, S., Sainath, T.N., Weiss, R.J., Li, B., Moreno, P., Weinstein, E., and Rao, K. (2018, January 15–20). Multilingual speech recognition with a single end-to-end model. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, AB, Canada.10.1109/ICASSP.2018.8461972 ↗
62. Miiller, M., Stiiker, S., and Waibel, A. (2018, January 15–20). Multilingual adaptation of RNN based ASR systems. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, AB, Canada.10.1109/ICASSP.2018.8461614 ↗
63. Song, T., Xu, Q., Ge, M., Wang, L., Shi, H., Lv, Y., Lin, Y., and Dang, J. (2022). Language-specific Characteristic Assistance for Code-switching Speech Recognition. arXiv.10.21437/Interspeech.2022-11426 ↗
64. Developing an HMM-Based Speech Synthesis System for Malay: A Comparison of Iterative and Isolated Unit Training10.1587/transinf.E97.D.1273 ↗
65. Emotional speech acoustic model for Malay: Iterative versus isolated unit training10.1121/1.4818741 ↗
66. Huang, Z., Wang, P., Wang, J., Miao, H., Xu, J., and Zhang, P. (2021). mproving Transformer Based End-to-End Code-Switching Speech Recognition Using Language Identification. Appl. Sci., 11.10.3390/app11199106 ↗