Branch-Transformer: A Parallel Branch Architecture to Capture Local and Global Features for Language Identification
Source abstract
Abstract
Currently, an increasing number of people are opting to use transformer models or conformer models for language identification, achieving outstanding results. Among them, transformer models based on self-attention can only capture global information, lacking finer local details. There are also approaches that employ conformer models by concatenating convolutional neural networks and transformers to capture both local and global information. However, this static single-branch architecture is difficult to interpret and modify, and it incurs greater inference difficulty and computational costs compared to dual-branch models. Therefore, in this paper, we propose a novel model called Branch-transformer (B-transformer). In contrast to traditional transformers, it consists of parallel dual-branch structures. One branch utilizes self-attention to capture global information, while the other employs a Convolutional Gated Multi-Layer Perceptron (cgMLP) module to extract local information. We also investigate various fusion methods for integrating global and local information and experimentally validate the effectiveness of our approach on the NIST LRE 2017 dataset.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Natural Language Processing Techniques · Speech Recognition and Synthesis · Topic Modeling
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Software, Xinjiang University, Urumqi 830017, China
School of Computer Science and Technology, Xinjiang University, Urumqi 830017, China
Department of Electronic Engineering and Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing 100084, China
Funding
National Natural Science Foundation of China
References (44)
Show source-provided references
1. Rajaa, S., Anandan, K., Dalmia, S., Gupta, T., and Chng, E.S. (2023, January 4–9). Improving Spoken Language Identification with Map-Mix. Proceedings of the ICASSP 2023—2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Rhodes Island, Greece.10.1109/ICASSP49357.2023.10095765 ↗
2. Kukk, K., and Alumäe, T. (2022, January 18–22). Improving Language Identification of Accented Speech. Proceedings of the Interspeech 2022, Incheon, Republic of Korea.10.21437/Interspeech.2022-10455 ↗
3. Shen, P., Lu, X., and Kawai, H. (2022, January 18–22). Transducer-based language embedding for spoken language identification. Proceedings of the Interspeech 2022, Incheon, Republic of Korea.10.21437/Interspeech.2022-11281 ↗
4. Comparison of four approaches to automatic language identification of telephone speech10.1109/TSA.1996.481450 ↗
5. Reviewing automatic language identification10.1109/79.317925 ↗
6. Matejka, P., Zhang, L., Ng, T., Glembek, O., Ma, J., Zhang, B., and Mallidi, S.H. (2014, January 16–19). Neural Network Bottleneck Features for Language Identification. Proceedings of the Speaker and Language Recognition Workshop (Odyssey 2014), Joensuu, Finland.10.21437/Odyssey.2014-45 ↗
7. Snyder, D., Garcia-Romero, D., McCree, A., Sell, G., Povey, D., and Khudanpur, S. (2018, January 26–29). Spoken Language Recognition using X-vectors. Proceedings of the Speaker and Language Recognition Workshop (Odyssey 2018), Les Sables d’Olonne, France.10.21437/Odyssey.2018-15 ↗
8. Cai, W., Cai, Z., Zhang, X., Wang, X., and Li, M. (2018, January 15–20). A Novel Learnable Dictionary Encoding Layer for End-to-End Language Identification. Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, AB, Canada.10.1109/ICASSP.2018.8462025 ↗
9. Schneider, S., Baevski, A., Collobert, R., and Auli, M. (2019, January 15–19). wav2vec: Unsupervised Pre-Training for Speech Recognition. Proceedings of the Interspeech 2019, Graz, Austria.10.21437/Interspeech.2019-1873 ↗
10. Baevski, A., Zhou, Y., Mohamed, A., and Auli, M. (2020, January 6–12). wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations. Proceedings of the Advances in Neural Information Processing Systems, Online.
11. Conneau, A., Baevski, A., Collobert, R., Mohamed, A., and Auli, M. (September, January 30). Unsupervised Cross-Lingual Representation Learning for Speech Recognition. Proceedings of the Interspeech 2021, Brno, Czech Republic.10.21437/Interspeech.2021-329 ↗
12. Efficient Self-Supervised Learning Representations for Spoken Language Identification10.1109/JSTSP.2022.3201445 ↗
13. Liu, H., Garcia Perera, L.P., Khong, A., Styles, S., and Khudanpur, S. (2022, January 18–22). PHO-LID: A Unified Model Incorporating Acoustic-Phonetic and Phonotactic Information for Language Identification. Proceedings of the Interspeech 2022, Incheon, Republic of Korea.10.21437/Interspeech.2022-354 ↗
14. Cai, W., Cai, D., Huang, S., and Li, M. (2019, January 12–17). Utterance-level end-to-end language identification using attention-based CNN-BLSTM. Proceedings of the ICASSP 2019—2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brighton, UK.10.1109/ICASSP.2019.8682386 ↗
15. Miao, X., McLoughlin, I., and Yan, Y. (2019, January 15–19). A New Time-Frequency Attention Mechanism for TDNN and CNN-LSTM-TDNN, with Application to Language Identification. Proceedings of the Interspeech, Graz, Austria.10.21437/Interspeech.2019-1256 ↗
16. Wan, L., Sridhar, P., Yu, Y., Wang, Q., and Moreno, I.L. (2019, January 12–17). Tuplemax Loss for Language Identification. Proceedings of the ICASSP 2019—2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brighton, UK.10.1109/ICASSP.2019.8683313 ↗
17. Luo, Q., and Zhou, R. (2023). Multi-Scale Channel Adaptive Time-Delay Neural Network and Balanced Fine-Tuning for Arabic Dialect Identification. Appl. Sci., 13.10.3390/app13074233 ↗
18. Li, Z., Zhao, Z., Wang, W., Zhang, P., and Zhao, Q. (2023). Explore Long-Range Context Features for Speaker Verification. Appl. Sci., 13.10.3390/app13031340 ↗
19. Cai, W., Chen, J., and Li, M. (2018, January 26–29). Exploring the Encoding Layer and Loss Function in End-to-End Speaker and Language Recognition System. Proceedings of the Speaker and Language Recognition Workshop (Odyssey 2018), Les Sables d’Olonne, France.10.21437/Odyssey.2018-11 ↗
20. Gonzalez-Dominguez, J., Lopez-Moreno, I., Sak, H., Gonzalez-Rodriguez, J., and Moreno, P.J. (2014, January 14–18). Automatic language identification using long short-term memory recurrent neural networks. Proceedings of the Interspeech 2014, Singapore.10.21437/Interspeech.2014-483 ↗
21. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep Residual Learning for Image Recognition. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
22. Desplanques, B., Thienpondt, J., and Demuynck, K. (2020, January 25–29). ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification. Proceedings of the Interspeech 2020, Shanghai, China.10.21437/Interspeech.2020-2650 ↗
23. Alumäe, T., Kukk, K., Le, V.B., Barras, C., Messaoudi, A., and Ben Kheder, W. (2023, January 20–24). Exploring the Impact of Pretrained Models and Web-Scraped Data for the 2022 NIST Language Recognition Evaluation. Proceedings of the Interspeech 2023, Dublin, Ireland.10.21437/Interspeech.2023-1790 ↗
24. Villalba, J., Borgstrom, J., Jahan, M., Kataria, S., Garcia, L.P., Torres-Carrasquillo, P., and Dehak, N. (2023, January 20–24). Advances in Language Recognition in Low Resource African Languages: The JHU-MIT Submission for NIST LRE22. Proceedings of the Interspeech 2023, Dublin, Ireland.10.21437/Interspeech.2023-1094 ↗
25. Silnova, A., Slavicek, J., Mošner, L., Klco, M., Plchot, O., Matejka, P., Peng, J., and Burget, T.S.L. (2023, December 12). ABC System Description for NIST LRE 2022. Available online: https://www.fit.vutbr.cz/research/groups/speech/publi/2022/NIST_LRE_2022_System_Description.pdf.
26. Sarni, S., Cumani, S., Siniscalchi, S.M., and Bottino, A. (2023, January 20–24). Description and analysis of the KPT system for NIST Language Recognition Evaluation 2022. Proceedings of the Interspeech 2023, Dublin, Ireland.10.21437/Interspeech.2023-155 ↗
27. Jia, F., Koluguri, N.R., Balam, J., and Ginsburg, B. (2023, January 20–24). A Compact End-to-End Model with Local and Global Context for Spoken Language Identification. Proceedings of the Interspeech 2023, Dublin, Ireland.10.21437/Interspeech.2023-2310 ↗
28. Romero, D., D’Haro, L.F., Estecha-Garitagoitia, M., and Salamea, C. (2022, January 22–27). Phonotactic Language Recognition Using A Universal Phoneme Recognizer and A Transformer Architecture. Proceedings of the ICASSP 2022—2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Singapore.10.1109/ICASSP43922.2022.9746459 ↗
29. Wang, F., Huang, L., Li, T., Hong, Q., and Li, L. (2023, January 20–24). Conformer-based Language Embedding with Self-Knowledge Distillation for Spoken Language Identification. Proceedings of the Interspeech 2023, Dublin, Ireland.10.21437/Interspeech.2023-1557 ↗
30. Peng, Z., Huang, W., Gu, S., Xie, L., Wang, Y., Jiao, J., and Ye, Q. (2021, January 10–17). Conformer: Local Features Coupling Global Representations for Visual Recognition. Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada.10.1109/ICCV48922.2021.00042 ↗
31. Peng, Y., Dalmia, S., Lane, I., and Watanabe, S. (2022, January 17–23). Branchformer: Parallel mlp-attention architectures to capture local and global context for speech recognition and understanding. Proceedings of the International Conference on Machine Learning, Baltimore, MD, USA.
32. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L.u., and Polosukhin, I. (2017, January 4–9). Attention is All you Need. Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA.
33. Sakuma, J., Komatsu, T., and Scheibler, R. (2023, January 16). MLP-Based Architecture with Variable Length Input for Automatic Speech Recognition. Available online: https://openreview.net/pdf?id=RA-zVvZLYIy.
34. Ba, J.L., Kiros, J.R., and Hinton, G.E. (2016). Layer Normalization. arXiv.
35. Dropout: A Simple Way to Prevent Neural Networks from Overfitting
36. Hendrycks, D., and Gimpel, K. (2016). Gaussian Error Linear Units (GELUs). arXiv.
37. Si, C., Yu, W., Zhou, P., Zhou, Y., Wang, X., and Yan, S. (December, January 28). Inception transformer. Proceedings of the Advances in Neural Information Processing Systems, New Orleans, LA, USA.
38. Sadjadi, S.O., Kheyrkhah, T., Tong, A., Greenberg, C., Reynolds, D., Singer, E., Mason, L., and Hernandez-Cordero, J. (2018, January 26–29). The 2017 NIST Language Recognition Evaluation. Proceedings of the Speaker and Language Recognition Workshop (Odyssey 2018), Les Sables d’Olonne, France.10.21437/Odyssey.2018-12 ↗
39. Cieri, C., Miller, D., and Walker, K. (2004, January 26–28). The Fisher Corpus: A Resource for the Next Generations of Speech-to-Text. Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC’04), Lisbon, Portugal.
40. Godfrey, J., Holliman, E., and McDaniel, J. (1992, January 23–26). SWITCHBOARD: Telephone speech corpus for research and development. Proceedings of the ICASSP-92: 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing, San Francisco, CA, USA.10.1109/ICASSP.1992.225858 ↗
41. Pratap, V., Xu, Q., Sriram, A., Synnaeve, G., and Collobert, R. (2020, January 25–29). MLS: A Large-Scale Multilingual Dataset for Speech Research. Proceedings of the Interspeech 2020, Shanghai, China.10.21437/Interspeech.2020-2826 ↗
42. Ardila, R., Branson, M., Davis, K., Kohler, M., Meyer, J., Henretty, M., Morais, R., Saunders, L., Tyers, F., and Weber, G. (2020, January 11–16). Common Voice: A Massively-Multilingual Speech Corpus. Proceedings of the Twelfth Language Resources and Evaluation Conference, Marseille, France.
43. Gales, M.J.F., Knill, K.M., Ragni, A., and Rath, S.P. (2014, January 14–16). Speech recognition and keyword spotting for low-resource languages: Babel project research at CUED. Proceedings of the 4th Workshop on Spoken Language Technologies for Under-Resourced Languages (SLTU 2014), St. Petersburg, Russia.
44. Liu, H., Perera, L.P.G., Khong, A.W.H., Dauwels, J., Styles, S.J., and Khudanpur, S. (July, January 28). Enhancing Language Identification Using Dual-Mode Model with Knowledge Distillation. Proceedings of the Speaker and Language Recognition Workshop (Odyssey 2022), Beijing, China.10.21437/Odyssey.2022-35 ↗