Phoneme Recognition in Korean Singing Voices Using Self-Supervised English Speech Representations
Source abstract
Abstract
In general, it is difficult to obtain a huge, labeled dataset for deep learning-based phoneme recognition in singing voices. Studying singing voices also offers inherent challenges, compared to speech, because of the distinct variations in pitch, duration, and intensity. This paper proposes a detouring method to overcome this insufficient dataset, and applies it to the recognition of Korean phonemes in singing voices. The method started with pre-training the HuBERT, a self-supervised speech representation model, on a large-scale English corpus. The model was then adapted to the Korean speech domain with a relatively small-scale Korean corpus, in which the Korean phonemes were interpreted as similar English ones. Finally, the speech-adapted model was again trained with a tiny-scale Korean singing voice corpus for speech–singing adaptation. In the final adaptation, melodic supervision was chosen, which utilizes pitch information to improve the performance. For evaluation, the performance on multi-level error rates based on Word Error Rate (WER) was taken. Using the HuBERT-based transfer learning for adaptation improved the phoneme-level error rate of Korean speech by as much as 31.19%. Again, on singing voices by melodic supervision, it improved the rate by 0.55%. The significant improvement in speech recognition underscores the considerable potential of a model equipped with general human voice representations captured from the English corpus that can improve phoneme recognition on less target speech data. Moreover, the musical variation in singing voices is beneficial for phoneme recognition in singing voices. The proposed method could be applied to the phoneme recognition of other languages that have less speech and singing voice corpora.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Speech Recognition and Synthesis · Speech and Audio Processing · Music and Audio Processing
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Artificial Intelligence Lab, Department of Computer Science and Engineering, Jeonbuk National University, Jeonju 54896, Republic of Korea
Funding
National Research Foundation of Korea (NRF)
References (42)
Show source-provided references
1. Recent advances in end-to-end automatic speech recognition10.1561/116.00000050 ↗
2. Gulati, A., Qin, J., Chiu, C.-C., Parmar, N., Zhang, Y., Yu, J., Han, W., Wang, S., Zhang, Z., and Wu, Y. (2020). Conformer: Convolution-augmented transformer for speech recognition. arXiv.10.21437/Interspeech.2020-3015 ↗
3. Automatic speech recognition: A survey10.1007/s11042-020-10073-7 ↗
4. Demirel, E., Ahlbäck, S., and Dixon, S. (2020, January 19–24). Automatic lyrics transcription using dilated convolutional neural networks with self-attention. Proceedings of the 2020 International Joint Conference on Neural Networks (IJCNN), Glasgow, UK.10.1109/IJCNN48605.2020.9207052 ↗
5. Gupta, C., Yılmaz, E., and Li, H. (2020, January 4–9). Automatic lyrics alignment and transcription in polyphonic music: Does background music help?. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Virtual.10.1109/ICASSP40776.2020.9054567 ↗
6. Hosoya, T., Suzuki, M., Ito, A., Makino, S., Smith, L.A., Bainbridge, D., and Witten, I.H. (2005, January 11–15). Lyrics recognition from a singing voice based on finite state automaton for music information retrieval. Proceedings of the 6th International Society for Music Information Retrieval Conference (ISMIR), London, UK.
7. Fujihara, H., Goto, M., and Ogata, J. (2008, January 14–18). Hyperlinking lyrics: A method for creating hyperlinks between phrases in song lyrics. Proceedings of the 9th International Society for Music Information Retrieval Conference (ISMIR), Philadelphia, PA, USA.
8. Dzhambazov, G. (2017). Knowledge-Based Probabilistic Modeling for Tracking Lyrics in Music Audio Signals. [Ph.D. Thesis, Universitat Pompeu Fabra].
9. Yong, S., Su, L., and Nam, J. (2023, January 4–10). A phoneme-informed neural network model for note-level singing transcription. Proceedings of the ICASSP 2023–2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Rhodes Island, Greece.10.1109/ICASSP49357.2023.10096707 ↗
10. Automatic lyrics transcription of polyphonic music with lyrics-chord multi-task learning10.1109/TASLP.2022.3190742 ↗
11. Graves, A., Fernández, S., Gomez, F., and Schmidhuber, J. (2006, January 25–29). Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks. Proceedings of the 23rd International Conference on Machine Learning, Pittsburgh, PA, USA.10.1145/1143844.1143891 ↗
12. Hubert: Self-supervised speech representation learning by masked prediction of hidden units10.1109/TASLP.2021.3122291 ↗
13. Self-supervised speech representation learning: A review10.1109/JSTSP.2022.3207050 ↗
14. Huang, J., Benetos, E., and Ewert, S. (2022, January 7–13). Improving lyrics alignment through joint pitch detection. Proceedings of the ICASSP 2022–2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Virtual.10.1109/ICASSP43922.2022.9746460 ↗
15. Chung, Y.-A., Wu, C.-C., Shen, C.-H., Lee, H.-Y., and Lee, L.-S. (2016). Audio word2vec: Unsupervised learning of audio segment representations using sequence-to-sequence autoencoder. arXiv.10.21437/Interspeech.2016-82 ↗
16. Liu, A.H., Chung, Y.-A., and Glass, J. (2020). Non-autoregressive predictive coding for learning speech representations from local dependencies. arXiv.10.21437/Interspeech.2021-349 ↗
17. van den Oord, A., Li, Y., and Vinyals, O. (2018). Representa-tion learning with contrastive predictive coding. arXiv.
18. Schneider, S., Baevski, A., Collobert, R., and Auli, M. (2019). wav2vec: Unsupervised pre-training for speech recognition. arXiv.10.21437/Interspeech.2019-1873 ↗
19. wav2vec 2.0: A framework for self-supervised learning of speech representations
20. Baevski, A., Schneider, S., and Auli, M. (2019). vq-wav2vec: Self-supervised learning of discrete speech representations. arXiv.
21. Wavlm: Large-scale self-supervised pre-training for full stack speech processing10.1109/JSTSP.2022.3188113 ↗
22. Meseguer-Brocal, G., Cohen-Hadria, A., and Peeters, G. (2019). Dali: A large dataset of synchronized audio, lyrics and notes, auto-matically created using teacher-student machine learning paradigm. arXiv.10.5334/tismir.30 ↗
23. Dabike, G.R., and Barker, J. (2024, September 21). Automatic Lyric Transcription from Karaoke Vocal Tracks: Resources and a Baseline System. Interspeech, Available online: https://www.isca-archive.org/interspeech_2019/dabike19_interspeech.html.10.21437/Interspeech.2019-2378 ↗
24. Zhang, C., Yu, J., Chang, L., Tan, X., Chen, J., Qin, T., and Zhang, K. (2021). Pdaugment: Data augmentation by pitch and duration adjustments for automatic lyrics transcription. arXiv.
25. Anna, M.K. (2015, January 26–30). Training phoneme models for singing with “songified” speech data. Proceedings of the 16th International Society for Music Information Retrieval Conference (ISMIR 2015), Malaga, Spain.
26. Mesaros, A., and Virtanen, T. (2009, January 24–28). Adaptation of a speech recognizer for singing voice. Proceedings of the 2009 17th European Signal Processing Conference, Scotland, UK.
27. Mesaros, A. (2013, January 16–19). Singing voice identification and lyrics tran- scription for music information retrieval invited paper. Proceedings of the 2013 7th Conference on Speech Technology and Human-Computer Dialogue (SpeD), Cluj-Napoca, Romania.10.1109/SpeD.2013.6682644 ↗
28. Ou, L., Gu, X., and Wang, Y. (2022). Transfer learning of wav2vec 2.0 for automatic lyric transcription. arXiv.
29. Deng, T., Nakamura, E., and Yoshii, K. (2022, January 4–8). End-to-end lyrics transcription informed by pitch and onset estimation. Proceedings of the 23rd International Society for Music Information Retrieval Conference, ISMIR, Bengaluru, India.
30. A comparative analysis of korean-english phonological structures and processes for pronunciation pedagogy in interpretation training10.7202/013253ar ↗
31. O’Grady, W., Archibald, J., Aronoff, M., and Rees-Miller, J. (2017). Contemporary Linguistics: An Introduction, St. Martin’s.
32. Nichols, E., Morris, D., Basu, S., and Raphael, C. (2009, January 9–13). Relationships between lyrics and melody in popular music. Proceedings of the ISMIR 2009-11th International Society for Music Information Retrieval Conference, Utrecht, The Netherlands.
33. Panayotov, V., Chen, G., Povey, D., and Khudanpur, S. (2015, January 19–24). Librispeech: An asr corpus based on public domain audio books. Proceedings of the 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), South Brisbane, Australia.10.1109/ICASSP.2015.7178964 ↗
34. National Institute of Korean Language (2024, September 21). Nikl the Corpus of Reading Aloud Korean Short Stories by Seoul People (v.2.0), Available online: https://kli.korean.go.kr/corpus/main/requestMain.do.
35. Choi, S., Kim, W., Park, S., Yong, S., and Nam, J. (2020, January 11–16). Children’s song dataset for singing voice research. Proceedings of the International Society for Music Information Retrieval Conference (ISMIR), Montreal, Canada.
36. Kyubyong Park (2024, September 21). g2pk. Available online: https://github.com/Kyubyong/g2pk.
37. (2024, September 21). National Institute of Korean Language. ‘Revised Romanization of Korean’. Available online: https://www.korean.go.kr/front_eng/roman/roman_01.do.
38. Molina, E., Barbancho-Perez, A.M., Tardon-Garcia, L.J., and Barbancho-Perez, I. (2014, January 27–31). Evaluation framework for automatic singing transcription. Proceedings of the 15th International Society for Music Information Retrieval Conference, Taipei, Taiwan.
39. Ott, M., Edunov, S., Baevski, A., Fan, A., Gross, S., Ng, N., Grangier, D., and Auli, M. (2019, January 2–7). fairseq: A fast, extensible toolkit for sequence modeling. Proceedings of the NAACL-HLT 2019: Demonstrations, Minneapolis, MN, USA.10.18653/v1/N19-4009 ↗
40. Kingma, D.P., and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv.
41. Park, D.S., Chan, W., Zhang, Y., Chiu, C.C., Zoph, B., Cubuk, E.D., and Le, Q.V. (2019). Specaugment: A simple data augmentation method for automatic speech recognition. arXiv.10.21437/Interspeech.2019-2680 ↗
42. Domain adaptation: Challenges, methods, datasets, and applications10.1109/ACCESS.2023.3237025 ↗