Improving End-to-End Models for Children’s Speech Recognition
Source abstract
Abstract
Children’s Speech Recognition (CSR) is a challenging task due to the high variability in children’s speech patterns and limited amount of available annotated children’s speech data. We aim to improve CSR in the often-occurring scenario that no children’s speech data is available for training the Automatic Speech Recognition (ASR) systems. Traditionally, Vocal Tract Length Normalization (VTLN) has been widely used in hybrid ASR systems to address acoustic mismatch and variability in children’s speech when training models on adults’ speech. Meanwhile, End-to-End (E2E) systems often use data augmentation methods to create child-like speech from adults’ speech. For adult speech-trained ASRs, we investigate the effectiveness of augmentation methods; speed perturbations and spectral augmentation, along with VTLN, in an E2E framework for the CSR task, comparing these across Dutch, German, and Mandarin. We applied VTLN at different stages (training/test) of the ASR and conducted age and gender analyses. Our experiments showed highly similar patterns across the languages: Speed Perturbations and Spectral Augmentation yield significant performance improvements, while VTLN provided further improvements while maintaining recognition performance on adults’ speech (depending on when it is applied). Additionally, VTLN showed performance improvement for both male and female speakers and was particularly effective for younger children.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Speech Recognition and Synthesis · Speech and Audio Processing · Music and Audio Processing
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Multimedia Computing Group, Delft University of Technology (TU Delft), 2628 XE Delft, The Netherlands
Funding
TU Delft
References (78)
Show source-provided references
1. Creating conversational interfaces for children10.1109/89.985544 ↗
2. (2023, November 22). SoapBox Labs: Speech Technology for Kids. Available online: https://www.soapboxlabs.com/.
3. Potamianos, A., Narayanan, S., and Lee, S. (1997, January 22–25). Automatic speech recognition for children. Proceedings of the Eurospeech, Rhodes, Greece.10.21437/Eurospeech.1997-623 ↗
4. Robust recognition of children’s speech10.1109/TSA.2003.818026 ↗
5. Qian, M. (2021). Computer Analysis of Children’s Non-Native English Speech for Language Learning and Assessment. [Ph.D. Thesis, University of Birmingham].
6. Acoustics of children’s speech: Developmental changes of temporal and spectral parameters10.1121/1.426686 ↗
7. Modeling children’s vocal tracts10.1121/1.2017178 ↗
8. Articulatory model for the study of speech production10.1121/1.1913427 ↗
9. Ritvo, D., Bavitz, C., Gupta, R., and Oberman, I. (2013). Privacy and Children’s Data—An Overview of the Children’s Online Privacy Protection Act and the Family Educational Rights and Privacy Act, Berkman Center Research Publication.
10. Voigt, P., and Von dem Bussche, A. (2017). The EU General Data Protection Regulation (GDPR): A Practical Guide, Springer International Publishing. [1st ed.].10.1007/978-3-319-57959-7 ↗
11. End-to-End neural systems for automatic children speech recognition: An empirical study
12. Kim, D., Umesh, M.G.S., Hain, T., and Woodland, P. (2004, January 4–8). Using VTLN for broadcast news transcription. Proceedings of the International Conference on Spoken Language Processing (ICSLP), Jeju Island, Republic of Korea.10.21437/Interspeech.2004-191 ↗
13. Shivakumar, P.G., Potamianos, A., Lee, S., and Narayanan, S. (2014, January 19). Improving speech recognition for children using acoustic adaptation and pronunciation modeling. Proceedings of the Workshop on Child, Computer and Interaction (WOCCI), Singapore.
14. Ghai, S., and Sinha, R. (2009, January 6–10). Exploring the role of spectral smoothing in context of children’s speech recognition. Proceedings of the Interspeech, Brighton, UK.10.21437/Interspeech.2009-209 ↗
15. Kathania, H.K., Shahnawazuddin, S., Adiga, N., and Ahmad, W. (2018, January 15–20). Role of prosodic features on Children’s Speech Recognition. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Process, (ICASSP), Calgary, AB, Canada.10.1109/ICASSP.2018.8461668 ↗
16. Singh, V.P., Sailor, H., Bhattacharya, S., and Pandey, A. (2022, January 18–22). Spectral modification based data augmentation For improving End-to-End ASR For children’s speech. Proceedings of the Interspeech, Incheon, Republic of Korea.10.21437/Interspeech.2022-11343 ↗
17. Shahnawazuddin, S., Adiga, N., Kumar, K., Poddar, A., and Ahmad, W. (2020, January 25–29). Voice conversion based data augmentation to improve children’s speech recognition in limited data scenario. Proceedings of the Interspeech, Shanghai, China.10.21437/Interspeech.2020-1112 ↗
18. Chen, G., Na, X., Wang, Y., Yan, Z., Zhang, J., Ma, S., and Wang, Y. (2020). Data augmentation for Children’s Speech Recognition - The “Ethiopian” system for the SLT 2021 children speech recognition challenge. arXiv.
19. Watanabe, S., Hori, T., Karita, S., Hayashi, T., Nishitoba, J., Unno, Y., Enrique Yalta Soplin, N., Heymann, J., Wiesner, M., and Chen, N. (2018, January 2–6). ESPnet: End-to-End Speech processing toolkit. Proceedings of the Interspeech, Hydrebad, India.10.21437/Interspeech.2018-1456 ↗
20. Karita, S., Chen, N., Hayashi, T., Hori, T., Inaguma, H., Jiang, Z., Someki, M., Soplin, N.E.Y., Yamamoto, R., and Wang, X. (2019, January 14–18). A comparative study on transformer vs RNN in speech applications. Proceedings of the Workshop on Automatic Speech Recognition and Understanding (ASRU), Sentosa, Singapore.10.1109/ASRU46091.2019.9003750 ↗
21. Ko, T., Peddinti, V., Povey, D., and Khudanpur, S. (2015, January 6–10). Audio augmentation for speech recognition. Proceedings of the Interspeech, Dresden, Germany.10.21437/Interspeech.2015-711 ↗
22. Park, D.S., Chan, W., Zhang, Y., Chiu, C.C., Zoph, B., Cubuk, E.D., and Le, Q.V. (2019, January 15–19). SpecAugment: A simple data augmentation method for automatic speech recognition. Proceedings of the Interspeech, Graz, Austria.10.21437/Interspeech.2019-2680 ↗
23. Zhang, Y., Herygers, A., Patel, T.B., Yue, Z., and Scharenborg, O. (2023, January 16–20). Exploring data augmentation in bias mitigation against non-native-accented speech. Proceedings of the Workshop on Automatic Speech Recognition and Understanding (ASRU), Taipei, Taiwan.10.1109/ASRU57964.2023.10389756 ↗
24. Miao, Y., Gowayyed, M., Na, X., Ko, T., Metze, F., and Waibel, A. (2016, January 20–25). An empirical exploration of CTC acoustic models. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Process, (ICASSP), Shanghai, China.10.1109/ICASSP.2016.7472152 ↗
25. Giuliani, D., and Gerosa, M. (2003, January 20–25). Investigating recognition of children’s speech. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Process, (ICASSP), Shanghai, China.
26. Vaghri, Z., Zermatten, J., Lansdown, G., and Ruggiero, R. (2022). Monitoring State Compliance with the UN Convention on the Rights of the Child: An Analysis of Attributes, Springer International Publishing.10.1007/978-3-030-84647-3 ↗
27. Feng, S., Kudina, O., Halpern, B.M., and Scharenborg, O. (2021). Quantifying bias in automatic speech recognition. arXiv.
28. Wikipedia (2024, February 24). Child Development. Available online: https://en.wikipedia.org/wiki/Child_development.
29. Eskenazi, M., Mostow, J., and Graff, D. (1997). The CMU Kids Corpus LDC97S63, Linguistic Data Consortium.
30. Shobaki, K., Hosom, J.P., and Cole, R.A. (2000, January 16–20). The OGI kids² speech corpus and recognizers. Proceedings of the Interspeech, Beijing, China.10.21437/ICSLP.2000-800 ↗
31. Cole, R., Hosom, P., and Pellom, B. (2006). Technical Report TR-CSLR-2006-02, University of Colorado.
32. Cole, R., and Pellom, B. (2006). Technical Report TR-CSLR-2006-03, University of Colorado.
33. Zhao, S., Singh, M., Woubie, A., and Karhila, R. (2023, January 20–24). Data augmentation for children ASR and child-adult speaker classification using voice conversion methods. Proceedings of the Interspeech, Centre Dublin, Ireland.10.21437/Interspeech.2023-702 ↗
34. Batliner, A., Blomberg, M., D’Arcy, S., Elenius, D., Giuliani, D., Gerosa, M., Hacker, C., Russell, M., Steidl, S., and Wong, M. (2005, January 4–8). The PFSTAR children’s speech corpus. Proceedings of the Interspeech, Lisbon, Portugal.10.21437/Interspeech.2005-705 ↗
35. Cosi, P., and Pellom, B.L. (2005, January 4–8). Italian Children’s Speech Recognition for advanced interactive literacy tutors. Proceedings of the Interspeech, Lisbon, Portugal.10.21437/Interspeech.2005-698 ↗
36. Cucchiarini, C., Hamme, H.V., van Herwijnen, O., and Smits, F. (2006, January 24–26). Jasmin-CGN: Extension of the spoken dutch corpus with speech of elderly people, children and non-natives in the human-machine interaction modality. Proceedings of the Language Resources and Evaluation Conference (LREC), Genoa, Italy.
37. Yu, F., Yao, Z., Wang, X., An, K., Xie, L., Ou, Z., Liu, B., Li, X., and Miao, G. (2021, January 19–22). The SLT 2021 children speech recognition challenge: Open datasets, rules and baselines. Proceedings of the IEEE Speech and Language Technology (SLT), Shenzhen, China.10.1109/SLT48900.2021.9383608 ↗
38. Rumberg, L., Gebauer, C., Ehlert, H., Wallbaum, M., Bornholt, L., Ostermann, J., and Lüdtke, U. (2022, January 18–22). kidsTALC: A Corpus of 3- to 11-year-old German children’s connected natural speech. Proceedings of the Interspeech, Incheon, Republic of Korea.10.21437/Interspeech.2022-330 ↗
39. Miller, J., Lee, S., Uchanski, R., Heidbreder, A., Richman, B., and Tadlock, J. (1996, January 7–10). Creation of two children’s speech databases. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Process, (ICASSP), Atlanta, GA, USA.
40. My Science Tutor: A conversational multimedia virtual tutor for elementary school Science10.1145/1998384.1998392 ↗
41. Gretter, R., Matassoni, M., Bannò, S., and Falavigna, D. (2020, January 11–16). TLT-school: A corpus of non native children speech. Proceedings of the Language Resources and Evaluation Conference LREC, Marseille, France.
42. Interspeech (2024, February 24). Special Session: Connecting Speech Science and Speech Technology for Children’s Speech. Available online: https://sites.google.com/view/sciencetech4childspeech-is23.
43. Interspeech (2024, February 24). Special Session: Shared Task on Automatic Speech Recognition for Non-Native Children’s Speech. Available online: https://sites.google.com/view/wocci/home/interspeech-2020-special-session.
44. Interspeech (2024, February 24). Special Session: Shared Task on Automatic Speech Recognition for Non-Native Children’s Speech. Available online: https://sites.google.com/fbk.eu/ss-is2021-nonnativechildren/.
45. Potamianos, A., and Narayanan, S. (1998, January 12–15). Spoken dialog systems for children. Proceedings of the IEEE International Conference. on Acoustics, Speech and Signal Process, (ICASSP), Seattle, WA, USA.
46. Kazemzadeh, A., You, H., Iseli, M., Jones, B., Cui, X., Heritage, M., Price, P., Anderson, E., Narayanan, S., and Alwan, A. (2005, January 4–8). TBALL data collection: The making of a young children’s speech corpus. Proceedings of the Interspeech, Lisbon, Portugal.10.21437/Interspeech.2005-462 ↗
47. Shahnawazuddin, S., Dey, A., and Sinha, R. (2016, January 8–12). Pitch-Adaptive front-end features for robust children’s ASR. Proceedings of the Interspeech, San Francisco, CA, USA.10.21437/Interspeech.2016-1020 ↗
48. Sivaraman, G., Mitra, V., Nam, H., Tiede, M., and Espy-Wilson, C. (2016, January 8–12). Vocal tract length normalization for speaker independent acoustic-to-articulatory speech inversion. Proceedings of the Interspeech, San Francisco, CA, USA.10.21437/Interspeech.2016-1399 ↗
49. Cosi, P. (2009, January 6–10). On the development of matched and mismatched Italian children’s speech recognition systems. Proceedings of the Interspeech, Brighton, UK.10.21437/Interspeech.2009-195 ↗
50. Transfer learning from adult to children for speech recognition: Evaluation, analysis and recommendations10.1016/j.csl.2020.101077 ↗
51. A formant modification method for improved ASR of children’s speech10.1016/j.specom.2021.11.003 ↗
52. Ng, S.I., Liu, W., Peng, Z., Feng, S., Huang, H.P., Scharenborg, O., and Lee, T. (2020). The CUHK-TUDelft system for the SLT 2021 children speech recognition challenge. arXiv.
53. Gelin, L., Pellegrini, T., Pinquier, J., and Daniel, M. (September, January 30). Simulating reading mistakes for child speech transformer-based phone recognition. Proceedings of the Interspeech, Brno, Czech Republic.
54. Sinha, R., and Ghai, S. (2009, January 6–10). On the use of pitch normalization for improving children’s speech recognition. Proceedings of the Interspeech, Brighton, UK.10.21437/Interspeech.2009-202 ↗
55. Jain, R., Barcovschi, A., Yiwere, M., Corcoran, P., and Cucu, H. (2023, January 20–24). Adaptation of Whisper models to child speech recognition. Proceedings of the Interspeech, Dublin, Ireland.10.21437/Interspeech.2023-935 ↗
56. Oostdijk, N. (June, January 31). The spoken Dutch corpus. Overview and first evaluation. Proceedings of the Language Resources and Evaluation Conference (LREC), Athens, Greece.
57. van Leeuwen, D.A., Kessens, J., Sanders, E., and van den Heuvel, H. (2009, January 6–10). Results of the n-best 2008 dutch speech recognition evaluation. Proceedings of the Interspeech, Brighton, UK.10.21437/Interspeech.2009-677 ↗
58. Ardila, R., Branson, M., Davis, K., Henretty, M., Kohler, M., Meyer, J., Morais, R., Saunders, L., Tyers, F.M., and Weber, G. (2020, January 11–16). Common Voice: A massively-multilingual speech corpus. Proceedings of the Language Resources and Evaluation Conference (LREC), Marseille, France.
59. Li, Z. (2023). Mitigating Regional Accent Bias in ASR Systems. [Master’s Thesis, TU Delft].
60. Sarkar, A.K., Rath, S.P., and Umesh, S. (2010, January 29–31). Vocal tract length normalization factor based speaker-cluster UBM for speaker verification. Proceedings of the National Conference on Communications (NCC), Chennai, India.10.1109/NCC.2010.5430207 ↗
61. The application of hidden Markov models in speech recognition10.1561/2000000004 ↗
62. Povey, D., Ghoshal, A., Boulianne, G., Burget, L., Glembek, O., Goel, N., Hannemann, M., Motlicek, P., Qian, Y., and Schwarz, P. (2011, January 11–15). The Kaldi speech recognition toolkit. Proceedings of the Workshop on Automatic Speech Recognition and Understanding (ASRU), Waikoloa, HI, USA.
63. Gulati, A., Qin, J., Chiu, C., Parmar, N., Zhang, Y., Yu, J., Han, W., Wang, S., Zhang, Z., and Wu, Y. (2020, January 25–29). Conformer: Convolution-augmented transformer for speech recognition. Proceedings of the Interspeech, Shanghai, China.10.21437/Interspeech.2020-3015 ↗
64. Guo, P., Boyer, F., Chang, X., Hayashi, T., Higuchi, Y., Inaguma, H., Kamo, N., Li, C., Garcia-Romero, D., and Shi, J. (2020). Recent developments on ESPnet toolkit boosted by conformer. arXiv.10.1109/ICASSP39728.2021.9414858 ↗
65. (2024, February 24). The ASR Training Recipes and Scripts. Available online: https://github.com/tanvinabpatel/E2ECSR-Methods/.
66. Radford, A., Kim, J., Xu, T., Brockman, G., McLeavey, C., and Sutskever, I. (2023, January 23–29). Robust speech recognition via large-scale weak supervision. Proceedings of the International Conference on Machine Learning (ICML), Honolulu, HI, USA.
67. Patel, T., and Scharenborg, O. (2023). Using data augmentations and VTLN to reduce bias in Dutch End-to-End speech recognition systems. arXiv.
68. Park, Y., Patwardhan, S., Visweswariah, K., and Gates, S.C. (2008, January 22–26). An empirical analysis of word error rate and keyword error rate. Proceedings of the Interspeech, Brisbane, Australia.10.21437/Interspeech.2008-537 ↗
69. (2024, February 24). WER Statistical Significance Test. Available online: https://github.com/talhanai/wer-sigtest.
70. Pallet, D., Fisher, W., and Fiscus, J. (1990, January 3–6). Tools for the analysis of benchmark speech recognition tests. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Process, (ICASSP), Albuquerque, NM, USA.
71. Towards inclusive automatic speech recognition10.1016/j.csl.2023.101567 ↗
72. Gebauer, C., Rumberg, L., Ehlert, H., Lüdtke, U., and Ostermann, J. (2023, January 20–24). Exploiting diversity of automatic transcripts from distinct speech recognition techniques for children’s speech. Proceedings of the Interspeech, Dublin, Ireland.10.21437/Interspeech.2023-926 ↗
73. Ghai, S., and Sinha, R. (2016). Adaptive Feature Truncation to Address Acoustic Mismatch in Automatic Recognition of Children’s Speech, Asia-Pacific Signal and Information Processing (APSIPA).10.1017/ATSIP.2016.16 ↗
74. Maturational constraints on language development10.1017/S0272263100009165 ↗
75. (2024, February 24). MAGICDATA Mandarin Chinese Read Speech Corpus. Available online: https://www.openslr.org/68/.
76. Bu, H., Du, J., Na, X., Wu, B., and Zheng, H. (2017, January 1–3). AIShell-1: An Open-Source Mandarin speech corpus and a speech recognition baseline. Proceedings of the Oriental COCOSDA, Seoul, Republic of Korea.10.1109/ICSDA.2017.8384449 ↗
77. Wang, D., Zhang, X., and Zhang, Z. (2015). THCHS-30: A free Chinese speech corpus. arXiv.
78. Lu, R., Shahin, M., and Ahmed, B. (2022). Improving Children’s Speech Recognition by fine-tuning self-supervised adults’ speech representations. arXiv.