Methodology for Obtaining High-Quality Speech Corpora
Source abstract
Abstract
Speech-based communication between users and machines is a very lively branch of research that covers speech recognition, synthesis, and, generally, natural language processing. Speech corpora are needed for training algorithms for human–machine communication, especially for automatic speech recognition and for speech synthesis. Generative artificial intelligence models also need corpora for training for every language implemented. Therefore, speech corpora are constantly being created. In this paper, we discuss how to create high-quality corpora. The technical parameters of the recordings and audio files are addressed, and a methodology is proposed for planning speech corpus creation with an emphasis on usability. The proposed methodology draws the attention of potential creators of speech corpora to often neglected aspects of the corpus creation process. The criteria for a quality assessment of particular components are also discussed. The author recommends not combining all quality metrics into one (or at least allowing users to adjust particular weights), as different users might be interested in different quality components. The presented guidelines lead to obtaining high-quality corpora that meet the needs of their end users and are easy to use.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Speech Recognition and Synthesis · Natural Language Processing Techniques · Literature, Language, and Rhetoric Studies
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Polish-Japanese Academy of Information Technology, Koszykowa 86, 02-008 Warsaw, Poland
Funding
Program Operacyjny Inteligentny Rozwój (Smart Growth Operational Programme) · National Centre for Research and Development in Poland · European Regional Development Fund and Polish funds
References (99)
Show source-provided references
1. Kamath, U., Liu, J., and Whitaker, J. (2019). Deep Learning for NLP and Speech Recognition, Springer Nature.10.1007/978-3-030-14596-5 ↗
2. Large-Vocabulary Continuous Speech Recognition Systems: A Look at Some Recent Advances10.1109/MSP.2012.2197156 ↗
3. Minaee, S., Mikolov, T., Nikzad, N., Chenaghlu, M., Socher, R., Amatriain, X., and Gao, J. (2024). Large Language Models: A Survey. arXiv.
4. Sahoo, P., Singh, A.K., Saha, S., Jain, V., Mondal, S., and Chadha, A. (2024). A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications. arXiv.10.1007/979-8-8688-0569-1_4 ↗
5. (2024, November 27). OpenAI Introducing ChatGPT. Available online: https://openai.com/blog/chatgpt.
6. Radford, A., Kim, J.W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, C. (2022). Robust Speech Recognition via Large-Scale Weak Supervision. arXiv.
7. Shewale, R. (2023, December 31). 32 Detailed ChatGPT Statistics—Users, Revenue and Trends. Available online: https://www.demandsage.com/chatgpt-statistics/.
8. Qi, W., Xu, X., Qian, K., Schuller, B.W., Fortino, G., and Aliverti, A. (2024). A Review of AIoT-based Human Activity Recognition: From Application to Technique. IEEE J. Biomed. Health Inform.10.1109/JBHI.2024.3406737 ↗
9. Wu, M., Nafziger, J., Scodary, A., and Maas, A. (2021). HarperValleyBank: A Domain-Specific Spoken Dialog Corpus. Workshop Mach. Learn. Speech Lang. Process., 2021.
10. Schiel, F., Draxler, C., Baumann, A., Ellbogen, T., and Steffen, A. (2024, November 27). The Production of Speech Corpora. Available online: https://epub.ub.uni-muenchen.de/13693/1/schiel_13693.pdf.
11. Schiel, F., Baumann, A., Draxler, C., Ellbogen, A., Hoole, P., and Steffen, A. (2024, November 27). The Validation of Speech Corpora. Available online: https://epub.ub.uni-muenchen.de/13698/1/schiel_13698.pdf.
12. Chen, G., Chai, S., Wang, G., Du, J., Zhang, W.-Q., Weng, C., Su, D., Povey, D., Trmal, J., and Zhang, J. (September, January 30). GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audio. Proceedings of the 22nd Annual Conference of the International Speech Communication Association INTERSPEECH 2021, Brno, Czechia.10.21437/Interspeech.2021-1965 ↗
13. MuST-C: A multilingual corpus for end-to-end speech translation10.1016/j.csl.2020.101155 ↗
14. Web-Questionnaire-Based Corpus Creation Under Assumption of Human as Speech Targets10.20965/jaciii.2022.p0513 ↗
15. CLARIN (2024, November 27). Spoken Corpora. Available online: https://www.clarin.eu/resource-families/spoken-corpora.
16. Butryna, A., Chu, S.-H.C., Demirşahin, I., Gutkin, A., Ha, L., He, F., Jansche, A., Johny, C., Katanova, A., and Kjartansson, O. (2019). Google Crowdsourced Speech Corpora and Related Open-Source Resources for Low-Resource Languages and Dialects: An Overview. Proc. Lang. Technol. All, 91–94.
17. An audio-visual corpus for multimodal automatic speech recognition10.1007/s10844-016-0438-z ↗
18. Articulation Rate in American English in a Corpus of YouTube Videos10.1177/0023830919894720 ↗
19. A multimedia speech corpus for audio visual research in virtual reality (L)10.1121/10.0001670 ↗
20. Zola, A. (2022, December 31). Crowdsourced Voice Recordings and Their Relevance for the Development of Speech Recognition Systems. Available online: https://www.clickworker.com/customer-blog/crowdsourced-voice-recordings/.
21. Summa Linguae Technologies (2024, November 27). Where to Find Speech Recognition Data: 5 Options to Consider. Available online: https://summalinguae.com/data/where-to-find-speech-data/.
22. Quality Control in Remote Speech Data Collection10.1109/JSTSP.2019.2904212 ↗
23. Publications Office of the European Union (2024, November 27). Data.Europa.eu Data Quality Guidelines. Publications Office., Available online: https://data.europa.eu/doi/10.2830/79367.
24. Noisy training for deep neural networks in speech recognition10.1186/s13636-014-0047-0 ↗
25. Pervaiz, A., Hussain, F., Israr, H., Tahir, M.A., Raja, F.R., Baloch, N.K., Ishmanov, F., and Zikria, Y.B. (2020). Incorporating Noise Robustness in Speech Command Recognition by Noise Augmentation of Training Data. Sensors, 20.10.3390/s20082326 ↗
26. Jucker, A.H., Schneider, K.P., and Bublitz, W. (2018). Corpus Annotation. Methods in Pragmatics, De Gruyter Mouton.10.1515/9783110424928 ↗
27. Kytö, M., and Lüdeling, A. (2008). Pragmatic annotation. Corpus Linguistics: An International Handbook, Mouton de Gruyter.10.1515/9783110211429 ↗
28. Flick, U. (2014). Transcription as a Crucial Step of Data Analysis. The SAGE Handbook of Qualitative Data Analysis, SAGE Publications, Inc.10.4135/9781446282243 ↗
29. Ide, N., and Pustejovsky, J. (2017). Handbook of Linguistic Annotation, Springer.10.1007/978-94-024-0881-2 ↗
30. Editorial. Speech annotation and corpus tools10.1016/S0167-6393(00)00066-2 ↗
31. Garside, R., Leech, G., and McEnery, T. (2013). Corpus Annotation: Linguistic Information from Computer Text Corpora, Taylor & Francis Group.
32. (2012). Language Resource Management—Linguistic Annotation Framework (LAF) (Standard No. ISO 24612:2012). Available online: https://www.iso.org/standard/37326.html.
33. The Importance of Reference Corpora. Hizkuntza-corpusak
34. Anderman, G., and Rogers, M. (2007). Parallel and comparable corpora: What is happening?. Incorporating Corpora: The Linguist and the Translator, Multilingual Matters.10.21832/9781853599873 ↗
35. Sharoff, S., Rapp, R., Zweigenbaum, P., and Fung, P. (2013). Overviewing Important Aspects of the Last Twenty Years of Research in Comparable Corpora. Building and Using Comparable Corpora, Springer.10.1007/978-3-642-20128-8 ↗
36. Sketch Engine (2024, November 27). Reference Corpora. Available online: https://www.sketchengine.eu/corpora-and-languages/reference-corpus/.
37. (2024, November 27). CLARIN Reference Corpora. Available online: https://www.clarin.eu/resource-families/reference-corpora.
38. Corpus-Analysis.com (2024, November 27). Tools for Corpus Linguistics. Available online: https://corpus-analysis.com/.
39. Ghyselen, A.-S., Breitbarth, A., Farasyn, M., Van Keymeulen, J., and van Hessen, A. (2020). Clearing the Transcription Hurdle in Dialect Corpus Building: The Corrichepus of Southern Dutch Dialects as Case Study. Front. Artif. Intell., 3.10.3389/frai.2020.00010 ↗
40. Representativeness in Corpus Design10.1093/llc/8.4.243 ↗
41. Sinclair, J. (2024, November 27). Corpus and Text—Basic Principles. In Developing Linguistic Corpora: A Guide to Good Practice; Wynne, M., Ed.; ISSN 1463 5194. Available online: https://users.ox.ac.uk/~martinw/dlc/index.htm.
42. Raineri, S., and Debras, C. (2019). Corpora and Representativeness: Where to go from now?. CogniTextes, 19.10.4000/cognitextes.1311 ↗
43. Wang, C., Riviere, M., Lee, A., Wu, A., Talnikar, C., Haziza, D., Williamson, M., Pino, J., and Dupoux, E. (2021, January 1–6). VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Virtual.10.18653/v1/2021.acl-long.80 ↗
44. Jia, Y., Ramanovich, M.T., Wang, Q., and Zen, H. (2022). CVSS Corpus and Massively Multilingual Speech-to-Speech Translation. arXiv.
45. Stanford Linguistics (2024, November 27). Corpus Tools. Available online: https://linguistics.stanford.edu/resourcesresources-corpora/corpus-tools.
46. (2024, November 27). CLARIN Tools. Available online: https://www.clarin.eu/content/tools.
47. Falcone, M. Corpora definition. European Commission, Ref. Ares(2017)3839168 - 31/07/2017 (2017).
48. Yin, Z., and Li, A. (November, January 1–3). A standardization program of speech corpus collection. Proceedings of the 2017 Conference of The Oriental Chapter of International Committee for Coordination and Standardization of Speech Databases and Assessment Technique (O-COCOSDA), Seoul, Republic of Korea.10.1109/ICSDA.2017.8384471 ↗
49. Grill, P., and Tučková, J. (2016). Speech Databases of Typical Children and Children with SLI. PLoS ONE, 11.10.1371/journal.pone.0150365 ↗
50. MacDonald, R.L., Jiang, P.-P., Cattiau, J., Heywood, R., Cave, R., Seaver, K., Ladewig, M.A., Tobin, J., Brenner, M.P., and Nelson, P.C. (September, January 30). Disordered Speech Data Collection: Lessons Learned at 1 Million Utterances from Project Euphonia. Proceedings of the 22nd Annual Conference of the International Speech Communication Association INTERSPEECH, Brno, Czechia.
51. The DELAD initiative for sharing language resources on speech disorders10.1007/s10579-023-09655-2 ↗
52. Kingsbury, P., Strassel, S., McLemore, C., and McIntyre, R. (1997). CALLHOME American English Transcripts LDC97T14. Linguist. Data Consort.
53. Dybkjær, L., Hemsen, H., and Minker, W. (2007). Linguistic Resources, Development, and Evaluation of Text and Speech Systems. Evaluation of Text and Speech Systems. Text, Speech and Language Technology Book Series (TLTB, Volume 37), Springer.10.1007/978-1-4020-5817-2 ↗
54. Garofolo, J.S., Lamel, L.F., Fisher, W.M., Fiscus, J.G., Pallett, D.S., Dahlgren, N.L., and Zue, V. (1993). TIMIT Acoustic-Phonetic Continuous Speech Corpus LDC93S1, Linguistic Data Consortium.
55. ELRA (2024, November 27). ELDA Vera am Mittag German Audio-Visual Emotional Speech Database. Available online: http://universal.elra.info/product_info.php?cPath=25&products_id=2064.
56. Leech, G. (2024, November 27). Adding Linguistic Annotation. In Developing Linguistic Corpora: A Guide to Good Practice; Wynne, M., Ed.; ISSN 1463 5194. Available online: https://users.ox.ac.uk/~martinw/dlc/index.htm.
57. Paquot, M., and Gries, S.T. (2022). Corpus Annotation. A Practical Handbook of Corpus Linguistics, Springer.
58. Pęzik, P. (2018, January 7–12). Increasing the Accessibility of Time-Aligned Speech Corpora with Spokes Mix. Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. Available online: https://aclanthology.org/L18-1678.
59. Baker, P., Hardie, A., and McEnery, T. (2006). A Glossary of Corpus Linguistics, Edinburgh University Press.10.1515/9780748626908 ↗
60. (2011). Systems and Software Engineering—Systems and Software Quality Requirements and Evaluation (SQuaRE)—System and Software Quality Models (Standard No. ISO/IEC 25010:2011). Available online: https://www.iso.org/standard/35733.html.
61. Lavitas, L., Redfield, O., Lee, A., Fletcher, D., Eck, M., and Janardhanan, S. (2021, January 6–14). Annotation Quality Framework—Accuracy, Credibility, and Consistency. Proceedings of the 35th Conference on Neural Information Processing Systems NeurIPS 2021, Online.
62. van den Heuvel, H., Boves, L., and Sanders, E. (2024, November 27). Validation of Content and Quality of Existing SLR: Overview and Methodology. ELRA/9901/VAL-1 Deliverable 1.1. Available online: http://www.elra.info/media/filer_public/2013/12/04/d1-1.pdf.
63. Christodoulides, G. (2018). Praaline: Integrating Tools for Speech Corpus Research. arXiv.
64. Koržinek, D., Marasek, K., and Brocki, Ł. (2016, January 26–28). Polish Read Speech Corpus for Speech Tools and Services. Proceedings of the CLARIN Annual Conference 2016, Aix-en-Provence, France.
65. McAuliffe, M., Stengel-Eskin, E., Socolof, M., and Sonderegger, M. (2017, January 20–24). Polyglot and Speech Corpus Tools: A system for representing, integrating, and querying speech corpora. Proceedings of the 18th Annual Conference of the International Speech Communication Association INTERSPEECH 2017, Stockholm, Sweden.10.21437/Interspeech.2017-1390 ↗
66. Russell, S.J., Jones, D.B., and Prys, D. (2022, January 20–25). BU-TTS: An Open-Source, Bilingual Welsh-English, Text-to-Speech Corpus. Proceedings of the 4th Celtic Language Technology Workshop Within LREC 2022, Marseille, France.
67. Sini, A., Lolive, D., Vidal, G., Tahon, M., and Delais-Roussarie, É. (2018, January 7–12). SynPaFlex-Corpus: An Expressive French Audiobooks Corpus Dedicated to Expressive Speech Synthesis. Proceedings of the 11th International Conference on Language Resources and Evaluation LREC 2018, Miyazaki, Japan.
68. Microsoft (2023, December 31). Speech Service Documentation. Available online: https://learn.microsoft.com/en-us/azure/cognitive-services/speech-service/how-to-custom-speech-evaluate-data?pivots=speech-studio.
69. Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. (2002, January 6–12). BLEU: A Method for Automatic Evaluation of Machine Translation. Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL), Philadelphia, PA, USA.10.3115/1073083.1073135 ↗
70. Do, P., Coler, M., Dijkstra, J., and Klabbers, E. (September, January 30). A Systematic Review and Analysis of Multilingual Data Strategies in Text-to-Speech for Low-Resource Languages. Proceedings of the 22nd Annual Conference of the International Speech Communication Association INTERSPEECH 2021, Brno, Czechia.10.21437/Interspeech.2021-1565 ↗
71. Kostek, B. (2022, January 28–30). Intelligent Audio Signal Processing—Do We Still Need Annotated Datasets?. Proceedings of the 14th Asian Conference on Intelligent Information and Database Systems ACIIDS, Ho Chi Minh City, Vietnam.10.1007/978-3-031-21967-2_55 ↗
72. Noh, Y., Kim, K., Lee, M., Heo, C., Jeong, Y., Jeong, Y., Hahm, Y., Oh, T., Choe, H., and Park, S. (2020, January 24–26). Enhancing Quality of Corpus Annotation: Construction of the Multi-Layer Corpus Annotation and Simplified Validation of the Corpus Annotation. Proceedings of the 34th Pacific Asia Conference on Language, Information and Computation, Hanoi, Vietnam.
73. Su, H., Ovur, S.E., Xu, Z., and Alfayad, S. (2024). Exploring the Potential of Fuzzy Sets in Cyborg Enhancement: A Comprehensive Review. IEEE Trans. Fuzzy Syst.10.1109/TFUZZ.2024.3491733 ↗
74. Basile, V., Fell, M., Fornaciari, T., Hovy, D., Paun, S., Plank, B., Poesio, M., and Uma, A. (2021, January 5–6). We Need to Consider Disagreement in Evaluation. Proceedings of the 1st Workshop on Benchmarking: Past, Present and Future 2021, Bangkok, Thailand.10.18653/v1/2021.bppf-1.3 ↗
75. Stefanowitsch, A. (2020). Corpus Linguistics: A Guide to the Methodology, Language Science Press.
76. Egbert, J., Biber, D., and Gray, B. (2022). Designing and Evaluating Language Corpora: A Practical Framework for Corpus Representativeness, Cambridge University Press.10.1017/9781316584880 ↗
77. de Groot, A.M.B., and Hagoort, P. (2017). Corpus linguistics. Research Methods in Psycholinguistics and the Neurobiology of Language: A Practical Guide, Wiley-Blackwell.10.1002/9781394259762 ↗
78. Hughes, T., Nakajima, K., Ha, L., Vasu, A., Moreno, P., and LeBeau, M. (2010, January 26–30). Building transcribed speech corpora quickly and cheaply for many languages. Proceedings of the 11th Annual Conference of the International Speech Communication Association INTERSPEECH 2010, Chiba, Japan.10.21437/Interspeech.2010-551 ↗
79. Gray, B., Egbert, J., and Biber, D. (2017, January 27–30). Exploring Methods for Evaluating Corpus Representativeness. Proceedings of the Corpus Linguistics International Conference 2017, St. Petersburg, UK.
80. Rapp, R. (2014, January 23–29). Using Collections of Human Language Intuitions to Measure Corpus Representativeness. Proceedings of the 25th International Conference on Computational Linguistics: Technical Papers (COLING 2014), Dublin, Ireland.
81. Survey of Word Co-occurrence Measures for Collocation Detection
82. Exploring Learner Language Through Corpora: Comparing and Interpreting Corpus Frequency Information10.1111/lang.12226 ↗
83. Toward more careful corpus statistics: Uncertainty estimates for frequencies, dispersions, association measures, and more10.1016/j.rmal.2021.100002 ↗
84. Rabadán, R. (2010). Size matters: A quantitative approach to corpus representativeness. Lengua, Traducción, Recepción. En Honor de Julio César Santoyo/Language, Translation, Reception. To Honor Julio César Santoyo, Publicaciones Universidad de León.
85. Le Bruyn, B., Fuchs, M., van der Klis, M., Liu, J., Mo, C., Tellings, J., and de Swart, H. (2022). Parallel Corpus Research and Target Language Representativeness: The Contrastive, Typological, and Translation Mining Traditions. Languages, 7.10.3390/languages7030176 ↗
86. Draxler, C., van den Heuvel, H., van Hessen, A., Calamai, S., Corti, L., and Scagliola, S. (2020, January 11–16). A CLARIN Transcription Portal for Interview Data. Proceedings of the 12th International Conference on Language Resources and Evaluation LREC, Marseille, France.
87. Voice analytics in the wild: Validity and predictive accuracy of common audio-recording devices10.3758/s13428-023-02139-9 ↗
88. (Don’t) try this at home! The effects of recording devices and software on phonetic analysis10.1353/lan.2021.0075 ↗
89. Fish, R., Hu, Q., and Boykin, S. (2024, November 27). Using Audio Quality to Predict Word Error Rate in an Automatic Speech Recognition System. Available online: https://www.mitre.org/sites/default/files/pdf/06_1154.pdf.
90. NIST (2024, November 27). NIST Speech Signal to Noise Ratio Measurements, Available online: https://www.nist.gov/itl/iad/mig/nist-speech-signal-noise-ratio-measurements.
91. (2025, February 10). Mc Squared System Design Group: The Speech Intelligibility Calculator. Available online: http://www.mcsquared.com/ssdesgnm.htm.
92. (2018). Ergonomics of Human-System Interaction—Part 11: Usability: Definitions and Concepts (Standard No. ISO 9241-11:2018(en)). Available online: https://www.iso.org/obp/ui/#iso:std:iso:9241:-11:ed-2:v1:en.
93. (2014). Systems and Software Engineering—Systems and Software Quality Requirements and Evaluation (SQuaRE)—Guide to SQuaRE (Standard No. ISO/IEC 25000:2014(en)). Available online: https://www.iso.org/obp/ui/#iso:std:iso-iec:25000:ed-2:v1:en.
94. (2019). Software Engineering. Software Product Quality Requirements and Evaluation (SQuaRE). Common Industry Format (CIF) for Usability Test Reports (Standard No. ISO/IEC 25062:2006). Available online: https://www.iso.org/standard/43046.html.
95. Six, J.M., and Macefield, R. (2024, November 27). How to Determine the Right Number of Participants for Usability Studies. UXmatters. Available online: https://www.uxmatters.com/mt/archives/2016/01/how-to-determine-the-right-number-of-participants-for-usability-studies.php.
96. Vare, K., Gerassimenko, O., and Drude, S. (2024, November 27). CLARIN+ D3.2 Usability Report. Available online: https://office.clarin.eu/v/CE-2016-0939-CLARINPLUS-D3_2.pdf.
97. Bakhturina, E., Lavrukhin, V., and Ginsburg, B. (2021, January 6–14). A Toolbox for Construction and Analysis of Speech Datasets. Proceedings of the 35th Conference on Neural Information Processing Systems NeurIPS 2021, Datasets and Benchmarks Track, Online.
98. CORAA ASR: A large corpus of spontaneous and prepared speech manually validated for speech recognition in Brazilian Portuguese10.1007/s10579-022-09621-4 ↗
99. Charpentier, F., and Stella, M. (1986, January 7–11). Diphone synthesis using an overlap-add technique for speech waveforms concatenation. Proceedings of the ICASSP ′86—IEEE International Conference on Acoustics, Speech, and Signal Processing, Tokyo, Japan.10.1109/ICASSP.1986.1168657 ↗