Gender and Accent Biases in AI-Based Tools for Spanish: A Comparative Study between Alexa and Whisper
Source abstract
Abstract
Considering previous research indicating the presence of biases based on gender and accent in AI-based tools such as virtual assistants or automatic speech recognition (ASR) systems, this paper examines these potential biases in both Alexa and Whisper for the major Spanish accent groups. The Mozilla Common Voice dataset is employed for testing, and after evaluating tens of thousands of audio fragments, descriptive statistics are calculated. After analyzing the data disaggregated by gender and accent, it is observed that, for this dataset, in terms of means and medians, Alexa performs slightly better for female voices than for male voices, while the opposite is true for Whisper. However, these differences in both cases are not considered significant. In the case of accents, a higher Word Error Rate (WER) is observed among certain accents, suggesting bias based on the spoken Spanish accent.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Speech and dialogue systems · AI in Service Interactions · Speech Recognition and Synthesis
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Women Studies Research Institute (IUEM), University of La Laguna, 38200 La Laguna, Spain
Funding
Ministry of Science and Innovation · Fundación Caja Canarias and Fundación La Caixa
References (43)
Show source-provided references
1. Beirl, D., Yuill, N., and Rogers, Y. (2023, November 16). Using Voice Assistant Skills in Family Life. Available online: https://repository.isls.org//handle/1/1750.
2. Porcheron, M., Fischer, J.E., Reeves, S., and Sharples, S. (2018, January 21–26). Voice Interfaces in Everyday Life. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI ’18), Montreal, QC, Canada.10.1145/3173574.3174214 ↗
3. Këpuska, V., and Bohouta, G. (2018, January 8–10). Next-Generation of Virtual Personal Assistants (Microsoft Cortana, Apple Siri, Amazon Alexa and Google Home). Proceedings of the 2018 IEEE 8th Annual Computing and Communication Workshop and Conference (CCWC), Las Vegas, NV, USA.10.1109/CCWC.2018.8301638 ↗
4. Alexa, Are You Listening to Me? An Analysis of Alexa Voice Service Network Traffic10.1007/s00779-018-1174-x ↗
5. (2023, November 16). Home Assistant. Available online: https://www.home-assistant.io/.
6. Evaluación del reconocimiento de voz entre los servicios de Google y Amazon aplicado al Sistema Integrado de Seguridad ECU 911
7. Whisper (2023, November 16). Python. 2022. Reprint, OpenAI. Available online: https://github.com/openai/whisper.
8. (2023, November 16). Coqui-Ai/STT: STT—The Deep Learning Toolkit for Speech-to-Text. Training and Deploying STT Models Has Never Been So Easy. Available online: https://github.com/coqui-ai/STT.
9. Seaborn, K., and Urakami, J. (2021, January 8–13). Measuring Voice UX Quantitatively: A Rapid Review. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI EA ’21), Yokohama, Japan.10.1145/3411763.3451712 ↗
10. Worldometer (2023, November 16). Worldometer—Real Time World Statistics. Available online: http://www.worldometers.info/.
11. Ethnologue (Free Dev) (2023, November 16). What Are the Top 200 Most Spoken Languages?. Available online: https://www.ethnologue.com/insights/ethnologue200/.
12. Derechos humanos de cuarta generación: Inclusión social y democratización del conocimiento
13. UNESCO (2022, April 19). Education for All 2000–2015: Achievements and Challenges | Global Education Monitoring Report. Available online: https://www.unesco.org/gem-report/en/efa-achievements-challenges.
14. Costa-jussà, M.R., Basta, C., and Gállego, G.I. (2022). Evaluating Gender Bias in Speech Translation. arXiv.
15. Reid, K., and Williams, E.T. (November, January 30). Common Voice and accent choice: Data contributors self-describe their spoken accents in diverse ways. Proceedings of the 3rd ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO ’23), Boston, MA, USA.
16. Chen, J.Y.C., Fragomeni, G., Degen, H., and Ntoa, S. (2022). HCI International 2022—Late Breaking Papers: Interacting with eXtended Reality and Artificial Intelligence, Springer Nature. Lecture Notes in Computer Sciencel.10.1007/978-3-031-21707-4 ↗
17. Feng, S., Kudina, O., Halpern, B.M., and Scharenborg, O. (2021). Quantifying Bias in Automatic Speech Recognition. arXiv.
18. Markl, N. (2023). Language Variation, Automatic Speech Recognition and Algorithmic Bias. [Ph.D. Thesis, The University of Edinburgh].
19. Uneven success: Automatic speech recognition and ethnicity-related dialects10.1016/j.specom.2022.03.009 ↗
20. Markl, N. (2022, January 21–24). Language variation and algorithmic bias: Understanding algorithmic bias in British English automatic speech recognition. Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency (FAccT ’22), Seoul, Republic of Korea.10.1145/3531146.3533117 ↗
21. Vorvoreanu, M., Zhang, L., Huang, Y.-H., Hilderbrand, C., Steine-Hanson, Z., and Burnett, M. (2019, January 4–9). From Gender Biases to Gender-Inclusive Design: An Empirical Investigation. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI ’19), Glasgow, UK.10.1145/3290605.3300283 ↗
22. Breslin, S., and Wadhwa, B. (2018). The Wiley Handbook of Human Computer Interaction, John Wiley & Sons, Ltd.10.1002/9781118976005.ch4 ↗
23. Mulas, C.M. (2014). Speech Signals Feature Extraction Model for a Speaker’s Gender and Age Identification System. [Ph.D. Thesis, E.T.S. de Ingenieros Informáticos (UPM)]. Available online: https://oa.upm.es/33121/.
24. Discriminating Male and Female Voices: Differentiating Pitch and Gender10.1007/s10548-011-0207-9 ↗
25. Physiologic and acoustic differences between male and female voices10.1121/1.397959 ↗
26. Chela-Flores, G. (2022). Dialectología Hispánica/The Routledge Handbook of Spanish Dialectology, Routledge.
27. Mozilla (2023, November 16). Mozilla Common Voice. Available online: https://commonvoice.mozilla.org/.
28. (2023, November 14). Bias in AI: What It Is, Types, Examples & 6 Ways to Fix It in 2023. Available online: https://research.aimultiple.com/ai-bias/.
29. A discourse analysis of interactions with Alexa virtual assistant showing reproductions of gender bias
30. Oliveira, C.B., and Amaral, M.A. (2021, January 25–29). An Analysis of the Reproduction of Gender Bias in the Speech of Alexa Virtual Assistant. Proceedings of the XIII Congress of Latin American Women in Computing, San José, Costa Rica.
31. Tatman, R. (2017, January 4). Gender and Dialect Bias in YouTube’s Automatic Captions. Proceedings of the First ACL Workshop on Ethics in Natural Language Processing, Valencia, Spain.10.18653/v1/W17-1606 ↗
32. Korhonen, A., Traum, D., and Màrquez, L. (August, January 28). Mitigating Gender Bias in Natural Language Processing: Literature Review. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy.
33. Meyer, J., Rauchenstein, L., Eisenberg, J.D., and Howell, N. (2020, January 11–16). Artie Bias Corpus: An Open Dataset for Detecting Demographic Bias in Speech Applications. Proceedings of the Twelfth Language Resources and Evaluation Conference, Marseille, France. Available online: https://aclanthology.org/2020.lrec-1.796.
34. Garnerin, M., Rossato, S., and Besacier, L. (2021). Investigating the Impact of Gender Representation in ASR Training Data: A Case Study on Librispeech, Association for Computational Linguistics.10.18653/v1/2021.gebnlp-1.10 ↗
35. Amazon Alexa (2023, November 16). Amazon Alexa Voice AI | Alexa Developer Official Site. Available online: https://developer.amazon.com/enUS/alexa.html.
36. (2023, November 16). Introducing Whisper. Available online: https://openai.com/research/whisper.
37. (2023, November 16). Selenium. Available online: https://www.selenium.dev/.
38. Eren, G., and The Coqui TTS Team (2021). Coqui TTS. Python.
39. Amazon Alexa (2023, November 16). Slot Type Reference | Alexa Skills Kit. Available online: https://developer.amazon.com/en-US/docs/alexa/customskills/slot-type-reference.html.
40. Morris, A.C., Maier, V., and Green, P. (2004, January 4–8). From WER and RIL to MER and WIL: Improved Evaluation Measures for Connected Speech Recognition. Proceedings of the Interspeech, Jeju Island, Republic of Korea.10.21437/Interspeech.2004-668 ↗
41. Nacimiento-Garcia, E. Menceybencomo/Comparative-Analysis-of-Gender-and-Accent-Biases-in-Alexa-and-Whisper-forthe-Spanish-Language: Paper Version.
42. Instituto Canario de Estadística (2023, November 16). Demografía. Available online: https://www.gobiernodecanarias.org/istac/estadisticas/demografia/index.html.
43. Centro Virtual Cervantes. CVC (2023, November 16). Anuario 2023. Informe 2023. El Español en Cifras. Instituto Cervantes. Available online: https://cvc.cervantes.es/lengua/anuario/anuario_23/informes_ic/p01.htm.