A Survey on Bias in Deep NLP
Source abstract
Abstract
Deep neural networks are hegemonic approaches to many machine learning areas, including natural language processing (NLP). Thanks to the availability of large corpora collections and the capability of deep architectures to shape internal language mechanisms in self-supervised learning processes (also known as “pre-training”), versatile and performing models are released continuously for every new network design. These networks, somehow, learn a probability distribution of words and relations across the training collection used, inheriting the potential flaws, inconsistencies and biases contained in such a collection. As pre-trained models have been found to be very useful approaches to transfer learning, dealing with bias has become a relevant issue in this new scenario. We introduce bias in a formal way and explore how it has been treated in several networks, in terms of detection and correction. In addition, available resources are identified and a strategy to deal with bias in deep NLP is proposed.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Natural Language Processing Techniques · Adversarial Robustness in Machine Learning
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Centro de Estudios Avanzados en TIC (CEATIC), 230071 Jaén, Spain
Funding
Spanish Government
References (89)
Show source-provided references
1. Trust and Bias in Robots: These elements of artificial intelligence present ethical challenges, which scientists are trying to solve10.1511/2019.107.2.86 ↗
2. A field study of the impact of gender and user’s technical experience on the performance of voice-activated medical tracking application10.1016/j.ijhcs.2003.09.005 ↗
3. Extracting semantic representations from word co-occurrence statistics: A computational study10.3758/BF03193020 ↗
4. Stubbs, M. (1996). Text and Corpus Analysis: Computer-Assisted Studies of Language and Culture, Blackwell.
5. Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., and Galstyan, A. (2019). A Survey on Bias and Fairness in Machine Learning. arXiv.
6. Dissecting racial bias in an algorithm used to manage the health of populations10.1126/science.aax2342 ↗
7. Kendall, A., and Gal, Y. (2017). What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?. arXiv.
8. Tolan, S., Miron, M., Gómez, E., and Castillo, C. (2019, January 17–21). Why machine learning may lead to unfairness: Evidence from risk assessment for juvenile justice in catalonia. Proceedings of the Seventeenth International Conference on Artificial Intelligence and Law, Montreal, QC, Canada.10.1145/3322640.3326705 ↗
9. Xu, J., Ju, D., Li, M., Boureau, Y.L., Weston, J., and Dinan, E. (2020). Recipes for Safety in Open-domain Chatbots. arXiv.
10. Semantics derived automatically from language corpora contain human-like biases10.1126/science.aal4230 ↗
11. Measuring individual differences in implicit cognition: The implicit association test10.1037/0022-3514.74.6.1464 ↗
12. Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I.D., and Gebru, T. (2018, January 23–24). Model Cards for Model Reporting. Proceedings of the Conference on Fairness, Accountability, and Transparency, New York, NY, USA.10.1145/3287560.3287596 ↗
13. Data Statements for Natural Language Processing: Toward Mitigating System Bias and Enabling Better Science10.1162/tacl_a_00041 ↗
14. Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., and Askell, A. (2020). Language Models are Few-Shot Learners. arXiv.
15. Language models are unsupervised multitask learners
16. Abid, A., Farooqi, M., and Zou, J. (2021). Persistent Anti-Muslim Bias in Large Language Models. arXiv.10.1145/3461702.3462624 ↗
17. On the psychology of prediction10.1037/h0034747 ↗
18. Gigerenzer, G. (2008). Bounded and rational. Philosophie: Grundlagen und Anwendungen/Philosophy: Foundations and Applications, Mentis.
19. Haselton, M.G., Nettle, D., and Murray, D.R. (2015). The evolution of cognitive bias. The Handbook of Evolutionary Psychology, John Wiley& Sons.10.1002/9781119125563.evpsych241 ↗
20. Schneider, D.J. (2005). The Psychology of Stereotyping, Guilford Press.
21. Gajane, P., and Pechenizkiy, M. (2017). On Formalizing Fairness in Prediction with Machine Learning. arXiv.
22. Verma, S., and Rubin, J. (2018, January 29). Fairness definitions explained. Proceedings of the 2018 IEEE/ACM International Workshop on Software Fairness (Fairware), Gothenburg, Sweden.10.1145/3194770.3194776 ↗
23. Pre-trained models for natural language processing: A survey10.1007/s11431-020-1647-3 ↗
24. Baader, F., Horrocks, I., and Sattler, U. (2004). Description logics. Handbook on Ontologies, Springer.10.1007/978-3-540-24750-0_1 ↗
25. Antoniou, G., and Van Harmelen, F. (2004). Web ontology language: Owl. Handbook on Ontologies, Springer.10.1007/978-3-540-24750-0_4 ↗
26. Bolukbasi, T., Chang, K.W., Zou, J.Y., Saligrama, V., and Kalai, A. (2016). Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings. arXiv.
27. Pennington, J., Socher, R., and Manning, C.D. (2014, January 25–29). GloVe: Global Vectors for Word Representation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar.10.3115/v1/D14-1162 ↗
28. Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013). Efficient Estimation of Word Representations in Vector Space. arXiv.
29. Lauscher, A., and Glavaš, G. (2019, January 6–7). Are We Consistently Biased? Multidimensional Analysis of Biases in Distributional Word Vectors. Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM 2019), Minneapolis, MN, USA.10.18653/v1/S19-1010 ↗
30. Lauscher, A., Glavas, G., Ponzetto, S.P., and Vulic, I. (2020). A General Framework for Implicit and Explicit Debiasing of Distributional Word Vector Spaces. arXiv.10.1609/aaai.v34i05.6325 ↗
31. Lauscher, A., Takieddin, R., Ponzetto, S.P., and Glavaš, G. (2020, January 12). AraWEAT: Multidimensional Analysis of Biases in Arabic Word Embeddings. Proceedings of the Fifth Arabic Natural Language Processing Workshop, Barcelona, Spain.
32. Gonen, H., and Goldberg, Y. (2019). Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Them. arXiv.10.18653/v1/N19-1061 ↗
33. Zhao, J., Zhou, Y., Li, Z., Wang, W., and Chang, K.W. (November, January 31). Learning Gender-Neutral Word Embeddings. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium.10.18653/v1/D18-1521 ↗
34. Jentzsch, S., Schramowski, P., Rothkopf, C., and Kersting, K. (2019, January 27–28). Semantics Derived Automatically from Language Corpora Contain Human-like Moral Choices. Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society (AIES’19), New York, NY, USA.10.1145/3306618.3314267 ↗
35. Swinger, N., De-Arteaga, M., Heffernan, N.T., Leiserson, M.D., and Kalai, A.T. (2019, January 27–28). What Are the Biases in My Word Embedding?. Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, New York, NY, USA.10.1145/3306618.3314270 ↗
36. Dev, S., and Phillips, J. (2019, January 16–18). Attenuating Bias in Word vectors. Proceedings of the The 22nd International Conference on Artificial Intelligence and Statistics, Okinawa, Japan.
37. Zhao, J., Wang, T., Yatskar, M., Ordonez, V., and Chang, K. (2018). Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods. arXiv.10.18653/v1/N18-2003 ↗
38. Manzini, T., Yao Chong, L., Black, A.W., and Tsvetkov, Y. (2019, January 2–7). Black is to Criminal as Caucasian is to Police: Detecting and Removing Multiclass Bias in Word Embeddings. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, MN, USA.10.18653/v1/N19-1062 ↗
39. Zhou, P., Shi, W., Zhao, J., Huang, K.H., Chen, M., and Chang, K.W. (2019). Analyzing and Mitigating Gender Bias in Languages with Grammatical Gender and Bilingual Word Embeddings, ACL.10.18653/v1/D19-1531 ↗
40. Conneau, A., Lample, G., Ranzato, M., Denoyer, L., and Jégou, H. (2018). Word Translation Without Parallel Data. arXiv.
41. Escudé Font, J., and Costa-jussà, M.R. (2019, January 1–2). Equalizing Gender Bias in Neural Machine Translation with Word Embeddings Techniques. Proceedings of the First Workshop on Gender Bias in Natural Language Processing, Florence, Italy.10.18653/v1/W19-3821 ↗
42. Ziemski, M., Junczys-Dowmunt, M., and Pouliquen, B. (2016, January 23–28). The United Nations Parallel Corpus v1.0. Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), Portorož, Slovenia.
43. Koehn, P. (2019, October 01). Europarl: A Parallel Corpus for Statistical Machine Translation. Available online: https://www.statmt.org/europarl/.
44. Papineni, K., Roukos, S., Ward, T., and Zhu, W.J. (2002, January 7–12). Bleu: A Method for Automatic Evaluation of Machine Translation. Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Philadelphia, PA, USA.10.3115/1073083.1073135 ↗
45. Vig, J. (2019). A Multiscale Visualization of Attention in the Transformer Model. arXiv.10.18653/v1/P19-3007 ↗
46. Dev, S., Li, T., Phillips, J.M., and Srikumar, V. (2020). OSCaR: Orthogonal Subspace Correction and Rectification of Biases in Word Embeddings. arXiv.10.18653/v1/2021.emnlp-main.411 ↗
47. Bhardwaj, R., Majumder, N., and Poria, S. (2020). Investigating Gender Bias in Bert. arXiv.10.1007/s12559-021-09881-2 ↗
48. Chaloner, K., and Maldonado, A. (2019, January 1–2). Measuring Gender Bias in Word Embeddings across Domains and Discovering New Gender Bias Word Categories. Proceedings of the First Workshop on Gender Bias in Natural Language Processing, Florence, Italy.10.18653/v1/W19-3804 ↗
49. Mind the GAP: A Balanced Corpus of Gendered Ambiguous Pronouns10.1162/tacl_a_00240 ↗
50. May, C., Wang, A., Bordia, S., Bowman, S.R., and Rudinger, R. (2019, January 2–7). On Measuring Social Biases in Sentence Encoders. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, MN, USA.10.18653/v1/N19-1063 ↗
51. Tan, Y., and Celis, L. (2019). Assessing Social and Intersectional Biases in Contextualized Word Representations. arXiv.
52. Hall Maudslay, R., Gonen, H., Cotterell, R., and Teufel, S. (2019, January 3–7). It’s All in the Name: Mitigating Gender Bias with Name-Based Counterfactual Data Substitution. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China.10.18653/v1/D19-1530 ↗
53. Nadeem, M., Bethke, A., and Reddy, S. (2020). StereoSet: Measuring stereotypical bias in pretrained language models. arXiv.10.18653/v1/2021.acl-long.416 ↗
54. Guo, W., and Çalişkan, A. (2020). Detecting Emergent Intersectional Biases: Contextualized Word Embeddings Contain a Distribution of Human-like Biases. arXiv.10.1145/3461702.3462536 ↗
55. Hidden Gender Bias in Big Data as Revealed by Neural Networks: Man is to Woman as Work is to Mother?
56. Boratto, L., Faralli, S., Marras, M., and Stilo, G. (2020). Mitigating Gender Bias in Machine Learning Data Sets. Bias and Social Aspects in Search and Recommendation, Springer International Publishing.10.1007/978-3-030-52485-2 ↗
57. Bartl, M., Nissim, M., and Gatt, A. (2020, January 12–13). Unmasking Contextual Stereotypes: Measuring and Mitigating BERT’s Gender Bias. Proceedings of the Second Workshop on Gender Bias in Natural Language Processing, Florence, Italy.
58. Rudinger, R., Naradowsky, J., Leonard, B., and Van Durme, B. (2018, January 1–6). Gender Bias in Coreference Resolution. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, New Orleans, LA, USA.10.18653/v1/N18-2002 ↗
59. Clark, K., and Manning, C. (2016). Deep Reinforcement Learning for Mention-Ranking Coreference Models. arXiv.10.18653/v1/D16-1245 ↗
60. Lu, K., Mardziel, P., Wu, F., Amancharla, P., and Datta, A. (2018). Gender Bias in Neural Natural Language Processing. arXiv.
61. Lee, K., He, L., Lewis, M., and Zettlemoyer, L. (2017, January 7–11). End-to-end Neural Coreference Resolution. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark.10.18653/v1/D17-1018 ↗
62. Clark, K., and Manning, C.D. (2016, January 1–5). Deep Reinforcement Learning for Mention-Ranking Coreference Models. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, TX, USA.10.18653/v1/D16-1245 ↗
63. Zhao, J., Wang, T., Yatskar, M., Cotterell, R., Ordonez, V., and Chang, K.W. (2019, January 2–7). Gender Bias in Contextualized Word Embeddings. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, MN, USA.10.18653/v1/N19-1064 ↗
64. McGuffie, K., and Newhouse, A. (2020). The Radicalization Risks of GPT-3 and Advanced Neural Language Models. arXiv.
65. GPT-3: Its Nature, Scope, Limits, and Consequences10.1007/s11023-020-09548-1 ↗
66. Farkas, A., and N’emeth, R. (2020). How to Measure Gender Bias in Machine Translation: Optimal Translators, Multiple Reference Points. arXiv.
67. Badjatiya, P., Gupta, M., and Varma, V. (2019, January 13–17). Stereotypical Bias Removal for Hate Speech Detection Task using Knowledge-based Generalizations. Proceedings of the World Wide Web Conference, San Francisco, CA, USA.10.1145/3308558.3313504 ↗
68. De-Arteaga, M., Romanov, A., Wallach, H., Chayes, J., Borgs, C., Chouldechova, A., Geyik, S., Kenthapadi, K., and Kalai, A.T. (2019, January 29–31). Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting. Proceedings of the Conference on Fairness, Accountability, and Transparency, Atlanta, GA, USA.10.1145/3287560.3287572 ↗
69. Heindorf, S., Scholten, Y., Engels, G., and Potthast, M. (2019, January 13–17). Debiasing Vandalism Detection Models at Wikidata. Proceedings of the World Wide Web Conference, San Francisco, CA, USA.10.1145/3308558.3313507 ↗
70. Zuckerman, M., and Last, M. (2019, January 3–7). Using Graphs for Word Embedding with Enhanced Semantic Relations. Proceedings of the Thirteenth Workshop on Graph-Based Methods for Natural Language Processing (TextGraphs-13), Hong Kong, China.10.18653/v1/D19-5305 ↗
71. Peng, X., Li, S., Frazier, S., and Riedl, M. (2020, January 7–10). Reducing Non-Normative Text Generation from Language Models. Proceedings of the 13th International Conference on Natural Language Generation, Dublin, Ireland.10.18653/v1/2020.inlg-1.43 ↗
72. Kiritchenko, S., and Mohammad, S. (2018, January 5–6). Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systems. Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, New Orleans, LA, USA.10.18653/v1/S18-2005 ↗
73. Assessing gender bias in machine translation: A case study with google translate
74. Sheng, E., Chang, K.W., Natarajan, P., and Peng, N. (2019, January 3–7). The Woman Worked as a Babysitter: On Biases in Language Generation. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China.10.18653/v1/D19-1339 ↗
75. Hutto, C., and Gilbert, E. (2015, January 26–29). VADER: A Parsimonious Rule-based Model for Sentiment Analysis of Social Media Text. Proceedings of the International AAAI Conference on Web and Social Media, Oxford, UK.
76. Stanovsky, G., Smith, N.A., and Zettlemoyer, L. (August, January 28). Evaluating Gender Bias in Machine Translation. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy.
77. Ott, M., Edunov, S., Grangier, D., and Auli, M. (November, January 31). Scaling Neural Machine Translation. Proceedings of the Third Conference on Machine Translation: Research Papers, Brussels, Belgium.
78. Basta, C., Costa-jussà, M.R., and Casas, N. (2019, January 1–2). Evaluating the Underlying Gender Bias in Contextualized Word Embeddings. Proceedings of the First Workshop on Gender Bias in Natural Language Processing, Florence, Italy.10.18653/v1/W19-3805 ↗
79. Groenwold, S., Ou, L., Parekh, A., Honnavalli, S., Levy, S., Mirza, D., and Wang, W.Y. (2020, January 16–20). Investigating African-American Vernacular English in Transformer-Based Text Generation. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), online.10.18653/v1/2020.emnlp-main.473 ↗
80. Blodgett, S.L., Green, L., and O’Connor, B. (2016, January 1–5). Demographic Dialectal Variation in Social Media: A Case Study of African-American English. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, TX, USA.10.18653/v1/D16-1120 ↗
81. Babaeianjelodar, M., Lorenz, S., Gordon, J., Matthews, J.N., and Freitag, E. (2020). Quantifying Gender Bias in Different Corpora. Companion Proceedings of the Web Conference 2020, ACM.10.1145/3366424.3383559 ↗
82. Iandola, F., Shaw, A., Krishna, R., and Keutzer, K. (2020, January 11). SqueezeBERT: What can computer vision teach NLP about efficient neural networks?. Proceedings of the SustaiNLP: Workshop on Simple and Efficient Natural Language Processing, online.10.18653/v1/2020.sustainlp-1.17 ↗
83. Hutchinson, B., Prabhakaran, V., Denton, E., Webster, K., Zhong, Y., and Denuyl, S. (2020, January 5–10). Social Biases in NLP Models as Barriers for Persons with Disabilities. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, online.10.18653/v1/2020.acl-main.487 ↗
84. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2019, January 2–7). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, MN, USA.
85. Davis, J. (2021, January 01). Gender Bias In Machine Translation. Available online: https://towardsdatascience.com/gender-bias-in-machine-translation-819ddce2c452.
86. Johnson, M. (2020, August 01). Providing Gender-Specific Translations in Google Translate. Google AI Blog. Available online: https://ai.googleblog.com/2018/12/providing-gender-specific-translations.html.
87. Johnson, M. (2020, September 01). A Scalable Approach to Reducing Gender Bias in Google Translate. Google AI Blog. Available online: https://ai.googleblog.com/2020/04/a-scalable-approach-to-reducing-gender.html.
88. Gupta, M., and Agrawal, P. (2020). Compression of Deep Learning Models for Text: A Survey. arXiv.
89. Hooker, S., Moorosi, N., Clark, G., Bengio, S., and Denton, E. (2020). Characterising Bias in Compressed Models. arXiv.