AI Fairness in Data Management and Analytics: A Review on Challenges, Methodologies and Applications
Source abstract
Abstract
This article provides a comprehensive overview of the fairness issues in artificial intelligence (AI) systems, delving into its background, definition, and development process. The article explores the fairness problem in AI through practical applications and current advances and focuses on bias analysis and fairness training as key research directions. The paper explains in detail the concept, implementation, characteristics, and use cases of each method. The paper explores strategies to reduce bias and improve fairness in AI systems, reviews challenges and solutions to real-world AI fairness applications, and proposes future research directions. In addition, this study provides an in-depth comparative analysis of the various approaches, utilizing cutting-edge research information to elucidate their different characteristics, strengths, and weaknesses. The results of the comparison provide guidance for future research. The paper concludes with an overview of existing challenges in practical applications and suggests priorities and solutions for future research. The conclusions provide insights for promoting fairness in AI systems. The information reviewed in this paper is drawn from reputable sources, including leading academic journals, prominent conference proceedings, and well-established online repositories dedicated to AI fairness. However, it is important to recognize that research nuances, sample sizes, and contextual factors may create limitations that affect the generalizability of the findings.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Ethics and Social Impacts of AI · Artificial Intelligence in Healthcare and Education · Blockchain Technology Applications and Security
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
College of Communication Engineering, PLA Army Engineering University, Nanjing 210007, China
Aerospace System Engineering Shanghai, Shanghai 201109, China
References (100+)
Show source-provided references
1. Fairness and explanation in AI-informed decision making10.3390/make4020026 ↗
2. Machine learning in business process monitoring: A comparison of deep learning and classical approaches used for outcome prediction10.1007/s12599-020-00645-0 ↗
3. Deep learning in business analytics and operations research: Models, applications and managerial implications10.1016/j.ejor.2019.09.018 ↗
4. Varona, D., and Suárez, J.L. (2022). Discrimination, bias, fairness, and trustworthy AI. Appl. Sci., 12.10.3390/app12125826 ↗
5. Saghiri, A.M., Vahidipour, S.M., Jabbarpour, M.R., Sookhak, M., and Forestiero, A. (2022). A survey of Artificial Intelligence challenges: Analyzing the definitions, relationships, and evolutions. Appl. Sci., 12.10.3390/app12084054 ↗
6. Big data’s disparate impact
7. Corsello, A., and Santangelo, A. (2023). May Artificial Intelligence Influence Future Pediatric Research?—The Case of ChatGPT. Children, 10.10.3390/children10040757 ↗
8. The cost of fairness in AI: Evidence from e-commerce
9. Liu, L.T., Dean, S., Rolf, E., Simchowitz, M., and Hardt, M. (2018, January 10–15). Delayed impact of fair machine learning. Proceedings of the International Conference on Machine Learning, PMLR, Stockholm, Sweden.10.24963/ijcai.2019/862 ↗
10. Cathy, O. (2016). How Big Data Increases Inequality and Threatens Democracy, Crown Publishing Group.
11. Equality of opportunity in supervised learning
12. Trewin, S. (2018). AI fairness for people with disabilities: Point of view. arXiv.
13. Kodiyan, A.A. (2019). An overview of ethical issues in using AI systems in hiring with a case study of Amazon’s AI based hiring tool. Researchgate Prepr., 1–19.
14. Unintended consequences of biased robotic and Artificial Intelligence systems [ethical, legal, and societal issues]10.1109/MRA.2019.2926996 ↗
15. Garg, P., Villasenor, J., and Foggo, V. (2020, January 10–13). Fairness metrics: A comparative analysis. Proceedings of the 2020 IEEE International Conference on Big Data (Big Data), Atlanta, GA, USA.10.1109/BigData50022.2020.9378025 ↗
16. Revisiting Group Fairness Metrics: The Effect of Networks10.1145/3555100 ↗
17. Ezzeldin, Y.H., Yan, S., He, C., Ferrara, E., and Avestimehr, A.S. (2023, January 7–14). Fairfed: Enabling group fairness in federated learning. Proceedings of the AAAI Conference on Artificial Intelligence, Washington, DC, USA.10.1609/aaai.v37i6.25911 ↗
18. Moving beyond “algorithmic bias is a data problem”10.1016/j.patter.2021.100241 ↗
19. Amini, A., Soleimany, A.P., Schwarting, W., Bhatia, S.N., and Rus, D. (2019, January 27–28). Uncovering and mitigating algorithmic bias through learned latent structure. Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, Honolulu, HI, USA.10.1145/3306618.3314243 ↗
20. An adversarial training framework for mitigating algorithmic biases in clinical machine learning10.1038/s41746-023-00805-y ↗
21. Li, S. (2023, January 27–29). Towards Trustworthy Representation Learning. Proceedings of the 2023 SIAM International Conference on Data Mining (SDM), Minneapolis, MN, USA.10.1137/1.9781611977653.ch109 ↗
22. Creager, E., Madras, D., Jacobsen, J.H., Weis, M., Swersky, K., Pitassi, T., and Zemel, R. (2019, January 10–15). Flexibly fair representation learning by disentanglement. Proceedings of the International Conference on Machine Learning, PMLR, Long Beach, CA, USA.
23. McNamara, D., Ong, C.S., and Williamson, R.C. (2019, January 27–28). Costs and benefits of fair representation learning. Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, Honolulu, HI, USA.10.1145/3306618.3317964 ↗
24. The politics and reciprocal (re) configuration of accountability and fairness in data-driven education10.1080/17439884.2021.1986065 ↗
25. Provable Detection of Propagating Sampling Bias in Prediction Models
26. Park, J., Ellezhuthil, R.D., Isaac, J., Mergerson, C., Feldman, L., and Singh, V. (May, January 30). Misinformation Detection Algorithms and Fairness across Political Ideologies: The Impact of Article Level Labeling. Proceedings of the 15th ACM Web Science Conference 2023, Austin, TX, USA.10.1145/3578503.3583617 ↗
27. Primary error detection and minimization (PEDMIN) strategies in social cognition: A reinterpretation of confirmation bias phenomena10.1037/0033-295X.100.2.298 ↗
28. Frincke, D., Tobin, D., McConnell, J., Marconi, J., and Polla, D. (1998, January 2–8). A framework for cooperative intrusion detection. Proceedings of the 21st NIST-NCSC National Information Systems Security Conference, Arlington, VA, USA.
29. Estivill-Castro, V., and Brankovic, L. (1999). International Conference on Data Warehousing and Knowledge Discovery, Springer.
30. Bellamy, R.K., Dey, K., Hind, M., Hoffman, S.C., Houde, S., Kannan, K., Lohia, P., Martino, J., Mehta, S., and Mojsilovic, A. (2018). AI Fairness 360: An extensible toolkit for detecting, understanding, and mitigating unwanted algorithmic bias. arXiv.
31. Zhang, Y., Bellamy, R.K., Singh, M., and Liao, Q.V. (2020, January 8–13). Introduction to AI fairness. Proceedings of the Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems, Yokohama, Japan.10.1145/3334480.3375059 ↗
32. Mahoney, T., Varshney, K., and Hind, M. (2020). AI Fairness, O’Reilly Media Incorporated.
33. Mosteiro, P., Kuiper, J., Masthoff, J., Scheepers, F., and Spruit, M. (2022). Bias discovery in machine learning models for mental health. Information, 13.10.3390/info13050237 ↗
34. Trustworthy AI10.1145/3448248 ↗
35. Accountability in AI: From principles to industry-specific accreditation10.3233/AIC-210080 ↗
36. Benjamins, R., Barbado, A., and Sierra, D. (2019). Responsible AI by design in practice. arXiv.
37. Dignum, V. (2021, January 15–18). The myth of complete AI-fairness. Proceedings of the Artificial Intelligence in Medicine: 19th International Conference on Artificial Intelligence in Medicine, AIME 2021, Virtual.10.1007/978-3-030-77211-6_1 ↗
38. Silberg, J., and Manyika, J. (2019). Notes from the AI Frontier: Tackling Bias in AI (and in Humans), McKinsey Global Institute.
39. Bird, S., Kenthapadi, K., Kiciman, E., and Mitchell, M. (2019, January 11–15). Fairness-aware machine learning: Practical challenges and lessons learned. Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, Melbourne, Australia.10.1145/3289600.3291383 ↗
40. Dwork, C., Hardt, M., Pitassi, T., Reingold, O., and Zemel, R. (2012, January 8–10). Fairness through awareness. Proceedings of the 3rd Innovations in Theoretical Computer Science Conference, Cambridge, MA, USA.10.1145/2090236.2090255 ↗
41. Islam, R., Keya, K.N., Pan, S., Sarwate, A.D., and Foulds, J.R. (2023). Differential Fairness: An Intersectional Framework for Fair AI. Entropy, 25.10.3390/e25040660 ↗
42. Fairness in machine learning
43. Zafar, M.B., Valera, I., Rogriguez, M.G., and Gummadi, K.P. (2017, January 20–22). Fairness constraints: Mechanisms for fair classification. Proceedings of the Artificial Intelligence and Statistics, PMLR, Ft. Lauderdale, FL, USA.
44. Auditing fairness under unawareness through counterfactual reasoning10.1016/j.ipm.2022.103224 ↗
45. Kusner, M.J., Loftus, J., Russell, C., and Silva, R. (2017). Counterfactual fairness. Adv. Neural Inf. Process. Syst., 30.
46. Feldman, M., Friedler, S.A., Moeller, J., Scheidegger, C., and Venkatasubramanian, S. (2015, January 11–14). Certifying and removing disparate impact. Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Sydney, Australia.10.1145/2783258.2783311 ↗
47. Kearns, M., Neel, S., Roth, A., and Wu, Z.S. (2018, January 10–15). Preventing fairness gerrymandering: Auditing and learning for subgroup fairness. Proceedings of the International Conference on Machine Learning, PMLR, Stockholm, Sweden.
48. Fleisher, W. (2021, January 19–21). What’s fair about individual fairness?. Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, New York, NY, USA.10.1145/3461702.3462621 ↗
49. Mukherjee, D., Yurochkin, M., Banerjee, M., and Sun, Y. (2020, January 16–19). Two simple ways to learn individual fairness metrics from data. Proceedings of the International Conference on Machine Learning, PMLR, Copenhagen, Denmark.
50. Dwork, C., and Ilvento, C. (2018, January 23–24). Group fairness under composition. Proceedings of the 2018 Conference on Fairness, Accountability, and Transparency (FAT* 2018), New York, NY, USA.
51. Binns, R. (2020, January 27–30). On the apparent conflict between individual and group fairness. Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, Barcelona, Spain.10.1145/3351095.3372864 ↗
52. Algorithmic fairness in Artificial Intelligence for medicine and healthcare10.1038/s41551-023-01056-8 ↗
53. Beyond bias: Artificial Intelligence and social justice
54. Fair AI: Challenges and opportunities10.1007/s12599-020-00650-3 ↗
55. Incomparable punishments: How economic inequality contributes to the disparate impact of legal fines and fees
56. Wang, L., and Zhu, H. (2022, January 21–24). How are ML-Based Online Content Moderation Systems Actually Used? Studying Community Size, Local Activity, and Disparate Treatment. Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, Seoul, Republic of Korea.10.1145/3531146.3533147 ↗
57. Tom, D., and Computing, D. (2023). Eliminating Disparate Treatment in Modeling Default of Credit Card Clients, Center for Open Science. Technical Report.10.31219/osf.io/cfyzv ↗
58. On learning fairness and accuracy on multiple subgroups
59. Open data: Accountability and transparency10.1177/2053951717718853 ↗
60. Zhou, N., Zhang, Z., Nair, V.N., Singhal, H., Chen, J., and Sudjianto, A. (2021). Bias, Fairness, and Accountability with AI and ML Algorithms. arXiv.
61. User perceptions of algorithmic decisions in the personalized AI system: Perceptual evaluation of fairness, accountability, transparency, and explainability10.1080/08838151.2020.1843357 ↗
62. Sokol, K., Hepburn, A., Poyiadzi, R., Clifford, M., Santos-Rodriguez, R., and Flach, P. (2022). Fat forensics: A python toolbox for implementing and deploying fairness, accountability and transparency algorithms in predictive systems. arXiv.10.1016/j.simpa.2022.100406 ↗
63. Fairness and accountability of AI in disaster risk management: Opportunities and challenges10.1016/j.patter.2021.100363 ↗
64. AI and accessibility10.1145/3356727 ↗
65. Equity, inclusivity, and innovative digital technologies to improve adolescent and young adult health10.1016/j.jadohealth.2020.05.014 ↗
66. Bias in data-driven Artificial Intelligence systems—An introductory survey10.1002/widm.1356 ↗
67. Bias on the web10.1145/3209581 ↗
68. Improving fairness of Artificial Intelligence algorithms in Privileged-Group Selection Bias data settings10.1016/j.eswa.2021.115667 ↗
69. Analyzing the impact of missing values and selection bias on fairness10.1007/s41060-021-00259-z ↗
70. Selection bias on intellectual ability in autism research: A cross-sectional review and meta-analysis10.1186/s13229-019-0260-x ↗
71. Bolukbasi, T., Chang, K.W., Zou, J.Y., Saligrama, V., and Kalai, A.T. (2016). Man is to computer programmer as woman is to homemaker? debiasing word embeddings. Adv. Neural Inf. Process. Syst., 29.
72. A survey on bias and fairness in machine learning10.1145/3457607 ↗
73. Torralba, A., and Efros, A.A. (2011, January 20–25). Unbiased look at dataset bias. Proceedings of the CVPR, Colorado Springs, CO, USA.10.1109/CVPR.2011.5995347 ↗
74. Liao, Y., and Naghizadeh, P. (2022, January 25–29). The impacts of labeling biases on fairness criteria. Proceedings of the 10th International Conference on Learning Representations, ICLR, Virtually.
75. Predictably unequal: Understanding and addressing concerns that algorithmic clinical prediction may increase health disparities10.1038/s41746-020-0304-9 ↗
76. Zhao, J., Wang, T., Yatskar, M., Ordonez, V., and Chang, K.W. (2017). Men also like shopping: Reducing gender bias amplification using corpus-level constraints. arXiv.10.18653/v1/D17-1323 ↗
77. Yang, N., Yuan, D., Liu, C.Z., Deng, Y., and Bao, W. (2023). FedIL: Federated Incremental Learning from Decentralized Unlabeled Data with Convergence Analysis. arXiv.
78. Tripathi, S., and Musiolik, T.H. (2023). Research Anthology on Improving Medical Imaging Techniques for Analysis and Intervention, IGI Global.
79. Mashhadi, A., Kyllo, A., and Parizi, R.M. (2022). Fairness in Federated Learning for Spatial-Temporal Applications. arXiv.
80. Equivalence between dropout and data augmentation: A mathematical check10.1016/j.neunet.2019.03.013 ↗
81. How temporal and social comparisons in performance evaluation affect fairness perceptions10.1016/j.obhdp.2018.01.003 ↗
82. Asiedu, M.N., Dieng, A., Oppong, A., Nagawa, M., Koyejo, S., and Heller, K. (2023). Globalizing Fairness Attributes in Machine Learning: A Case Study on Health in Africa. arXiv.
83. Hutiri, W.T., and Ding, A.Y. (2022, January 21–24). Bias in automated speaker recognition. Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, Seoul, Republic of Korea.10.1145/3531146.3533089 ↗
84. Machine learning fairness notions: Bridging the gap with real-world applications10.1016/j.ipm.2021.102642 ↗
85. Kallus, N., and Zhou, A. (2018, January 10–15). Residual unfairness in fair machine learning from prejudiced data. Proceedings of the International Conference on Machine Learning, PMLR, Stockholm, Sweden.
86. Asynchronous Semi-Supervised Federated Learning with Provable Convergence in Edge Computing10.1109/MNET.001.2200223 ↗
87. So, W., Lohia, P., Pimplikar, R., Hosoi, A., and D’Ignazio, C. (2022, January 21–24). Beyond Fairness: Reparative Algorithms to Address Historical Injustices of Housing Discrimination in the US. Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, Seoul, Republic of Korea.10.1145/3531146.3533160 ↗
88. A review of predictive policing from the perspective of fairness10.1007/s10506-021-09286-4 ↗
89. Ensuring fairness in machine learning to advance health equity10.7326/M18-1990 ↗
90. Bias, fairness, and validity in graduate-school admissions: A psychometric perspective10.1177/17456916211055374 ↗
91. Weerts, H., Pfisterer, F., Feurer, M., Eggensperger, K., Bergman, E., Awad, N., Vanschoren, J., Pechenizkiy, M., Bischl, B., and Hutter, F. (2023). Can Fairness be Automated? Guidelines and Opportunities for Fairness-aware AutoML. arXiv.
92. Hauer, K.E., Park, Y.S., Bullock, J.L., and Tekian, A. (2023). “My Assessments Are Biased!” Measurement and Sociocultural Approaches to Achieve Fairness in Assessment in Medical Education. Acad. Med. J. Assoc. Am. Med. Coll., online ahead of print.10.1097/ACM.0000000000005245 ↗
93. Chen, Y., Mahoney, C., Grasso, I., Wali, E., Matthews, A., Middleton, T., Njie, M., and Matthews, J. (2021, January 19–21). Gender bias and under-representation in natural language processing across human languages. Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, New York, NY, USA.10.1145/3461702.3462530 ↗
94. Chai, J., and Wang, X. (2022, January 17–23). Fairness with adaptive weights. Proceedings of the International Conference on Machine Learning, PMLR, Baltimore, MD, USA.
95. Fairness in Forecasting of Observations of Linear Dynamical Systems10.1613/jair.1.14050 ↗
96. Fairdrop: Biased edge dropout for enhancing fairness in graph representation learning10.1109/TAI.2021.3133818 ↗
97. Professionalism and homophily bias: A study of Airbnb stay choice and review positivity10.1016/j.ijhm.2023.103433 ↗
98. Homophily, biased attention, and the gender gap in science10.5465/AMBPP.2019.14784abstract ↗
99. Confirmation Bias as a Mechanism to Focus Attention Enhances Signal Detection10.18564/jasss.4954 ↗
100. Kulkarni, A., Shivananda, A., and Manure, A. (2023). Introduction to Prescriptive AI: A Primer for Decision Intelligence Solutioning with Python, Springer.10.1007/978-1-4842-9568-7 ↗