Shapley Values as a Strategy for Ensemble Weights Estimation
Source abstract
Abstract
This study introduces a novel performance-based weighting scheme for ensemble learning using the Shapley value. The weighting uses the reciprocal of binary cross-entropy as a base learner’s performance metric and estimates its Shapley value to measure the overall contribution of a learner to an equally weighted ensemble of various sizes. Two variants of this strategy were empirically compared with a single monolith model and other static weighting strategies using two large banking-related datasets. A variant that discards learners with a negative Shapley value was ranked as first or at least second when constructing homogeneous ensembles, whereas for heterogeneous ensembles this strategy resulted in a better or at least similar detection performance to other weighting strategies tested. The main limitation being the computational complexity of Shapley calculations, the explored weighting strategy could be considered as a generalization of performance-based weighting.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Data Stream Mining Techniques · Time Series Analysis and Forecasting · Anomaly Detection Techniques and Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Information Systems, Faculty of Informatics, Kaunas University of Technology, 44249 Kaunas, Lithuania
References (64)
Show source-provided references
1. The information catastrophe10.1063/5.0019941 ↗
2. Federated learning: Challenges, methods, and future directions
3. The strength of weak learnability10.1007/BF00116037 ↗
4. A practical tutorial on bagging and boosting based ensembles for machine learning: Algorithms, software tools, performance study, practical perspectives and opportunities10.1016/j.inffus.2020.07.007 ↗
5. Fan, W., Stolfo, S.J., and Zhang, J. (1999, January 15–18). The application of AdaBoost for distributed, scalable and on-line learning. Proceedings of the Fifth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Diego, CA, USA.10.1145/312129.312283 ↗
6. Online ensemble learning: An empirical study10.1023/A:1025619426553 ↗
7. Street, W.N., and Kim, Y. (2001, January 26). A streaming ensemble algorithm (SEA) for large-scale classification. Proceedings of the Seventh ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA.10.1145/502512.502568 ↗
8. Wang, H., Fan, W., Yu, P.S., and Han, J. (2003, January 24). Mining concept-drifting data streams using ensemble classifiers. Proceedings of the Ninth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Washington, DC, USA.10.1145/956750.956778 ↗
9. Dynamic weighted majority: An ensemble method for drifting concepts
10. Sharing Data While Protecting Privacy in Citizen Science10.1145/2540032 ↗
11. A survey on distributed machine learning
12. Wei, J., Dai, W., Qiao, A., Ho, Q., Cui, H., Ganger, G.R., Gibbons, P.B., Gibson, G.A., and Xing, E.P. (2015, January 27–29). Managed communication and consistency for fast data-parallel iterative analytics. Proceedings of the Sixth ACM Symposium on Cloud Computing, Kohala Coast, HI, USA.10.1145/2806777.2806778 ↗
13. When federated learning meets blockchain: A new distributed learning paradigm10.1109/MCI.2022.3180932 ↗
14. Alzheimer’s Disease Neuroimaging Initiative, Building machine learning models without sharing patient data: A simulation-based analysis of distributed learning by ensembling10.1016/j.jbi.2020.103424 ↗
15. Blockchain and federated learning for privacy-preserved data sharing in industrial IoT10.1109/TII.2019.2942190 ↗
16. Chen, X., Ji, J., Luo, C., Liao, W., and Li, P. (2018, January 10). When machine learning meets blockchain: A decentralized, privacy-preserving and secure design. Proceedings of the IEEE International Conference on Big Data (Big Data), Seattle, WA, USA.10.1109/BigData.2018.8622598 ↗
17. Xu, R., Baracaldo, N., and Joshi, J. (2021). Privacy-preserving machine learning: Methods, challenges and directions. arXiv.
18. A value for n-person games
19. Rozemberczki, B., and Sarkar, R. (2021, January 1–5). The shapley value of classifiers in ensemble games. Proceedings of the 30th ACM International Conference on Information & Knowledge Management. Virtual Event, Queensland, Australia.10.1145/3459637.3482302 ↗
20. Drungilas, V., Vaičiukynas, E., Ablonskis, L., and Čeponienė, L. (2022, January 9–11). Heterogeneous Models Inference Using Hyperledger Fabric Oracles. Proceedings of the First Blockchain and Cryptocurrency Conference B2C’ 2022, Barcelona, Spain.10.3390/app11031010 ↗
21. Drungilas, V., Vaičiukynas, E., Jurgelaitis, M., Butkienė, R., and Čeponienė, L. (2021). Towards blockchain-based federated machine learning: Smart contract for model inference. Appl. Sci., 11.10.3390/app11031010 ↗
22. Rokach, L. (2019). Ensemble Learning: Pattern Classification Using Ensemble Methods, World Scientific.10.1142/11325 ↗
23. Ensemble learning: A survey10.1002/widm.1249 ↗
24. Dogan, A., and Birant, D. (2019, January 11–15). A weighted majority voting ensemble approach for classification. Proceedings of the 2019 4th International Conference on Computer Science and Engineering UBMK, Samsun, Turkey.10.1109/UBMK.2019.8907028 ↗
25. Effective and efficient pruning of meta-classifiers in a distributed data mining system
26. Optimizing ensemble weights and hyperparameters of machine learning models for regression problems
27. Explaining prediction models and individual predictions with feature contributions10.1007/s10115-013-0679-x ↗
28. A unified approach to interpreting model predictions
29. Rozemberczki, B., Watson, L., Bayer, P., Yang, H.T., Kiss, O., Nilsson, S., and Sarkar, R. (2022). The shapley value in machine learning. arXiv.10.24963/ijcai.2022/778 ↗
30. Data valuation for medical imaging using Shapley value and application to a large-scale chest X-ray dataset
31. A principled approach to data valuation for federated learning
32. Induced subgraph game for ensemble selection10.1142/S021821301760003X ↗
33. A novel GSCI-based ensemble approach for credit scoring10.1109/ACCESS.2020.3043937 ↗
34. Machine learning for credit scoring: Improving logistic regression with non-linear decision-tree effects10.1016/j.ejor.2021.06.053 ↗
35. Laaksonen, J., and Oja, E. (1996, January 3–6). Classification with learning k-nearest neighbors. Proceedings of the International Conference on Neural Networks (ICNN’96), Washington, DC, USA.10.1109/ICNN.1996.549118 ↗
36. Prognostic kalman filter based bayesian learning model for data accuracy prediction
37. Classification and regression trees10.1002/widm.8 ↗
38. (2008). Binary Logistic Regression. Best Practices in Quantitative Methods, Jason SAGE Publications, Inc.
39. Wang, Q., Ma, Y., Zhao, K., and Tian, Y. (2020). A comprehensive survey of loss functions in machine learning. Ann. Data Sci., 1–26.
40. Roth, A.E. (1988). The Shapley Value: Essays in Honor of Lloyd S. Shapley, Cambridge University Press.10.1017/CBO9780511528446 ↗
41. Machine learning for the prediction of volume responsiveness in patients with oliguric acute kidney injury in critical care10.1186/s13054-019-2411-z ↗
42. Agarap, A.F. (2018). Deep Learning using Rectified Linear Units (ReLU). arXiv.
43. Sergio, M., Laureano, R., and Cortez, P. (2011, January 24–26). Using Data Mining for Bank Direct Marketing: An Application of the CRISP-DM Methodology. Proceedings of the European Simulation and Modelling Conference-ESM’2011, Guimarães, Portugal.
44. Simplifying decision trees10.1016/S0020-7373(87)80053-6 ↗
45. van Rijn, J.N., Holmes, G., Pfahringer, B., and Vanschoren, J. (2014, January 8–10). Algorithm selection on data streams. Proceedings of the Discovery Science: 17th International Conference, Bled, Slovenia.10.1007/978-3-319-11812-3_28 ↗
46. Hsieh, K., Phanishayee, A., Mutlu, O., and Gibbons, P. (2020, January 12–18). The non-iid data quagmire of decentralized machine learning. Proceedings of the International Conference on Machine Learning, Virtual Event.
47. Semi-synchronous federated learning for energy-efficient training and accelerated convergence in cross-silo settings
48. Michieli, U., and Ozay, M. (2021). Prototype guided federated learning of visual feature representations. arXiv.
49. Arnold, S., and Yesilbas, D. (2021). Demystifying the effects of non-independence in federated learning. arXiv.
50. Joint client scheduling and resource allocation under channel uncertainty in federated learning10.1109/TCOMM.2021.3088528 ↗
51. In-edge ai: Intelligentizing mobile edge computing, caching and communication by federated learning10.1109/MNET.2019.1800286 ↗
52. Time-triggered Federated Learning over Wireless Networks10.1109/TWC.2022.3189601 ↗
53. (2023, March 15). Critical Difference Diagram with Wilcoxon-Holm Post-Hoc Analysis. Available online: https://github.com/hfawaz/cd-diagram.
54. Deep learning for time series classification: A review10.1007/s10618-019-00619-1 ↗
55. A comparison of alternative tests of significance for the problem of m rankings10.1214/aoms/1177731944 ↗
56. Should we really use post-hoc tests based on mean-ranks?
57. Individual comparisons of grouped data by ranking methods10.1093/jee/39.2.269 ↗
58. A simple sequentially rejective multiple test procedure
59. (2023, May 10). The Official Implementation of “The Shapley Value of Classifiers in Ensemble Games” (CIKM 2021). Available online: https://github.com/benedekrozemberczki/shapley.
60. (2023, May 10). Experiment Results for All Tested Ensemble Sizes and Datasets. Available online: https://github.com/HurrisLT/ShapleyWeighting.
61. Heaton, J. (April, January 30). An empirical analysis of feature engineering for predictive modeling. Proceedings of the SoutheastCon, Norfolk, VA, USA.
62. Polynomial calculation of the Shapley value based on sampling10.1016/j.cor.2008.04.004 ↗
63. Maleki, S., Tran-Thanh, L., Hines, G., Rahwan, T., and Rogers, A. (2013). Bounding the estimation error of sampling-based Shapley value approximation. arXiv.
64. Adaptive ensemble reduction and inflation