Variable Selection Using Deep Variational Information Bottleneck with Drop-Out-One Loss
Source abstract
Abstract
The information bottleneck (IB) model aims to find the optimal representations of input variables with respect to the response variable. While it has been widely used in the machine-learning community, research from the perspective of the information-theoretic method has been rarely reported regarding variable selection. In this paper, we investigate DNNs for variable selection through an information-theoretic lens. To be specific, we first state the rationality of variable selection with IB and then propose a new statistic to measure the variable importance. On this basis, a new algorithm based on a deep variational information bottleneck is developed to calculate the statistic, in which we consider the Gaussian distribution and the exponential distribution to estimate the Kullback–Leibler divergence. Empirical evaluations on simulated and real-world data show that the proposed method performs better than classical variable-selection methods. This confirms the feasibility of the variable selection from the perspective of IB.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Statistical Methods and Inference · Distributed Sensor Networks and Detection Algorithms · Bayesian Modeling and Causal Inference
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
College of Science, Huazhong Agricultural University, Wuhan 430070, China
Engineering Research Center of Intelligent Technology for Agriculture, Ministry of Education, Wuhan 430070, China
Beijing Jiyun Intelligent Technology Co., Ltd., Beijing 100096, China
School of Finance, Hubei University of Economics, Wuhan 430205, China
Hubei Financial Development and Financial Security Research Center, Hubei University of Economics, Wuhan 430205, China
Funding
Fundamental Research Funds for the Central Universities of China · Knowledge Innovation Program of Wuhan-Shuguang Project · Doctoral Scientific Research Foundation
References (35)
Show source-provided references
1. Regression shrinkage and selection via the lasso10.1111/j.2517-6161.1996.tb02080.x ↗
2. Model selection and estimation in regression with grouped variables10.1111/j.1467-9868.2005.00532.x ↗
3. Variable selection via nonconcave penalized likelihood and its oracle properties10.1198/016214501753382273 ↗
4. Sparse additive models10.1111/j.1467-9868.2009.00718.x ↗
5. Sparse modal additive model10.1109/TNNLS.2020.3005144 ↗
6. Yin, J., Chen, X., and Xing, E.P. (July, January 26). Group sparse additive models. Proceedings of the International Conference on Machine Learning, Edinburgh, Scotland.
7. Learning coordinate covariances via gradients
8. Learning sparse gradients for variable selection and dimension reduction10.1007/s10994-012-5284-9 ↗
9. Robust gradient learning with applications10.1109/TNNLS.2015.2425215 ↗
10. Liu, L., Song, B., Pan, Z., Yang, C., Xiao, C., and Li, W. (2022). Gradient learning under tilted empirical risk minimization. Entropy, 24.10.3390/e24070956 ↗
11. A knockoff filter for high-dimensional selective inference10.1214/18-AOS1755 ↗
12. Panning for gold: ‘model-X’ knockoffs for high dimensional controlled variable selection10.1111/rssb.12265 ↗
13. Breiman, L., Friedman, J.H., Olshen, R.A., and Stone, C.J. (2017). Classification and Regression Trees, Routledge.10.1201/9781315139470 ↗
14. Lemhadri, I., Ruan, F., and Tibshirani, R. (2021, January 13–15). Lassonet: Neural networks with feature sparsity. Proceedings of the International Conference on Artificial Intelligence and Statistics, Virtual.
15. Feng, J., and Simon, N. (2017). Sparse-input neural networks for high-dimensional nonparametric regression and classification. arXiv.
16. Agarwal, R., Melnick, L., Frosst, N., Zhang, X., Lengerich, B., Caruana, R., and Hinton, G.E. (2021, January 6–14). Neural additive models: Interpretable machine learning with neural nets. Proceedings of the Advances in Neural Information Processing Systems, Virtual.
17. Lu, Y., Fan, Y., Lv, J., and Stafford Noble, W. (2018, January 3–8). DeepPINK: Reproducible feature selection in deep neural networks. Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada.
18. Shwartz-Ziv, R., and Tishby, N. (2017). Opening the black box of deep neural networks via information. arXiv.
19. Tishby, N., Pereira, F.C., and Bialek, W. (2000, January 22–25). The information bottleneck method. Proceedings of the Annual Allerton Conference on Communication, Control, and Computing, Monticello, VA, USA.
20. Zaidi, A., Estella-Aguerri, I., and Shamai, S. (2020). On the information bottleneck problems: Models, connections, applications and information theoretic views. Entropy, 22.10.3390/e22020151 ↗
21. Ahuja, K., Caballero, E., Zhang, D., Gagnon-Audet, J.C., Bengio, Y., Mitliagkas, I., and Rish, I. (2021, January 6–14). Invariance principle meets information bottleneck for out-of-distribution generalization. Proceedings of the Advances in Neural Information Processing Systems, Virtual.
22. Chalk, M., Marre, O., and Tkacik, G. (2016, January 5–10). Relevant sparse codes with variational information bottleneck. Proceedings of the Advances in Neural Information Processing Systems, Barcelona, Spain.
23. Voloshynovskiy, S., Taran, O., Kondah, M., Holotyak, T., and Rezende, D. (2020). Variational information bottleneck for semi-supervised classification. Entropy, 22.10.3390/e22090943 ↗
24. The information bottleneck and geometric clustering10.1162/neco_a_01136 ↗
25. Still, S., Bialek, W., and Bottou, L. (2003, January 5). Geometric clustering using the information bottleneck method. Proceedings of the Advances in Neural Information Processing Systems, Xi’an, China.
26. Alemi, A.A., Fischer, I., Dillon, J.V., and Murphy, K. (2016). Deep variational information bottleneck. arXiv.
27. Ye, M., and Sun, Y. (2018, January 10–15). Variable selection via penalized neural network: A drop-out-one loss approach. Proceedings of the International Conference on Machine Learning, Stockholm, Sweden.
28. Rate distortion theory: A mathematical basis for data compression10.1109/TCOM.1972.1091311 ↗
29. Hafez-Kolahi, H., and Kasaei, S. (2019). Information bottleneck and its applications in deep learning. arXiv.
30. Wu, T. (2020). Intelligence, physics and information–the tradeoff between accuracy and simplicity in machine learning. arXiv.
31. Entropy estimation from ranked set samples with application to test of fit10.15446/rce.v40n2.58944 ↗
32. Regularization and variable selection via the elastic net10.1111/j.1467-9868.2005.00503.x ↗
33. Quinlan, J.R. (1996, January 4–8). Bagging, boosting, and C4.5. Proceedings of the National Conference on Artificial Intelligence, Portland, OR, USA.
34. Model-free variable selection in reproducing kernel Hilbert space
35. Kingma, D.P., and Welling, M. (2013). Auto-encoding variational bayes. arXiv.