Label-Free Model Evaluation with Out-of-Distribution Detection
Source abstract
Abstract
In recent years, label-free model evaluation has been developed to estimate the performance of models on unlabeled test sets. However, we find that existing methods perform poorly in environments with out-of-distribution (OOD) data. To address this issue, we propose a novel automatic model evaluation method using OOD detection to reduce the impact of OOD data on model evaluation. Specifically, we use the representation of datasets to train a neural network for accuracy prediction and employ energy-based OOD detection to exclude OOD data during testing. We conducted experiments on several benchmark datasets with varying amounts of OOD data (SVHN, ISUN, ImageNet, and LSUN) and demonstrated that our method reduces the RMSE compared to existing methods by at least 1.27%. Additionally, we tested our method on transformed datasets and datasets with a high proportion of OOD data, and the results show its robustness.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Machine Learning and Data Classification · Domain Adaptation and Few-Shot Learning · Advanced Neural Network Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer Science and Information Engineering, Hefei University of Technology, Hefei 230601, China
Funding
University Synergy Innovation Program of Anhui Province · Natural Science Foundation of Shanxi Province
References (39)
Show source-provided references
1. Platanios, E.A., Dubey, A., and Mitchell, T. (2016, January 20–22). Estimating accuracy from unlabeled data: A bayesian approach. Proceedings of the International Conference on Machine Learning, PMLR, New York, NY, USA.
2. Unsupervised Supervised Learning I: Estimating Classification and Regression Errors without Labels
3. Deng, W., and Zheng, L. (2021, January 20–25). Are labels always necessary for classifier accuracy evaluation?. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.01482 ↗
4. Sun, X., Hou, Y., Li, H., and Zheng, L. (2021). Label-free model evaluation with semi-structured dataset representations. arXiv.
5. Sun, X., Hou, Y., Deng, W., Li, H., and Zheng, L. (2021, January 11–17). Ranking models in unlabeled new environments. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.10.1109/ICCV48922.2021.01155 ↗
6. Deng, W., Gould, S., and Zheng, L. (2021, January 13–15). What does rotation prediction tell us about classifier accuracy under varying testing environments?. Proceedings of the International Conference on Machine Learning, PMLR, Virtual.
7. Yang, J., Zhou, K., Li, Y., and Liu, Z. (2021). Generalized out-of-distribution detection: A survey. arXiv.
8. Bogdoll, D., Nitsche, M., and Zöllner, J.M. (2022, January 18–24). Anomaly detection in autonomous driving: A survey. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPRW56347.2022.00495 ↗
9. Shafaei, A., Schmidt, M., and Little, J.J. (2018). Does your model know the digit 6 is not a cat? A less biased evaluation of “outlier” detectors. arXiv.
10. Garg, S., Balakrishnan, S., Lipton, Z.C., Neyshabur, B., and Sedghi, H. (2022). Leveraging unlabeled data to predict out-of-distribution performance. arXiv.
11. Guillory, D., Shankar, V., Ebrahimi, S., Darrell, T., and Schmidt, L. (2021, January 11–17). Predicting with confidence on unseen distributions. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.10.1109/ICCV48922.2021.00117 ↗
12. Energy-based out-of-distribution detection
13. Zhu, Y., Chen, Y., Xie, C., Li, X., Zhang, R., Xue, H., Tian, X., Zheng, B., and Chen, Y. (2022). Boosting Out-of-distribution Detection with Typical Features. arXiv.
14. Defard, T., Setkov, A., Loesch, A., and Audigier, R. (2021, January 10–15). Padim: A patch distribution modeling framework for anomaly detection and localization. Proceedings of the Pattern Recognition, ICPR International Workshops and Challenges, Virtual Event. Proceedings, Part IV.10.1007/978-3-030-68799-1_35 ↗
15. Eilertsen, G., Jönsson, D., Ropinski, T., Unger, J., and Ynnerman, A. (2020). Classifying the classifier: Dissecting the weight space of neural networks. arXiv.
16. Jiang, Y., Neyshabur, B., Mobahi, H., Krishnan, D., and Bengio, S. (2019). Fantastic generalization measures and where to find them. arXiv.
17. Understanding deep learning (still) requires rethinking generalization10.1145/3446776 ↗
18. Corneanu, C.A., Escalera, S., and Martinez, A.M. (2020, January 13–19). Computing the testing error without a testing set. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00275 ↗
19. Hendrycks, D., and Gimpel, K. (2016). A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv.
20. Liang, S., Li, Y., and Srikant, R. (2017). Enhancing the reliability of out-of-distribution image detection in neural networks. arXiv.
21. Zhang, Y., Deng, W., and Zheng, L. (2023). Unsupervised Evaluation of Out-of-distribution Detection: A Data-centric Perspective. arXiv.
22. Yu, Q., and Aizawa, K. (November, January 27). Unsupervised out-of-distribution detection by maximum classifier discrepancy. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.
23. Chen, X., Lan, X., Sun, F., and Zheng, N. (2020, January 23–28). A boundary based out-of-distribution classifier for generalized zero-shot learning. Proceedings of the Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK. Proceedings, Part 18 XXIV 16.10.1007/978-3-030-58586-0_34 ↗
24. Lee, K., Lee, K., Lee, H., and Shin, J. (2018). A simple unified framework for detecting out-of-distribution samples and adversarial attacks. Adv. Neural Inf. Process. Syst., 31, Available online: https://proceedings.neurips.cc/paper/2018/hash/abdeb6f575ac5c6676b747bca8d09cc2-Abstract.html.
25. Wang, H., Li, Z., Feng, L., and Zhang, W. (2022, January 18–24). Vim: Out-of-distribution with virtual-logit matching. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00487 ↗
26. Agreement-on-the-line: Predicting the performance of neural networks under distribution shift
27. Estimating and explaining model performance when both covariates and labels shift
28. Maggio, S., Bouvier, V., and Dreyfus-Schmidt, L. (2022, January 21–25). Performance Prediction Under Dataset Shift. Proceedings of the 2022 26th International Conference on Pattern Recognition (ICPR), IEEE, Montreal, QC, Canada.10.1109/ICPR56361.2022.9956676 ↗
29. Risser-Maroix, O., and Chamand, B. (2023, January 2–7). What can we Learn by Predicting Accuracy?. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, Waikoloa, HI, USA.10.1109/WACV56688.2023.00242 ↗
30. The Fréchet distance between multivariate normal distributions10.1016/0047-259X(82)90077-X ↗
31. Krizhevsky, A., and Hinton, G. (2023, April 12). Learning Multiple Layers of Features from Tiny Images. Available online: https://www.cs.toronto.edu/~kriz/.
32. Recht, B., Roelofs, R., Schmidt, L., and Shankar, V. (2018). Do CIFAR-10 Classifiers Generalize to CIFAR-10?. arXiv.
33. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., and Fei-Fei, L. (, January 20–25). Imagenet: A large-scale hierarchical image database. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Miami, FL, USA.10.1109/CVPR.2009.5206848 ↗
34. Griffin, G., Holub, A., and Perona, P. (2023, April 12). Caltech-256 Object Category Dataset. Available online: https://resolver.caltech.edu/CaltechAUTHORS:CNS-TR-2007-001.
35. Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., and Ng, A.Y. (2023, April 12). Reading Digits in Natural Images with Unsupervised Feature Learning. Available online: http://ufldl.stanford.edu/housenumbers.
36. Yu, F., Seff, A., Zhang, Y., Song, S., Funkhouser, T., and Xiao, J. (2015). Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv.
37. Xu, P., Ehinger, K.A., Zhang, Y., Finkelstein, A., Kulkarni, S.R., and Xiao, J. (2015). Turkergaze: Crowdsourcing saliency with webcam based eye tracking. arXiv.
38. He, K., Zhang, X., Ren, S., and Sun, J. (July, January 26). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.
39. 80 million tiny images: A large data set for nonparametric object and scene recognition10.1109/TPAMI.2008.128 ↗