Deep Active Learning for Computer Vision Tasks: Methodologies, Applications, and Challenges
Source abstract
Abstract
Active learning is a label-efficient machine learning method that actively selects the most valuable unlabeled samples to annotate. Active learning focuses on achieving the best possible performance while using as few, high-quality sample annotations as possible. Recently, active learning achieved promotion combined with deep learning-based methods, which are named deep active learning methods in this paper. Deep active learning plays a crucial role in computer vision tasks, especially in label-insensitive scenarios, such as hard-to-label tasks (medical images analysis) and time-consuming tasks (autonomous driving). However, deep active learning still has some challenges, such as unstable performance and dirty data, which are future research trends. Compared with other reviews on deep active learning, our work introduced the deep active learning from computer vision-related methodologies and corresponding applications. The expected audience of this vision-friendly survey are researchers who are working in computer vision but willing to utilize deep active learning methods to solve vision problems. Specifically, this review systematically focuses on the details of methods, applications, and challenges in vision tasks, and we also introduce the classic theories, strategies, and scenarios of active learning in brief.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Machine Learning and Algorithms · Machine Learning and Data Classification · Machine Learning in Materials Science
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
College of Computer, National University of Defense Technology, Changsha 410073, China
Funding
National Key Research and Development Program of China · Natural Science Foundation of Hunan Province of China
References (100+)
Show source-provided references
1. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., and Li, F.-F. (2009, January 20–25). Imagenet: A large-scale hierarchical image database. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.10.1109/CVPR.2009.5206848 ↗
2. Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., and Schiele, B. (2016, January 27–30). The cityscapes dataset for semantic urban scene understanding. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.350 ↗
3. Ma, J., Zhang, Y., Gu, S., Zhu, C., Ge, C., Zhang, Y., An, X., Wang, C., Wang, Q., and Liu, X. (2021). AbdomenCT-1K: Is Abdominal Organ Segmentation A Solved Problem. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI).10.1109/TPAMI.2021.3100536 ↗
4. Settles, B. (2004). Active Learning Literature Survey, University of Wisconsin–Madison. Computer Sciences Technical Report 1648.
5. Human-in-the-loop active learning via brain computer interface10.1007/s10472-020-09689-0 ↗
6. A survey on active learning and human-in-the-loop deep learning for medical image analysis10.1016/j.media.2021.102062 ↗
7. Active learning query strategies for classification, regression, and clustering: A survey10.1007/s11390-020-9487-4 ↗
8. A survey of deep active learning
9. Zhan, X., Wang, Q., Huang, K.H., Xiong, H., Dou, D., and Chan, A.B. (2022). A comparative survey of deep active learning. arXiv.
10. Confidence-based active learning10.1109/TPAMI.2006.156 ↗
11. Agrawal, A., Tripathi, S., and Vardhan, M. (2021). Multicore based least confidence query sampling strategy to speed up active learning approach for named entity recognition. Computing, 1–19.10.1007/s00607-021-01000-1 ↗
12. Active learning approach using a modified least confidence sampling strategy for named entity recognition10.1007/s13748-021-00230-w ↗
13. Joshi, A.J., Porikli, F., and Papanikolopoulos, N. (2009, January 20–25). Multi-class active learning for image classification. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.10.1109/CVPRW.2009.5206627 ↗
14. Zhou, J., and Sun, S. (2014, January 17–19). Improved margin sampling for active learning. Proceedings of the Chinese Conference on Pattern Recognition, Changsha, China.10.1007/978-3-662-45646-0_13 ↗
15. Active learning combining uncertainty and diversity for multi-class image classification10.1049/iet-cvi.2014.0140 ↗
16. Multi-class active learning by uncertainty sampling with diversity maximization10.1007/s11263-014-0781-x ↗
17. Active learning and semi-supervised learning for speech recognition: A unified framework using the global entropy reduction maximization criterion10.1016/j.csl.2009.03.004 ↗
18. Ozdemir, F., Peng, Z., Tanner, C., Fuernstahl, P., and Goksel, O. (2018). Active learning for segmentation by optimizing content information for maximal entropy. Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, Springer.10.1007/978-3-030-00889-5_21 ↗
19. Brinker, K. (2003, January 21). Incorporating diversity in active learning with support vector machines. Proceedings of the 20th International Conference on Machine Learning, Washington, DC, USA.
20. Transductive reliability estimation for medical diagnosis10.1016/S0933-3657(03)00043-5 ↗
21. Active batch selection via convex relaxations with guaranteed solution bounds10.1109/TPAMI.2015.2389848 ↗
22. Active, continual fine tuning of convolutional neural networks for reducing annotation efforts10.1016/j.media.2021.101997 ↗
23. Seung, H.S., Opper, M., and Sompolinsky, H. (1992, January 1). Query by Committee. Proceedings of the Fifth Annual Workshop on Computational Learning Theory, Pittsburgh, PA, USA.10.1145/130385.130417 ↗
24. Getoor, L., and Scheffer, T. (2011, January 28). Active Learning from Crowds. Proceedings of the 28th International Conference on Machine Learning, Bellevue, WA, USA.
25. Dagan, I., and Engelson, S.P. (1995). Committee-based sampling for training probabilistic classifiers. Machine Learning Proceedings 1995, Proceedings of the Twelfth International Conference on Machine Learning, Tahoe City, CA, USA, 9–12 July 1995, Elsevier.10.1016/B978-1-55860-377-6.50027-X ↗
26. Zhou, Z., Shin, J., Zhang, L., Gurudu, S., Gotway, M., and Liang, J. (2017, January 21–26). Fine-tuning convolutional neural networks for biomedical image analysis: Actively and incrementally. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.506 ↗
27. Queries and Concept Learning10.1007/BF00116828 ↗
28. Schumann, R., and Rehbein, I. (2019, January 3–4). Active learning via membership query synthesis for semi-supervised sentence classification. Proceedings of the 23rd Conference on Computational Natural Language Learning, Hong Kong, China.10.18653/v1/K19-1044 ↗
29. Alabdulmohsin, I., Gao, X., and Zhang, X. (2015, January 25–30). Efficient active learning of halfspaces via query synthesis. Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, Austin, TX, USA.10.1609/aaai.v29i1.9563 ↗
30. Touretzky, D. (1989). Training Connectionist Networks with Queries and Selective Sampling. Advances in Neural Information Processing Systems, Morgan-Kaufmann.
31. Balasubramanian, V., Chakraborty, S., and Panchanathan, S. (October, January 27). Generalized query by transduction for online active learning. Proceedings of the IEEE 12th International Conference on Computer Vision (ICCV) Workshops, Kyoto, Japan.
32. Query by transduction10.1109/TPAMI.2007.70811 ↗
33. Monteleoni, C., and Kaariainen, M. (2007, January 17–22). Practical online active learning for classification. Proceedings of the 2007 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Minneapolis, MN, USA.10.1109/CVPR.2007.383437 ↗
34. Lewis, D.D., and Gale, W.A. (1994, January 3–6). A Sequential Algorithm for Training Text Classifiers. Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Dublin, Ireland.10.1007/978-1-4471-2099-5_1 ↗
35. Pool-based sequential active learning for regression10.1109/TNNLS.2018.2868649 ↗
36. Zhan, X., Liu, H., Li, Q., and Chan, A.B. (2021, January 19–27). A Comparative Survey: Benchmarking for Pool-based Active Learning. Proceedings of the 30th International Joint Conference on Artificial Intelligence (IJCAI 2021), Virtual.10.24963/ijcai.2021/634 ↗
37. Pool-based active learning in approximate linear regression10.1007/s10994-009-5100-3 ↗
38. Gal, Y., and Ghahramani, Z. (2016, January 19–24). Dropout as a bayesian approximation: Representing model uncertainty in deep learning. Proceedings of the International Conference on Machine Learning, New York, NY, USA.
39. Gal, Y., and Ghahramani, Z. (2015). Bayesian convolutional neural networks with Bernoulli approximate variational inference. arXiv.
40. Gal, Y., Islam, R., and Ghahramani, Z. (2017, January 6–11). Deep bayesian active learning with image data. Proceedings of the International Conference on Machine Learning (ICML), Sydney, NSW, Australia.
41. Gradient-based learning applied to document recognition10.1109/5.726791 ↗
42. Codella, N.C., Gutman, D., Celebi, M.E., Helba, B., Marchetti, M.A., Dusza, S.W., Kalloo, A., Liopyris, K., Mishra, N., and Kittler, H. (2018, January 4–7). Skin lesion analysis toward melanoma detection: A challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), hosted by the International Skin Imaging Collaboration (ISIC). Proceedings of the 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), Washington, DC, USA.10.1109/ISBI.2018.8363547 ↗
43. Houlsby, N., Huszár, F., Ghahramani, Z., and Lengyel, M. (2011). Bayesian active learning for classification and preference learning. arXiv.
44. A mathematical theory of communication10.1145/584091.584093 ↗
45. Neural network ensembles10.1109/34.58871 ↗
46. Beluch, W.H., Genewein, T., Nürnberger, A., and Köhler, J.M. (2018, January 18–23). The power of ensembles for active learning in image classification. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00976 ↗
47. Krizhevsky, A. (2022, July 10). Learning Multiple Layers of Features from Tiny Images. Available online: https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf.
48. Sener, O., and Savarese, S. (May, January 30). Active Learning for Convolutional Neural Networks: A Core-Set Approach. Proceedings of the International Conference on Learning Representations (ICLR), Vancouver, BC, Canada.
49. Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., and Ng, A.Y. (2011, January 16). Reading digits in natural images with unsupervised feature learning. Proceedings of the NIPS Workshop on Deep Learning and Unsupervised Feature Learning, Granada, Spain.
50. Janz, D., van der Westhuizen, J., and Hernández-Lobato, J.M. (2017). Actively learning what makes a discrete sequence valid. arXiv.
51. Kirsch, A., Van Amersfoort, J., and Gal, Y. (2019, January 8–14). Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning. Proceedings of the NIPS’19: Proceedings of the 33rd International Conference on Neural Information Processing Systems, Vancouver, BC, Canada.
52. Yoo, D., and Kweon, I.S. (2019, January 15–20). Learning loss for active learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00018 ↗
53. The pascal visual object classes (voc) challenge10.1007/s11263-009-0275-4 ↗
54. Andriluka, M., Pishchulin, L., Gehler, P., and Schiele, B. (2014, January 23–28). 2d human pose estimation: New benchmark and state of the art analysis. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Columbus, OH, USA.10.1109/CVPR.2014.471 ↗
55. François, D. (2008). High-dimensional data analysis. From Optimal Metric to Feature Selection. [Ph.D. Thesis, Université Catholique de Louvain].
56. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014, January 8–13). Generative Adversarial Nets. Proceedings of the Advances in Neural Information Processing Systems 27 (NIPS 2014), Montreal, QC, USA.
57. Zhu, J., and Bento, J. (2017). Generative Adversarial Active Learning. arXiv.
58. Backpropagation Applied to Handwritten Zip Code Recognition10.1162/neco.1989.1.4.541 ↗
59. Tran, T., Do, T.T., Reid, I., and Carneiro, G. (2019, January 9–15). Bayesian generative active deep learning. Proceedings of the International Conference on Machine Learning (ICML), Long Beach, CA, USA.
60. Mayer, C., and Timofte, R. (2020, January 1–5). Adversarial Sampling for Active Learning. Proceedings of the 2020 IEEE Winter Conference on Applications of Computer Vision (WACV), Snowmass, CO, USA.10.1109/WACV45572.2020.9093556 ↗
61. Liu, Z., Luo, P., Wang, X., and Tang, X. (2015, January 7–13). Deep Learning Face Attributes in the Wild. Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile.10.1109/ICCV.2015.425 ↗
62. Yu, F., Zhang, Y., Song, S., Seff, A., and Xiao, J. (2015). LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop. arXiv.
63. Sinha, S., Ebrahimi, S., and Darrell, T. (2019, January 27). Variational Adversarial Active Learning. Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Korea.10.1109/ICCV.2019.00607 ↗
64. Griffin, G., Holub, A., and Perona, P. (2022, July 10). Caltech-256 Object Category Dataset. Available online: https://data.caltech.edu/records/20087.
65. Yu, F., Chen, H., Wang, X., Xian, W., Chen, Y., Liu, F., Madhavan, V., and Darrell, T. (2020, January 13–19). Bdd100k: A diverse driving dataset for heterogeneous multitask learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR42600.2020.00271 ↗
66. Huijser, M., and Gemert, J.C.v. (2017, January 22–29). Active Decision Boundary Annotation with Deep Generative Models. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy.10.1109/ICCV.2017.565 ↗
67. Odena, A., Olah, C., and Shlens, J. (2017, January 6–11). Conditional image synthesis with auxiliary classifier gans. Proceedings of the International Conference on Machine Learning (ICML), Sydney, NSW, Australia.
68. Larsen, A.B.L., Sønderby, S.K., Larochelle, H., and Winther, O. (2016, January 19–24). Autoencoding beyond pixels using a learned similarity metric. Proceedings of the International Conference on Machine Learning (ICML), New York, NY, USA.
69. Li, C., Chen, W., Luo, X., He, Y., and Tan, Y. (2022, January 22–27). Adaptive Pseudo Labeling for Source-Free Domain Adaptation in Medical Image Segmentation. Proceedings of the ICASSP 2022—2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Singapore.10.1109/ICASSP43922.2022.9746286 ↗
70. McCallum, A., and Nigam, K. (1998, January 24–27). Employing EM and Pool-Based Active Learning for Text Classification. Proceedings of the Fifteenth International Conference on Machine Learning (ICML), Madison, WI, USA.
71. Muslea, I., Minton, S., and Knoblock, C.A. (2002, January 8–12). Active + semi-supervised learning = robust multi-view learning. Proceedings of the Fifteenth International Conference on Machine Learning (ICML), Sydney, NSW, Australia.
72. Zhou, Z.H., Chen, K.J., and Jiang, Y. (2004, January 20–24). Exploiting unlabeled data in content-based image retrieval. Proceedings of the European Conference on Machine Learning (ECML), Pisa, Italy.10.1007/978-3-540-30115-8_48 ↗
73. Blum, A., and Mitchell, T. (1998, January 24–26). Combining Labeled and Unlabeled Data with Co-Training. Proceedings of the Eleventh Annual Conference on Computational Learning Theory (COLT), Madison, WI, USA.10.1145/279943.279962 ↗
74. Tri-training: Exploiting unlabeled data using three classifiers10.1109/TKDE.2005.186 ↗
75. Han, W., Coutinho, E., Ruan, H., Li, H., Schuller, B., Yu, X., and Zhu, X. (2016). Semi-supervised active learning for sound classification in hybrid learning environments. PLoS ONE, 11.10.1371/journal.pone.0162075 ↗
76. Tomanek, K., and Hahn, U. (2009, January 2–7). Semi-supervised active learning for sequence labeling. Proceedings of the 47th Annual Meeting of the Association of Computational Linguistics (ACL), Singapore.10.3115/1690219.1690291 ↗
77. Combining active and semi-supervised learning for spoken language understanding10.1016/j.specom.2004.08.002 ↗
78. Song, S., Berthelot, D., and Rostamizadeh, A. (2019). Combining mixmatch and active learning for better accuracy with fewer labels. arXiv.
79. Guo, J., Shi, H., Kang, Y., Kuang, K., Tang, S., Jiang, Z., Sun, C., Wu, F., and Zhuang, Y. (2021, January 11–17). Semi-supervised active learning for semi-supervised models: Exploit adversarial examples with graph-based virtual labels. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada.10.1109/ICCV48922.2021.00289 ↗
80. Van den Oord, A., Li, Y., and Vinyals, O. (2018). Representation learning with contrastive predictive coding. arXiv.
81. Poole, B., Ozair, S., Van Den Oord, A., Alemi, A., and Tucker, G. (2019, January 9–15). On variational bounds of mutual information. Proceedings of the International Conference on Machine Learning (ICML), Long Beach, CA, USA.
82. McAllester, D., and Stratos, K. (2020, January 3–5). Formal limitations on the measurement of mutual information. Proceedings of the International Conference on Artificial Intelligence and Statistics (PMLR), Palermo, Italy.
83. He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. (2020, January 14–19). Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR42600.2020.00975 ↗
84. Chen, X., Fan, H., Girshick, R., and He, K. (2020). Improved baselines with momentum contrastive learning. arXiv.
85. Chen, X., Xie, S., and He, K. (2021, January 11–17). An Empirical Study of Training Self-Supervised Vision Transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada.10.1109/ICCV48922.2021.00950 ↗
86. Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. (2020, January 12–18). A simple framework for contrastive learning of visual representations. Proceedings of the International Conference on Machine Learning (ICML), PMLR, Vienna, Austria.
87. Big self-supervised models are strong semi-supervised learners
88. Saunshi, N., Plevrakis, O., Arora, S., Khodak, M., and Khandeparkar, H. (2019, January 10–15). A theoretical analysis of contrastive unsupervised representation learning. Proceedings of the International Conference on Machine Learning (ICML), PMLR, Long Beach, CA, USA.
89. Ma, S., Zeng, Z., McDuff, D., and Song, Y. (2021, January 3–7). Active Contrastive Learning of Audio-Visual Video Representations. Proceedings of the International Conference on Learning Representations (ICLR), New Orleans, LA, USA.
90. Du, P., Zhao, S., Chen, H., Chai, S., Chen, H., and Li, C. (2021, January 11–17). Contrastive coding for active learning under class distribution mismatch. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada.10.1109/ICCV48922.2021.00880 ↗
91. Zhu, Y., Xu, W., Liu, Q., and Wu, S. (2020). When contrastive learning meets active learning: A novel graph active learning paradigm with self-supervision. arXiv.
92. Krishnan, R., Ahuja, N., Sinha, A., Subedar, M., Tickoo, O., and Iyer, R. (2021). Improving robustness and efficiency in active learning with contrastive loss. arXiv.
93. Gao, B., Zhao, X., and Zhao, H. (2022). An Active and Contrastive Learning Framework for Fine-Grained Off-Road Semantic Segmentation. arXiv.10.1109/TITS.2022.3218403 ↗
94. Li, C., Luo, X., Chen, W., He, Y., Wu, M., and Tan, Y. (2021, January 9–12). AttENT: Domain-Adaptive Medical Image Segmentation via Attention-Aware Translation and Adversarial Entropy Minimization. Proceedings of the 2021 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), Houston, TX, USA.10.1109/BIBM52615.2021.9669620 ↗
95. Li, C., Chen, W., Wu, M., Luo, X., He, Y., and Tan, Y. (2021, January 9–12). Tri-Directional Tasks Complementary Learning for Unsupervised Domain Adaptation of Cross-modality Medical Image Semantic Segmentation. Proceedings of the 2021 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), Houston, TX, USA.10.1109/BIBM52615.2021.9669615 ↗
96. Chattopadhyay, R., Fan, W., Davidson, I., Panchanathan, S., and Ye, J. (2013, January 16–21). Joint transfer and batch-mode active learning. Proceedings of the International Conference on Machine Learning (ICML), PMLR, Atlanta, GA, USA.
97. Huang, S.J., Zhao, J.W., and Liu, Z.Y. (2018, January 19–23). Cost-effective training of deep cnns with active model adaptation. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD), London, UK.10.1145/3219819.3220026 ↗
98. Ning, M., Lu, D., Wei, D., Bian, C., Yuan, C., Yu, S., Ma, K., and Zheng, Y. (2021, January 11–17). Multi-anchor active domain adaptation for semantic segmentation. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada.10.1109/ICCV48922.2021.00898 ↗
99. He, Y., Zhang, L., Chen, W., Luo, X., Jia, X., and Li, C. (2021, January 10–15). CenterRepp: Predict Central Representative Point Set’s Distribution For Detection. Proceedings of the 2020 25th International Conference on Pattern Recognition (ICPR), Milan, Italy.10.1109/ICPR48806.2021.9412163 ↗
100. Jia, X., Chen, W., Li, C., Liang, Z., Wu, M., Tan, Y., and Huang, L. (2021, January 3–7). Multi-scale cost volumes cascade network for stereo matching. Proceedings of the 2021 IEEE International Conference on Robotics and Automation (ICRA), New Orleans, LA, USA.10.1109/ICRA48506.2021.9560864 ↗