Unified Open-Set Recognition and Novel Class Discovery via Prototype-Guided Representation
Source abstract
Abstract
The existing research on open-set recognition (OSR) and novel class discovery (NCD) has largely treated these tasks as independent fields. OSR aims to identify samples that do not belong to the training set classes, while NCD seeks to further classify such unseen, unlabeled samples into novel classes. However, there is a lack of a unified framework to automate both tasks systematically. In this paper, we propose a unified training framework to identify and categorize unseen samples. Specifically, we conduct a comprehensive evaluation of existing post hoc OSR methods and observe that their performance is highly sensitive to the temperature scaling factor. To address this, we introduce a distance-based evaluation method for OSR, which not only outperforms existing post hoc approaches but also integrates seamlessly with them to deliver enhanced performance. Furthermore, we developed a prototype-based classification head leveraging this distance metric, which facilitates compact feature representations for known classes and guides the clustering of unknown classes, thereby significantly enhancing the classification accuracy for novel classes. On the CUB-200-2011 dataset, our unified framework achieves a 0.95–6.12% improvement in AUROC scores on OSR benchmarks and a 3.19% increase in classification accuracy for novel classes. Extensive experiments and visualizations validate the effectiveness of the proposed approach. We believe that this unified framework will pave the way for automating the integration of OSR and NCD, offering a more efficient and systematic approach to addressing these tasks.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Domain Adaptation and Few-Shot Learning · Face and Expression Recognition · Anomaly Detection Techniques and Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer and Information Engineering, Institute for Artificial Intelligence, Shanghai Polytechnic University, 2360 Jinhai Road, Pudong, Shanghai 201219, China
Department of Electronic Engineering, Jeonbuk National University, 567 Baekje-daero, Deokjin-gu, Jeonju 54896, Jeollabuk-do, Republic of Korea
Funding
Natural Science Foundation of Shanghai
References (52)
Show source-provided references
1. Machine learning and deep learning: A review of methods and applications
2. Multilevel Face Recognition System10.1007/s10559-024-00655-w ↗
3. Object detection in 20 years: A survey10.1109/JPROC.2023.3238524 ↗
4. Koley, S., Bhunia, A.K., Sain, A., Chowdhury, P.N., Xiang, T., and Song, Y.Z. (2024, January 16–22). You’ll Never Walk Alone: A Sketch and Text Duet for Fine-Grained Image Retrieval. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR52733.2024.01562 ↗
5. Challenges, evaluation and opportunities for open-world learning10.1038/s42256-024-00852-4 ↗
6. Yang, J., Wang, P., Zou, D., Zhou, Z., Ding, K., Peng, W., Wang, H., Chen, G., Li, B., and Sun, Y. (December, January 28). OpenOOD: Benchmarking Generalized Out-of-Distribution Detection. Proceedings of the Thirty-Sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, Virtual.
7. Fini, E., Sangineto, E., Lathuilière, S., Zhong, Z., Nabi, M., and Ricci, E. (2021, January 11–17). A unified objective for novel class discovery. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.10.1109/ICCV48922.2021.00915 ↗
8. Enhancing Few-Shot Out-of-Distribution Detection with Pre-Trained Model Features10.1109/TIP.2024.3468874 ↗
9. Hendrycks, D., and Gimpel, K. (2016, January 2–4). A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks. Proceedings of the International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico.
10. Liang, S., Li, Y., and Srikant, R. (May, January 30). Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks. Proceedings of the International Conference on Learning Representations, Vancouver, BC, Canada.
11. Liu, W., Wang, X., Owens, J., and Li, Y. (2020, January 6–12). Energy-based out-of-distribution detection. Proceedings of the 34th Conference on Neural Information Processing Systems (NeurIPS 2020), Vancouver, BC, Canada.
12. Hendrycks, D., Basart, S., Mazeika, M., Zou, A., Kwon, J., Mostajabi, M., Steinhardt, J., and Song, D. (2022, January 17–23). Scaling Out-of-Distribution Detection for Real-World Settings. Proceedings of the International Conference on Machine Learning, PMLR, Baltimore, MD, USA.
13. Miller, D., Sunderhauf, N., Milford, M., and Dayoub, F. (2021, January 5–9). Class anchor clustering: A loss for distance-based open set recognition. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, Virtual.10.1109/WACV48630.2021.00361 ↗
14. Orientational distribution learning with hierarchical spatial attention for open set recognition
15. Hsu, Y.C., Lv, Z., and Kira, Z. (May, January 30). Learning to cluster in order to transfer across domains and tasks. Proceedings of the International Conference on Learning Representations, Vancouver, BC, Canada.
16. Han, K., Vedaldi, A., and Zisserman, A. (November, January 27). Learning to discover novel visual categories via deep transfer clustering. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.
17. Autonovel: Automatically discovering and learning novel visual categories10.1109/TPAMI.2021.3091944 ↗
18. Zhong, Z., Fini, E., Roy, S., Luo, Z., Ricci, E., and Sebe, N. (2021, January 20–25). Neighborhood contrastive learning for novel class discovery. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.01072 ↗
19. Han, K., Rebuffi, S., Ehrhardt, S., Vedaldi, A., and Zisserman, A. (May, January 26). Automatically discovering and learning new visual categories with ranking statistics. Proceedings of the 8th Intennational Conference on Learning Representations, ICLR 2020, Schloss Dagstuhl-Leibniz-Zentrum für Informatik, Virtual.
20. Jaiswal, A., Babu, A.R., Zadeh, M.Z., Banerjee, D., and Makedon, F. (2020). A survey on contrastive self-supervised learning. Technologies, 9.10.3390/technologies9010002 ↗
21. Contrastive self-supervised learning in recommender systems: A survey
22. Asano, Y., Rupprecht, C., and Vedaldi, A. (2020, January 30). Self-labelling via simultaneous clustering and representation learning. Proceedings of the International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia.
23. Gu, P., Zhang, C., Xu, R., and He, X. (2023, January 1–6). Class-relation Knowledge Distillation for Novel Class Discovery. Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision (ICCV), Paris, France.10.1109/ICCV51070.2023.01510 ↗
24. An, W., Tian, F., Shi, W., Chen, Y., Wu, Y., Wang, Q., and Chen, P. (2024, January 26–27). Transfer and alignment network for generalized category discovery. Proceedings of the AAAI Conference on Artificial Intelligence, Vancouver, BC, Canada.
25. Bharadwaj, R., Naseer, M., Khan, S., and Khan, F.S. (2023). Enhancing Novel Object Detection via Cooperative Foundational Models. arXiv.
26. Hayes, T.L., de Souza, C.R., Kim, N., Kim, J., Volpi, R., and Larlus, D. (2024). PANDAS: Prototype-based Novel Class Discovery and Detection. arXiv.
27. Liu, J., Wang, Y., Zhang, T., Fan, Y., Yang, Q., and Shao, J. (2023, January 19–25). Open-world semi-supervised novel class discovery. Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, Macao, China.10.24963/ijcai.2023/445 ↗
28. Xiao, R., Feng, L., Tang, K., Zhao, J., Li, Y., Chen, G., and Wang, H. (2024, January 16–22). Targeted representation alignment for open-world semi-supervised learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR52733.2024.02177 ↗
29. Zhang, C., Xu, R., and He, X. (2023). Novel Class Discovery for Long-tailed Recognition. arXiv.
30. Novel category discovery without forgetting for automatic target recognition10.1109/JSTARS.2024.3358449 ↗
31. Liu, Y., Cai, Y., Jia, Q., Qiu, B., Wang, W., and Pu, N. (2024, January 16–2). Novel class discovery for ultra-fine-grained visual categorization. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR52733.2024.01674 ↗
32. Determine the number of unknown targets in open world based on elbow method10.1109/TFUZZ.2020.2966182 ↗
33. The application of cluster analysis in strategic management research: An analysis and critique10.1002/(SICI)1097-0266(199606)17:6<441::AID-SMJ819>3.0.CO;2-G ↗
34. An examination of procedures for determining the number of clusters in a data set10.1007/BF02294245 ↗
35. Who belongs in the family?10.1007/BF02289263 ↗
36. Silhouettes: A graphical aid to the interpretation and validation of cluster analysis10.1016/0377-0427(87)90125-7 ↗
37. A dendrite method for cluster analysis10.1080/03610927408827101 ↗
38. A cluster separation measure10.1109/TPAMI.1979.4766909 ↗
39. Oquab, M., Darcet, T., Moutakanni, T., Vo, H.V., Szafraniec, M., Khalidov, V., Fernandez, P., HAZIZA, D., Massa, F., and El-Nouby, A. (2023). DINOv2: Learning Robust Visual Features without Supervision. arXiv.
40. Krizhevsky, A., Sutskever, I., and Hinton, G.E. (2012, January 3–6). Imagenet classification with deep convolutional neural networks. Proceedings of the Advances in Neural Information Processing Systems 25 (NIPS 2012), Lake Tahoe, NV, USA.
41. Du, X., Wang, Z., Cai, M., and Li, Y. (2022). Vos: Learning what you don’t know by virtual outlier synthesis. arXiv.
42. Powers, D.M. (2020). Evaluation: From precision, recall and F-measure to ROC, informedness, markedness and correlation. arXiv.
43. The Hungarian method for the assignment problem10.1002/nav.3800020109 ↗
44. Adaptive open domain recognition by coarse-to-fine prototype-based network10.1016/j.patcog.2022.108657 ↗
45. Toward open set recognition10.1109/TPAMI.2012.256 ↗
46. Zhao, B., Wen, X., and Han, K. (2023, January 1–6). Learning semi-supervised gaussian mixture models for generalized category discovery. Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France.10.1109/ICCV51070.2023.01524 ↗
47. Krause, J., Jin, H., Yang, J., and Fei-Fei, L. (2015, January 7–12). Fine-grained recognition without part annotations. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7299194 ↗
48. Maji, S., Rahtu, E., Kannala, J., Blaschko, M., and Vedaldi, A. (2013). Fine-grained visual classification of aircraft. arXiv.
49. Luo, Z., Liu, Y., Schiele, B., and Sun, Q. (2023, January 17–24). Class-incremental exemplar compression for class-incremental learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.01094 ↗
50. Tan, K.C., Liu, Y., Ambrose, B., Tulig, M., and Belongie, S. (2019). The herbarium challenge 2019 dataset. arXiv.
51. Vaze, S., Han, K., Vedaldi, A., and Zisserman, A. (2022, January 18–24). Generalized category discovery. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00734 ↗
52. Hughes, D., and Salathé, M. (2015). An open access repository of images on plant health to enable the development of mobile disease diagnostics. arXiv.