Hierarchical Prototypes Polynomial Softmax Loss Function for Visual Classification
Source abstract
Abstract
A well-designed loss function can effectively improve the characterization ability of network features without increasing the amount of calculation in the model inference stage, and has become the focus of attention in recent research. Given that the existing lightweight network adds a loss to the last layer, which severely attenuates the gradient during the backpropagation process, we propose a hierarchical polynomial kernel prototype loss function in this study. In this function, the addition of a polynomial kernel loss function to multiple stages of the deep neural network effectively enhances the efficiency of gradient return, and only adds multi-layer prototype loss functions in the training stage without increasing the calculation of the inference stage. In addition, the good non-linear expression ability of the polynomial kernel improves the characteristic expression performance of the network. Verification on multiple public datasets shows that the lightweight network trained with the proposed hierarchical polynomial kernel loss function has a higher accuracy than other loss functions.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Domain Adaptation and Few-Shot Learning · Advanced Neural Network Applications · Machine Learning and Data Classification
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Information and Control Engineering, China University of Mining and Technology, Xuzhou 221008, China
IOT Perception Mine Research Center, China University of Mining and Technology, Xuzhou 221008, China
School of Information Engineering, Xuzhou University of Technology, Xuzhou 221000, China
Funding
National Natural Science Foundation of China · State Key Research Development Program of China
References (34)
Show source-provided references
1. Leibe, B., Matas, J., Sebe, N., and Welling, M. (2016). A Discriminative Feature Learning Approach for Deep Face Recognition. Computer Vision—ECCV 2016, Springer.10.1007/978-3-319-46478-7 ↗
2. Squeeze-and-Excitation Networks10.1109/TPAMI.2019.2913372 ↗
3. ImageNet classification with deep convolutional neural networks10.1145/3065386 ↗
4. Masi, I., Wu, Y., Hassner, T., and Natarajan, P. (November, January 29). Deep Face Recognition: A Survey. Proceedings of the 2018 31st SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI), Parana, Brazil.10.1109/SIBGRAPI.2018.00067 ↗
5. Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. (2017). MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. arXiv.
6. Ferrari, V., Hebert, M., Sminchisescu, C., and Weiss, Y. (2018). ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design. Computer Vision—ECCV 2018, Springer.
7. EPKF: Energy Efficient Communication Schemes Based on Kalman Filter for IoT10.1109/JIOT.2019.2900853 ↗
8. Tan, M.X., and Le, Q.V. (2019, January 9–15). EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. Proceedings of the 36th International Conference on Machine Learning (ICML), Long Beach, CA, USA.
9. Wan, W., Zhong, Y., Li, T., and Chen, J. (2018, January 18–23). Rethinking Feature Distribution for Loss Functions in Image Classification. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00950 ↗
10. Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.C. (2018, January 18–23). MobileNetV2: Inverted Residuals and Linear Bottlenecks. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00474 ↗
11. Howard, A., Sandler, M., Chen, B., Wang, W., Chen, L.C., Tan, M., Chu, G., Vasudevan, V., Zhu, Y., and Pang, R. (November, January 27). Searching for MobileNetV3. Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Korea.10.1109/ICCV.2019.00140 ↗
12. Liu, Y., Li, H., and Wang, X. (2017). Learning Deep Features via Congenerous Cosine Loss for Person Recognition. arXiv.
13. Lee, B.S., Phattharaphon, R., Yean, S., Liu, J.G., and Shakya, M. (2020, January 9–11). Euclidean Distance based Loss Function for Eye-Gaze Estimation. Proceedings of the 15th IEEE Sensors Applications Symposium (SAS), Kuala Lumpur, Malaysia.
14. Sun, Y., Chen, Y.H., Wang, X.G., and Tang, X.O. (2014, January 8–13). Deep Learning Face Representation by Joint Identification-Verification. Proceedings of the 28th Conference on Neural Information Processing Systems (NIPS), Montreal, QC, Canada.
15. Margin Loss: Making Faces More Separable10.1109/LSP.2017.2789251 ↗
16. Yang, H.M., Zhang, X.Y., Yin, F., and Liu, C.L. (2018, January 18–23). Robust Classification with Convolutional Prototype Learning. Proceedings of the 31st IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.10.1109/CVPR.2018.00366 ↗
17. What is a support vector machine?10.1038/nbt1206-1565 ↗
18. Lin, T.Y., Dollár, P., Girshick, R., He, K., Hariharan, B., and Belongie, S. (2017, January 21–26). Feature Pyramid Networks for Object Detection. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.106 ↗
19. Ranjan, R., Castillo, C., and Chellappa, R. (2019). L2 constrained Softmax Loss for Discriminative Face Verification. arXiv.
20. Liu, W.Y., Wen, Y.D., Yu, Z.D., and Yang, M. (2016, January 20–22). Large-Margin Softmax Loss for Convolutional Neural Networks. Proceedings of the 33rd International Conference on Machine Learning, New York, NY, USA.
21. Liu, W.Y., Wen, Y.D., Yu, Z.D., Li, M., Raj, B., and Song, L. (2017, January 21–26). SphereFace: Deep Hypersphere Embedding for Face Recognition. Proceedings of the 30th IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.713 ↗
22. Wu, Y., Liu, H., Li, J., and Fu, Y. (2017, January 23–27). Deep Face Recognition with Center Invariant Loss. Proceedings of the Thematic Workshop’17: Proceedings of the on Thematic Workshops of ACM Multimedia 2017, Mountain View, CA, USA.10.1145/3126686.3126693 ↗
23. Chopra, S., Hadsell, R., and LeCun, Y. (2005, January 20–25). Learning a similarity metric discriminatively, with application to face verification. Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), San Diego, CA, USA.
24. Schroff, F., Kalenichenko, D., and Philbin, J. (2015, January 7–12). FaceNet: A unified embedding for face recognition and clustering. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.10.1109/CVPR.2015.7298682 ↗
25. Wu, C.Y., Manmatha, R., Smola, A.J., and Krähenbühl, P. (2017, January 22–29). Sampling Matters in Deep Embedding Learning. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy.10.1109/ICCV.2017.309 ↗
26. Ding, E., Cheng, Y., Xiao, C., Liu, Z., and Yu, W. (2021). Efficient Attention Mechanism for Dynamic Convolution in Lightweight Neural Network. Appl. Sci., 11.10.3390/app11073111 ↗
27. Goldberg, Y., and Elhadad, M. (2008, January 16–17). splitSVM: Fast, space-efficient, non-heuristic, polynomial kernel computation for NLP applications. Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies: Short Papers, Columbus, OH, USA.10.3115/1557690.1557758 ↗
28. Deng, J., Dong, W., Socher, R., Li, L.J., Kai, L., and Li, F.-F. (2009, January 20–25). ImageNet: A large-scale hierarchical image database. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.10.1109/CVPR.2009.5206848 ↗
29. Han, K., Wang, Y., Zhang, Q., Zhang, W., Xu, C., and Zhang, T. (2020). Model Rubik’s Cube: Twisting Resolution, Depth and Width for TinyNets. arXiv.
30. Khosla, A., Jayadevaprakash, N., Yao, B., and Fei-Fei, L. (2012, January 16–21). Novel Dataset for Fine-Grained Image Categorization: Stanford Dogs. Proceedings of the CVPR Workshop on Fine-Grained Visual Categorization (FGVC), Providence, RI, USA.
31. Krause, J., Stark, M., Deng, J., and Fei-Fei, L. (2013, January 1–8). 3D Object Representations for Fine-Grained Categorization. Proceedings of the 2013 IEEE International Conference on Computer Vision (ICCV), Sydney, Australia.10.1109/ICCVW.2013.77 ↗
32. Welinder, P., Branson, S., Mita, T., Wah, C., Schroff, F., Belongie, S., and Perona, P. (2011). The Caltech-UCSD Birds-200-2011 Dataset, California Institute of Technology. Technical Report CNS-TR-2010-001.
33. Ketkar, N. (2017). Introduction to PyTorch. Deep Learning with Python: A Hands-on Introduction, Apress.10.1007/978-1-4842-2766-4 ↗
34. Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., and Torralba, A. (2016, January 27–30). Learning Deep Features for Discriminative Localization. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.319 ↗