EFCMF: A Multimodal Robustness Enhancement Framework for Fine-Grained Recognition
Source abstract
Abstract
Fine-grained recognition has many applications in many fields and aims to identify targets from subcategories. This is a highly challenging task due to the minor differences between subcategories. Both modal missing and adversarial sample attacks are easily encountered in fine-grained recognition tasks based on multimodal data. These situations can easily lead to the model needing to be fixed. An Enhanced Framework for the Complementarity of Multimodal Features (EFCMF) is proposed in this study to solve this problem. The model’s learning of multimodal data complementarity is enhanced by randomly deactivating modal features in the constructed multimodal fine-grained recognition model. The results show that the model gains the ability to handle modal missing without additional training of the model and can achieve 91.14% and 99.31% accuracy on Birds and Flowers datasets. The average accuracy of EFCMF on the two datasets is 52.85%, which is 27.13% higher than that of Bi-modal PMA when facing four adversarial example attacks, namely FGSM, BIM, PGD and C&W. In the face of missing modal cases, the average accuracy of EFCMF is 76.33% on both datasets respectively, which is 32.63% higher than that of Bi-modal PMA. Compared with existing methods, EFCMF is robust in the face of modal missing and adversarial example attacks in multimodal fine-grained recognition tasks. The source code is available at https://github.com/RPZ97/EFCMF (accessed on 8 January 2023).
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Adversarial Robustness in Machine Learning · Domain Adaptation and Few-Shot Learning · Advanced Neural Network Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
College of Electronic Engineering, National University of Defense Technology, Hefei 230037, China
State Key Laboratory of Pulsed Power Laser Technology, Hefei 230037, China
Key Laboratory of Infrared and Low Temperature Plasma of Anhui Province, Hefei 230037, China
Funding
National Science Foundation of China
References (42)
Show source-provided references
1. Wah, C., Branson, S., Welinder, P., Perona, P., and Belongie, S. (2011). The Caltech-Ucsd Birds-200-2011 Dataset, California Institute of Technology.
2. Nilsback, M.E., and Zisserman, A. (2008, January 16–19). Automated flower classification over a large number of classes. Proceedings of the 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, Bhubaneswar, India.10.1109/ICVGIP.2008.47 ↗
3. Khosla, A., Jayadevaprakash, N., Yao, B., and Li, F.F. (2011). Proceedings of the CVPR Workshop on Fine-Grained Visual Categorization (FGVC), Citeseer.
4. Krause, J., Stark, M., Deng, J., and Fei-Fei, L. (2013, January 1–8). 3D Object Representations for Fine-Grained Categorization. Proceedings of the IEEE International Conference on Computer Vision (ICCV) Workshops, Sydney, Australia.10.1109/ICCVW.2013.77 ↗
5. Hou, S., Feng, Y., and Wang, Z. (2017, January 22–29). Vegfru: A domain-specific dataset for fine-grained visual categorization. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.66 ↗
6. Wei, X.S., Cui, Q., Yang, L., Wang, P., and Liu, L. (2019). RPC: A large-scale retail product checkout dataset. arXiv.
7. Peng, J., Xiao, C., and Li, Y. (2020). RP2K: A large-scale retail product dataset for fine-grained image classification. arXiv.
8. Fine-Grained Image Analysis with Deep Learning: A Survey10.1109/TPAMI.2021.3126648 ↗
9. Cortes, C., Lawrence, N., Lee, D., Sugiyama, M., and Garnett, R. (2015). Proceedings of the Advances in Neural Information Processing Systems, Curran Associates, Inc.
10. Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., and Zagoruyko, S. (2020, January 23–28). End-to-end object detection with transformers. Proceedings of the European Conference on Computer Vision, Glasgow, UK.10.1007/978-3-030-58452-8_13 ↗
11. Redmon, J., and Farhadi, A. (2018). Yolov3: An incremental improvement. arXiv.
12. Long, J., Shelhamer, E., and Darrell, T. (2015, January 7–12). Fully convolutional networks for semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298965 ↗
13. Ronneberger, O., Fischer, P., and Brox, T. (2015, January 5–9). U-net: Convolutional networks for biomedical image segmentation. Proceedings of the International Conference on Medical image Computing and Computer-Assisted Intervention, Munich, Germany.10.1007/978-3-319-24574-4_28 ↗
14. Huang, S., Xu, Z., Tao, D., and Zhang, Y. (2016, January 27–30). Part-stacked cnn for fine-grained visual categorization. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.132 ↗
15. Zhang, N., Donahue, J., Girshick, R., and Darrell, T. (2014). European Conference on Computer Vision, Springer.
16. Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K.Q. (2017, January 21–26). Densely connected convolutional networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.243 ↗
17. Simonyan, K., and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv.
18. Lin, T.Y., RoyChowdhury, A., and Maji, S. (2015, January 7–13). Bilinear cnn models for fine-grained visual recognition. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.10.1109/ICCV.2015.170 ↗
19. Multi-objective matrix normalization for fine-grained visual recognition10.1109/TIP.2020.2977457 ↗
20. Niu, L., Veeraraghavan, A., and Sabharwal, A. (2018, January 18–22). Webly supervised learning meets zero-shot learning: A hybrid approach for fine-grained classification. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00749 ↗
21. He, X., and Peng, Y. (2017, January 21–26). Fine-grained image classification via combining vision and language. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.775 ↗
22. Li, J., Zhu, L., Huang, Z., Lu, K., and Zhao, J. (2018). Proceedings of the 26th ACM International Conference on Multimedia, MM’18, Association for Computing Machinery.
23. Zhang, H., Cao, X., and Wang, R. (2018). Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, AAAI’18/IAAI’18/EAAI’18, AAAI Press.
24. Marino, K., Salakhutdinov, R., and Gupta, A. (2016). The more you know: Using knowledge graphs for image classification. arXiv.10.1109/CVPR.2017.10 ↗
25. Chen, T., Lin, L., Chen, R., Wu, Y., and Luo, X. (2018). Knowledge-embedded representation learning for fine-grained image recognition. arXiv.10.24963/ijcai.2018/87 ↗
26. Knowledge Graph Representation Fusion Framework for Fine-Grained Object Recognition in Smart Cities10.1155/2021/8041029 ↗
27. Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., and Fergus, R. (2013). Intriguing properties of neural networks. arXiv.
28. Goodfellow, I.J., Shlens, J., and Szegedy, C. (2014). Explaining and harnessing adversarial examples. arXiv.
29. Kurakin, A., Goodfellow, I.J., and Bengio, S. (2018). Artificial Intelligence Safety and Security, Chapman and Hall/CRC.
30. Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. (2017). Towards deep learning models resistant to adversarial attacks. arXiv.
31. Moosavi-Dezfooli, S.M., Fawzi, A., and Frossard, P. (2016, January 27–30). Deepfool: A simple and accurate method to fool deep neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.282 ↗
32. Carlini, N., and Wagner, D. (2017, January 22–24). Towards evaluating the robustness of neural networks. Proceedings of the 2017 IEEE Symposium on Security and Privacy, San Jose, CA, USA.10.1109/SP.2017.49 ↗
33. Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. (2021, January 10–17). Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada.10.1109/ICCV48922.2021.00986 ↗
34. Zhang, Y., and Wallace, B. (2015). A sensitivity analysis of (and practitioners’ guide to) convolutional neural networks for sentence classification. arXiv.
35. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., and Fei-Fei, L. (2009, January 20–25). Imagenet: A large-scale hierarchical image database. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.10.1109/CVPR.2009.5206848 ↗
36. Reed, S., Akata, Z., Lee, H., and Schiele, B. (2016, January 27–30). Learning deep representations of fine-grained visual descriptions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.13 ↗
37. Wallach, H., Larochelle, H., Beygelzimer, A., d’Alché-Buc, F., Fox, E., and Garnett, R. (2019). Proceedings of the Advances in Neural Information Processing Systems, Curran Associates, Inc.
38. Kim, H. (2020). Torchattacks: A pytorch repository for adversarial attacks. arXiv.
39. Cui, Y., Song, Y., Sun, C., Howard, A., and Belongie, S. (2018, January 18–22). Large scale fine-grained categorization and domain-specific transfer learning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00432 ↗
40. Touvron, H., Cord, M., El-Nouby, A., Verbeek, J., and Jégou, H. (2022). Three things everyone should know about Vision Transformers. arXiv.10.1007/978-3-031-20053-3_29 ↗
41. Part-Guided Relational Transformers for Fine-Grained Visual Recognition10.1109/TIP.2021.3126490 ↗
42. Bi-modal progressive mask attention for fine-grained recognition10.1109/TIP.2020.2996736 ↗