Multi-Resolution and Semantic-Aware Bidirectional Adapter for Multi-Scale Object Detection
Source abstract
Abstract
Scale variation presents a significant challenge in object detection. To address this, multi-level feature fusion techniques have been proposed, exemplified by methods such as the feature pyramid network (FPN) and its extensions. Nonetheless, the input features provided to these methods and the interaction among features across different levels are limited and inflexible. In order to fully leverage the features of multi-scale objects and amplify feature interaction and representation, we introduce a novel and efficient framework known as a multi-resolution and semantic-aware bidirectional adapter (MSBA). Specifically, MSBA comprises three successive components: multi-resolution cascaded fusion (MCF), a semantic-aware refinement transformer (SRT), and bidirectional fine-grained interaction (BFI). MCF adaptively extracts multi-level features to enable cascaded fusion. Subsequently, SRT enriches the long-range semantic information within high-level features. Following this, BFI facilitates ample fine-grained interaction via bidirectional guidance. Benefiting from the coarse-to-fine process, we can acquire robust multi-scale representations for a variety of objects. Each component can be individually integrated into different backbone architectures. Experimental results substantiate the superiority of our approach and validate the efficacy of each proposed module.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Neural Network Applications · Advanced Image and Video Retrieval Techniques · Robotics and Sensor-Based Localization
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
National Computer Network Emergency Response Technical Team/Coordination Center of China (CNCERT/CC), Beijing 100029, China
Aerospace Information Research Institute, Chinese Academy of Sciences, No. 9 Dengzhuang South Road, Haidian District, Beijing 100094, China
Department of Key Laboratory of Computational Optical Imaging Technology, Chinese Academy of Sciences, No. 9 Dengzhuang South Road, Haidian District, Beijing 100094, China
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100094, China
Funding
National Natural Science Foundation of China · National Key Research and Development Program of China · Beijing Natural Science Foundation
References (58)
Show source-provided references
1. Yang, Z., Liu, S., Hu, H., Wang, L., and Lin, S. (2019). RepPoints: Point Set Representation for Object Detection. arXiv.10.1109/ICCV.2019.00975 ↗
2. Wang, X., Zhang, S., Yu, Z., Feng, L., and Zhang, W. (2020, January 13–19). Scale-Equalizing Pyramid Convolution for Object Detection. Proceedings of the CVPR 2020: Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.01337 ↗
3. Guo, C., Fan, B., Zhang, Q., Xiang, S., and Pan, C. (2020, January 13–19). AugFPN: Improving Multi-Scale Feature Learning for Object Detection. Proceedings of the CVPR 2020: Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.01261 ↗
4. Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., and Zagoruyko, S. (2020, January 23–28). End-to-end object detection with transformers. Proceedings of the European Conference on Computer Vision, Glasgow, UK.10.1007/978-3-030-58452-8_13 ↗
5. Lin, T.Y., Maire, M., Belongie, S.J., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C.L. (2014, January 8–11). Microsoft COCO: Common Objects in Context. Proceedings of the European Conference on Computer Vision, Zurich, Switzerland.10.1007/978-3-319-10602-1_48 ↗
6. Singh, B., and Davis, L.S. (2018, January 18–23). An analysis of scale invariance in object detection snip. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00377 ↗
7. Dai, J., Li, Y., He, K., and Sun, J. (2016, January 5–10). R-fcn: Object detection via region-based fully convolutional networks. Proceedings of the Advances in Neural Information Processing Systems, Barcelona, Spain.
8. Lin, T.Y., Dollár, P., Girshick, R., He, K., Hariharan, B., and Belongie, S. (2017, January 21–26). Feature pyramid networks for object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.106 ↗
9. Sdtp: Semantic-aware decoupled transformer pyramid for dense image prediction10.1109/TCSVT.2022.3162069 ↗
10. PBNet: Part-based convolutional neural network for complex composite object detection in remote sensing imagery10.1016/j.isprsjprs.2020.12.015 ↗
11. Multi-scale object detection in remote sensing imagery with convolutional neural networks10.1016/j.isprsjprs.2018.04.003 ↗
12. Yi, H., Shi, S., Ding, M., Sun, J., Xu, K., Zhou, H., Wang, Z., Li, S., and Wang, G. (June, January 31). Segvoxelnet: Exploring semantic context and depth-aware features for 3d vehicle detection from point cloud. Proceedings of the 2020 IEEE International Conference on Robotics and Automation (ICRA), Paris, France.10.1109/ICRA40945.2020.9196556 ↗
13. Anand, B., Barsaiyan, V., Senapati, M., and Rajalakshmi, P. (2020, January 15). Region of interest and car detection using lidar data for advanced traffic management system. Proceedings of the 2020 IEEE 6th World Forum on Internet of Things (WF-IoT), Online.10.1109/WF-IoT48130.2020.9221354 ↗
14. Girshick, R., Donahue, J., Darrell, T., and Malik, J. (2014, January 23–28). Rich feature hierarchies for accurate object detection and semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.81 ↗
15. Girshick, R. (2015, January 7–13). Fast r-cnn. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.10.1109/ICCV.2015.169 ↗
16. Ren, S., He, K., Girshick, R., and Sun, J. (2015, January 7–12). Faster r-cnn: Towards real-time object detection with region proposal networks. Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada.
17. Cai, Z., and Vasconcelos, N. (2018, January 7–12). Cascade r-cnn: Delving into high quality object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Montreal, QC, Canada.10.1109/CVPR.2018.00644 ↗
18. Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. (2016, January 27–30). You only look once: Unified, real-time object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.91 ↗
19. Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.Y., and Berg, A.C. (2016, January 11–14). Ssd: Single shot multibox detector. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.10.1007/978-3-319-46448-0_2 ↗
20. Lin, T.Y., Goyal, P., Girshick, R., He, K., and Dollár, P. (2017, January 22–29). Focal loss for dense object detection. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.324 ↗
21. Zhang, S., Wen, L., Bian, X., Lei, Z., and Li, S.Z. (2018, January 18–23). Single-shot refinement neural network for object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00442 ↗
22. Tian, Z., Shen, C., Chen, H., and He, T. (November, January 27). FCOS: Fully Convolutional One-Stage Object Detection. Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea.10.1109/ICCV.2019.00972 ↗
23. Law, H., and Deng, J. (2018, January 8–14). Cornernet: Detecting objects as paired keypoints. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01264-9_45 ↗
24. Zhou, X., Zhuo, J., and Krahenbuhl, P. (2019, January 15–20). Bottom-up object detection by grouping extreme and center points. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00094 ↗
25. Zhang, S., Chi, C., Yao, Y., Lei, Z., and Li, S.Z. (2020, January 13–19). Bridging the Gap Between Anchor-Based and Anchor-Free Detection via Adaptive Training Sample Selection. Proceedings of the CVPR 2020: Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00978 ↗
26. Zhu, X., Su, W., Lu, L., Li, B., Wang, X., and Dai, J. (2020). Deformable DETR: Deformable Transformers for End-to-End Object Detection. arXiv.
27. Sun, P., Zhang, R., Jiang, Y., Kong, T., Xu, C., Zhan, W., Tomizuka, M., Li, L., Yuan, Z., and Wang, C. (2020). Sparse r-cnn: End-to-end object detection with learnable proposals. arXiv.10.1109/CVPR46437.2021.01422 ↗
28. Liu, S., Li, F., Zhang, H., Yang, X., Qi, X., Su, H., Zhu, J., and Zhang, L. (2022). Dab-detr: Dynamic anchor boxes are better queries for detr. arXiv.
29. Li, F., Zhang, H., Liu, S., Guo, J., Ni, L.M., and Zhang, L. (2022, January 21–24). Dn-detr: Accelerate detr training by introducing query denoising. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01325 ↗
30. Zhang, H., Li, F., Liu, S., Zhang, L., Su, H., Zhu, J., Ni, L.M., and Shum, H.Y. (2022). Dino: Detr with improved denoising anchor boxes for end-to-end object detection. arXiv.
31. Pathiraja, B., Gunawardhana, M., and Khan, M.H. (2023, January 17–24). Multiclass Confidence and Localization Calibration for Object Detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.01890 ↗
32. Ge, C., Song, Y., Ma, C., Qi, Y., and Luo, P. (IEEE Trans. Image Process., 2023). Rethinking Attentive Object Detection via Neural Attention Learning, IEEE Trans. Image Process., early access.10.1109/TIP.2023.3251693 ↗
33. Zohar, O., Wang, K.C., and Yeung, S. (2023, January 17–24). Prob: Probabilistic objectness for open world object detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.01101 ↗
34. Singh, B., Najibi, M., and Davis, L.S. (2018, January 3–8). SNIPER: Efficient multi-scale training. Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada.
35. Liu, S., Qi, L., Qin, H., Shi, J., and Jia, J. (2018, January 18–23). Path aggregation network for instance segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00913 ↗
36. Chen, K., Cao, Y., Loy, C.C., Lin, D., and Feichtenhofer, C. (2020). Feature Pyramid Grids. arXiv.
37. Li, Z., Liu, Y., Li, B., Hu, W., and Zhang, H. (2021, January 5–9). DSIC: Dynamic Sample-Individualized Connector for Multi-Scale Object Detection. Proceedings of the 2021 IEEE International Conference on Multimedia and Expo (ICME), Shenzhen, China.10.1109/ICME51207.2021.9428167 ↗
38. Yan, Z., Qi, Y., Li, G., Liu, X., Zhang, W., Yang, M.H., and Huang, Q. (IEEE Trans. Circuits Syst. Video Technol., 2023). Progressive Multi-resolution Loss for Crowd Counting, IEEE Trans. Circuits Syst. Video Technol., early access.10.1109/TCSVT.2023.3317518 ↗
39. Gu, J., Kwon, H., Wang, D., Ye, W., Li, M., Chen, Y.H., Lai, L., Chandra, V., and Pan, D.Z. (2022, January 19–21). Multi-scale high-resolution vision transformer for semantic segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01178 ↗
40. Li, Y., Wu, C.Y., Fan, H., Mangalam, K., Xiong, B., Malik, J., and Feichtenhofer, C. (2022, January 19–21). Mvitv2: Improved multiscale vision transformers for classification and detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00476 ↗
41. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2020). An image is worth 16 × 16 words: Transformers for image recognition at scale. arXiv.
42. Yuan, L., Chen, Y., Wang, T., Yu, W., Shi, Y., Jiang, Z., Tay, F.E., Feng, J., and Yan, S. (2021). Tokens-to-token vit: Training vision transformers from scratch on imagenet. arXiv.10.1109/ICCV48922.2021.00060 ↗
43. Han, K., Xiao, A., Wu, E., Guo, J., Xu, C., and Wang, Y. (2021). Transformer in transformer. arXiv.
44. Chu, X., Zhang, B., Tian, Z., Wei, X., and Xia, H. (2021). Do we really need explicit position encodings for vision transformers?. arXiv.
45. Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. (2021). Swin transformer: Hierarchical vision transformer using shifted windows. arXiv.10.1109/ICCV48922.2021.00986 ↗
46. Ye, H., Li, G., Qi, Y., Wang, S., Huang, Q., and Yang, M.H. (2022, January 19–21). Hierarchical modular network for video captioning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01741 ↗
47. Gu, X., Chen, G., Wang, Y., Zhang, L., Luo, T., and Wen, L. (2023, January 17–24). Text with Knowledge Graph Augmented Transformer for Video Captioning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.01816 ↗
48. An, D., Qi, Y., Li, Y., Huang, Y., Wang, L., Tan, T., and Shao, J. (2022). BEVBert: Topo-Metric Map Pre-training for Language-guided Navigation. arXiv.
49. Majumdar, A., Shrivastava, A., Lee, S., Anderson, P., Parikh, D., and Batra, D. (2020, January 23–28). Improving vision-and-language navigation with image-text pairs from the web. Proceedings of the Computer Vision—ECCV 2020: 16th European Conference, Glasgow, UK.10.1007/978-3-030-58539-6_16 ↗
50. Chen, Q., Tan, M., Qi, Y., Zhou, J., Li, Y., and Wu, Q. (2022, January 19–21). V2C: Visual voice cloning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.02056 ↗
51. Fastspeech: Fast, robust and controllable text to speech
52. Wang, W., Xie, E., Li, X., Fan, D.P., Song, K., Liang, D., Lu, T., Luo, P., and Shao, L. (2021). Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. arXiv.10.1109/ICCV48922.2021.00061 ↗
53. Wu, H., Xiao, B., Codella, N., Liu, M., Dai, X., Yuan, L., and Zhang, L. (2021). Cvt: Introducing convolutions to vision transformers. arXiv.10.1109/ICCV48922.2021.00009 ↗
54. Patel, K., Bur, A.M., Li, F., and Wang, G. (2022). Aggregating Global Features into Local Vision Transformer. arXiv.10.1109/ICPR56361.2022.9956379 ↗
55. Chen, Q., Wang, Y., Yang, T., Zhang, X., Cheng, J., and Sun, J. (2021). You only look one-level feature. arXiv.10.1109/CVPR46437.2021.01284 ↗
56. Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., and Lerer, A. (2017, January 9). Automatic Differentiation in Pytorch. Proceedings of the NIPS 2017 Workshop Autodiff, Long Beach, CA, USA.
57. Chen, K., Wang, J., Pang, J., Cao, Y., Xiong, Y., Li, X., Sun, S., Feng, W., Liu, Z., and Xu, J. (2019). MMDetection: Open MMLab Detection Toolbox and Benchmark. arXiv.
58. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., and Fei-Fei, L. (2009, January 20–25). Imagenet: A large-scale hierarchical image database. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.10.1109/CVPR.2009.5206848 ↗