Temporal-Guided Label Assignment for Video Object Detection
Source abstract
Abstract
In video object detection, the deterioration of an object’s appearance in a single frame brings challenges for recognition; therefore, it is natural to exploit temporal information to boost the robustness of video object detection. Existing methods usually utilize temporal information to enhance features, often ignoring the information in label assignments. Label assignment, which assigns labels to anchors for training, is an essential part of object detection. It is also challenged in video object detection and can be improved by temporal information. In this work, a temporal-guided label assignment framework is proposed for the learning task of a region proposal network (RPN). Specifically, we propose a feature instructing module (FIM) to establish the relation model among labels through feature similarity in the temporal dimension. The proposed video object detection framework was evaluated on the ImageNet VID benchmark. Without any additional inference cost, our work obtained a 0.8 mean average precision (mAP(%)) improvement over the baseline and achieved a mAP(%) of 82.0. The result was on par with the state-of-the-art accuracy without using any post-processing methods.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Neural Network Applications · Advanced Image and Video Retrieval Techniques · Visual Attention and Saliency Detection
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer and Communication Engineering, University of Science and Technology Beijing, Beijing 100083, China
Funding
National Key Research and Development Program of China · National Natural Science Foundation of China · Interdisciplinary Research Project for Young Teachers of USTB
References (33)
Show source-provided references
1. Zhu, X., Xiong, Y., Dai, J., Yuan, L., and Wei, Y. (2017, January 21–26). Deep feature flow for video recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.441 ↗
2. Zhu, X., Wang, Y., Dai, J., Yuan, L., and Wei, Y. (2017, January 22–29). Flow-guided feature aggregation for video object detection. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Venice, Italy.10.1109/ICCV.2017.52 ↗
3. Zhu, X., Dai, J., Yuan, L., and Wei, Y. (2018, January 18–23). Towards high performance video object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.10.1109/CVPR.2018.00753 ↗
4. Dosovitskiy, A., Fischer, P., Ilg, E., Hausser, P., Hazirbas, C., Golkov, V., van der Smagt, P., Cremers, D., and Brox, T. (2015, January 7–13). Flownet: Learning optical flow with convolutional networks. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Santiago, Chile.10.1109/ICCV.2015.316 ↗
5. Guo, C., Fan, B., Gu, J., Zhang, Q., Xiang, S., Prinet, V., and Pan, C. (2019, January 27–28). Progressive sparse local attention for video object detection. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea.10.1109/ICCV.2019.00401 ↗
6. Jiang, Z., Liu, Y., Yang, C., Liu, J., Gao, P., Zhang, Q., Xiang, S., and Pan, C. (2020). Learning where to focus for efficient video object detection. European Conference on Computer Vision, Springer.10.1007/978-3-030-58517-4_2 ↗
7. Bertasius, G., Torresani, L., and Shi, J. (2018, January 8–14). Object detection in video with spatiotemporal sampling networks. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01258-8_21 ↗
8. Wu, H., Chen, Y., Wang, N., and Zhang, Z. (2019, January 27–28). Sequence level semantics aggregation for video object detection. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea.10.1109/ICCV.2019.00931 ↗
9. Deng, J., Pan, Y., Yao, T., Zhou, W., Li, H., and Mei, T. (2019, January 27–28). Relation distillation networks for video object detection. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea.10.1109/ICCV.2019.00712 ↗
10. Han, M., Wang, Y., Chang, X., and Qiao, Y. (2020). Mining inter-video proposal relations for video object detection. European Conference on Computer Vision, Springer.10.1007/978-3-030-58589-1_26 ↗
11. Gong, T., Chen, K., Wang, X., Chu, Q., Zhu, F., Lin, D., Yu, N., and Feng, H. (2021, January 2–9). Temporal ROI align for video object recognition. Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), Virtual.10.1609/aaai.v35i2.16234 ↗
12. Lin, T.Y., Goyal, P., Girshick, R., He, K., and Dollár, P. (2017, January 22–29). Focal loss for dense object detection. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Venice, Italy.10.1109/ICCV.2017.324 ↗
13. Tian, Z., Shen, C., Chen, H., and He, T. (2019, January 27–28). Fcos: Fully convolutional one-stage object detection. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea.10.1109/ICCV.2019.00972 ↗
14. Wang, J., Chen, K., Yang, S., Loy, C.C., and Lin, D. (2019, January 15–20). Region proposal by guided anchoring. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00308 ↗
15. Yang, T., Zhang, X., Li, Z., Zhang, W., and Sun, J. (2018). Metaanchor: Learning to detect objects with customized anchors. Advances in Neural Information Processing Systems, Curran Associates.
16. Zhu, C., He, Y., and Savvides, M. (2019, January 15–20). Feature selective anchor-free module for single-shot object detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00093 ↗
17. Ge, Z., Liu, S., Li, Z., Yoshie, O., and Sun, J. (2021, January 20–25). Ota: Optimal transport assignment for object detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.10.1109/CVPR46437.2021.00037 ↗
18. Zhu, B., Wang, J., Jiang, Z., Zong, F., Liu, S., Li, Z., and Sun, J. (2020). Autoassign: Differentiable label assignment for dense object detection. arXiv.
19. Girshick, R., Donahue, J., Darrell, T., and Malik, J. (2014, January 23–28). Rich feature hierarchies for accurate object detection and semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Columbus, OH, USA.10.1109/CVPR.2014.81 ↗
20. Girshick, R. (2015, January 7–13). Fast r-cnn. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Santiago, Chile.10.1109/ICCV.2015.169 ↗
21. Ren, S., He, K., Girshick, R., and Sun, J. (2015). Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in Neural Information Processing Systems, Curran Associates.
22. Dai, J., Li, Y., He, K., and Sun, J. (2016). R-fcn: Object detection via region-based fully convolutional networks. Advances in Neural Information Processing Systems, Curran Associates.
23. Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. (2016, January 27–30). You only look once: Unified, real-time object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.91 ↗
24. Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.Y., and Berg, A.C. (2016). Ssd: Single shot multibox detector. European Conference on Computer Vision, Springer.10.1007/978-3-319-46448-0_2 ↗
25. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, Curran Associates.
26. Video super-resolution based on a spatio-temporal matching network10.1016/j.patcog.2020.107619 ↗
27. Attention-aware perceptual enhancement nets for low-resolution image classification10.1016/j.ins.2019.12.013 ↗
28. Imagenet large scale visual recognition challenge10.1007/s11263-015-0816-y ↗
29. T-cnn: Tubelets with convolutional neural networks for object detection from videos10.1109/TCSVT.2017.2736553 ↗
30. Lee, B., Erdenee, E., Jin, S., Nam, M.Y., Jung, Y.G., and Rhee, P.K. (2016). Multi-class multi-object tracking using changing point detection. European Conference on Computer Vision, Springer.10.1007/978-3-319-48881-3_6 ↗
31. Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., and Antiga, L. (2019). Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, Curran Associates.
32. Feichtenhofer, C., Pinz, A., and Zisserman, A. (2017, January 22–29). Detect to track and track to detect. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Venice, Italy.10.1109/ICCV.2017.330 ↗
33. Shvets, M., Liu, W., and Berg, A.C. (2019, January 27–28). Leveraging Long-Range Temporal Relationships Between Proposals for Video Object Detection. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea.10.1109/ICCV.2019.00985 ↗