SKVOS: Sketch-Based Video Object Segmentation with a Large-Scale Benchmark
Source abstract
Abstract
In this paper, we propose sketch-based video object segmentation (SKVOS), a novel task that segments objects consistently across video frames using human-drawn sketches as queries. Traditional reference-based methods, such as photo masks and language descriptions, are commonly used for segmentation. Photo masks provide high precision but are labor intensive, limiting scalability. While language descriptions are easy to provide, they often lack the specificity needed to distinguish visually similar objects within a frame. Despite their simplicity, sketches capture rich, fine-grained details of target objects and can be rapidly created, even by non-experts, making them an attractive alternative for segmentation tasks. We introduce a new approach that utilizes sketches as efficient and informative references for video object segmentation. To evaluate sketch-guided segmentation, we introduce a new benchmark consisting of three datasets: Sketch-DAVIS16, Sketch-DAVIS17, and Sketch-YouTube-VOS. Building on a memory-based framework for semi-supervised video object segmentation, we explore effective strategies for integrating sketch-based references. To ensure robust spatiotemporal coherence, we introduce two key innovations: the Temporal Relation Module and Sketch-Anchored Contrastive Learning. These modules enhance the model’s ability to maintain consistency both across time and across different object instances. Our method is evaluated on the Sketch-VOS benchmark, demonstrating superior performance with overall improvements of 1.9%, 3.3%, and 2.0% over state-of-the-art methods on the Sketch-YouTube-VOS, Sketch-DAVIS 2016, and Sketch-DAVIS 2017 validation sets, respectively. Additionally, on the YouTube-VOS validation set, our method outperforms the leading language-based VOS approach by 10.1%.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Visual Attention and Saliency Detection · Advanced Image and Video Retrieval Techniques · Video Surveillance and Tracking Methods
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Artificial Intelligence, Beijing University of Posts and Telecommunications, Beijing 100876, China
SketchX, Centre for Vision, Speech and Signal Processing, University of Surrey, Surrey GU2 7XH, UK
Department of Computer Science, National University of Singapore, Singapore 119077, Singapore
References (80)
Show source-provided references
1. Zhang, Y., Wei, Y., Jiang, D., Zhang, X., Zuo, W., and Tian, Q. (2023). Controlvideo: Training-free controllable text-to-video generation. arXiv.
2. Zhou, Q., Zhang, S., Yang, N., Qian, Y., and Li, H. (2024). Motion Control for Enhanced Complex Action Video Generation. arXiv.
3. Zhang, Z., Fidler, S., and Urtasun, R. (2016, January 27–30). Instance-level segmentation for autonomous driving with deep densely connected MRFs. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.79 ↗
4. Pei, G., Shen, F., Yao, Y., Xie, G.S., Tang, Z., and Tang, J. (2022, January 23–27). Hierarchical feature alignment network for unsupervised video object segmentation. Proceedings of the European Conference on Computer Vision, Tel Aviv, Israel.10.1007/978-3-031-19830-4_34 ↗
5. Wang, W., Lu, X., Shen, J., Crandall, D.J., and Shao, L. (November, January 27). Zero-shot video object segmentation via attentive graph neural networks. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.
6. Wang, W., Song, H., Zhao, S., Shen, J., Zhao, S., Hoi, S.C., and Ling, H. (2019, January 15–20). Learning unsupervised video object segmentation through visual attention. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00318 ↗
7. Yang, Z., Wang, Q., Bertinetto, L., Hu, W., Bai, S., and Torr, P.H. (November, January 27). Anchor diffusion for unsupervised video object segmentation. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.
8. Gowda, S.N., Eustratiadis, P., Hospedales, T., and Sevilla-Lara, L. (2020). Alba: Reinforcement learning for video object segmentation. arXiv.
9. Miao, J., Wei, Y., and Yang, Y. (2020, January 13–19). Memory aggregation networks for efficient interactive video object segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.01038 ↗
10. Cheng, H.K., Tai, Y.W., and Tang, C.K. (2021, January 20–25). Modular interactive video object segmentation: Interaction-to-mask, propagation and difference-aware fusion. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.00551 ↗
11. Yin, Z., Zheng, J., Luo, W., Qian, S., Zhang, H., and Gao, S. (2021, January 20–25). Learning to recommend frame for interactive video object segmentation in the wild. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.01519 ↗
12. Heo, Y., Koh, Y.J., and Kim, C.S. (2021, January 20–25). Guided interactive video object segmentation using reliability-based attention maps. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.00724 ↗
13. Xu, N., Yang, L., Fan, Y., Yang, J., Yue, D., Liang, Y., Price, B., Cohen, S., and Huang, T. (2018, January 8–14). Youtube-vos: Sequence-to-sequence video object segmentation. Proceedings of the European Conference on Computer Vision, Munich, Germany.10.1007/978-3-030-01228-1_36 ↗
14. Khoreva, A., Rohrbach, A., and Schiele, B. (2018, January 2–6). Video object segmentation with language referring expressions. Proceedings of the Asian Conference on Computer Vision, Perth, Australia.10.1007/978-3-030-20870-7_8 ↗
15. Seo, S., Lee, J.Y., and Han, B. (2020, January 23–28). URVOS: Unified referring video object segmentation network with a large-scale benchmark. Proceedings of the European Conference on Computer Vision, Glasgow, UK.10.1007/978-3-030-58555-6_13 ↗
16. Dey, S., Riba, P., Dutta, A., Llados, J., and Song, Y.Z. (2019, January 15–20). Doodle to search: Practical zero-shot sketch-based image retrieval. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00228 ↗
17. Collomosse, J., Bui, T., and Jin, H. (2019, January 15–20). Livesketch: Query perturbations for guided sketch-based visual search. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00299 ↗
18. Sain, A., Bhunia, A.K., Yang, Y., Xiang, T., and Song, Y.Z. (2021, January 20–25). Stylemeup: Towards style-agnostic sketch-based image retrieval. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.00840 ↗
19. Chowdhury, P.N., Bhunia, A.K., Gajjala, V.R., Sain, A., Xiang, T., and Song, Y.Z. (2022, January 18–24). Partially does it: Towards scene-level FG-SBIR with partial input. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00243 ↗
20. Bhunia, A.K., Yang, Y., Hospedales, T.M., Xiang, T., and Song, Y.Z. (2020, January 13–19). Sketch less for more: On-the-fly fine-grained sketch-based image retrieval. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00980 ↗
21. Bhunia, A.K., Chowdhury, P.N., Sain, A., Yang, Y., Xiang, T., and Song, Y.Z. (2021, January 20–25). More photos are all you need: Semi-supervised learning for fine-grained sketch based image retrieval. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.00423 ↗
22. Bhunia, A.K., Koley, S., Khilji, A.F.U.R., Sain, A., Chowdhury, P.N., Xiang, T., and Song, Y.Z. (2022, January 18–24). Sketching without worrying: Noise-tolerant sketch-based image retrieval. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00107 ↗
23. Zeng, Y., Lin, Z., and Patel, V.M. (2022, January 18–24). Sketchedit: Mask-free local image manipulation with partial sketches. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00586 ↗
24. Wang, S.Y., Bau, D., and Zhu, J.Y. (2021, January 10–17). Sketch your own gan. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada.10.1109/ICCV48922.2021.01379 ↗
25. Yang, R., Li, D., Hu, C., Hospedales, T., Zhang, H., and Song, Y.Z. (2023, January 20–24). Sketch-based Video Object Segmentation: Benchmark and Analysis. Proceedings of the 34th British Machine Vision Conference 2023, Aberdeen, UK.
26. Perazzi, F., Pont-Tuset, J., McWilliams, B., Van Gool, L., Gross, M., and Sorkine-Hornung, A. (2016, January 27–30). A benchmark dataset and evaluation methodology for video object segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.85 ↗
27. Pont-Tuset, J., Perazzi, F., Caelles, S., Arbeláez, P., Sorkine-Hornung, A., and Van Gool, L. (2017). The 2017 davis challenge on video object segmentation. arXiv.
28. Chowdhury, P.N., Bhunia, A.K., Sain, A., Koley, S., Xiang, T., and Song, Y.Z. (2023, January 17–24). What can human sketches do for object detection?. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.01448 ↗
29. Yang, Z., Tang, Y., Bertinetto, L., Zhao, H., and Torr, P.H. (2021, January 22–25). Hierarchical interaction network for video object segmentation from referring expressions. Proceedings of the British Machine Vision Association, Online.
30. Li, D., Li, R., Wang, L., Wang, Y., Qi, J., Zhang, L., Liu, T., Xu, Q., and Lu, H. (March, January 22). You only infer once: Cross-modal meta-transfer for referring video object segmentation. Proceedings of the AAAI Conference on Artificial Intelligence, Online.
31. Wu, D., Dong, X., Shao, L., and Shen, J. (2022, January 18–24). Multi-Level Representation Learning with Semantic Alignment for Referring Video Object Segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00494 ↗
32. Ding, Z., Hui, T., Huang, J., Wei, X., Han, J., and Liu, S. (2022, January 18–24). Language-bridged spatial-temporal interaction for referring video object segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00491 ↗
33. Botach, A., Zheltonozhskii, E., and Baskin, C. (2022, January 18–24). End-to-end referring video object segmentation with multimodal transformers. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00493 ↗
34. Wu, J., Jiang, Y., Sun, P., Yuan, Z., and Luo, P. (2022, January 18–24). Language as queries for referring video object segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00492 ↗
35. Miao, B., Bennamoun, M., Gao, Y., and Mian, A. (2023, January 1–6). Spectrum-guided multi-granularity referring video object segmentation. Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France.10.1109/ICCV51070.2023.00091 ↗
36. Yuan, L., Shi, M., Yue, Z., and Chen, Q. (2024, January 16–22). Losh: Long-short text joint prediction network for referring video object segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR52733.2024.01328 ↗
37. Voigtlaender, P., and Leibe, B. (2017). Online adaptation of convolutional neural networks for video object segmentation. arXiv.10.5244/C.31.116 ↗
38. Cheng, J., Tsai, Y.H., Wang, S., and Yang, M.H. (2017, January 22–29). Segflow: Joint learning for video object segmentation and optical flow. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.81 ↗
39. Voigtlaender, P., Chai, Y., Schroff, F., Adam, H., Leibe, B., and Chen, L.C. (2019, January 15–20). Feelvos: Fast end-to-end embedding learning for video object segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00971 ↗
40. Collaborative video object segmentation by multi-scale foreground-background integration
41. Liu, Y., Yu, R., Yin, F., Zhao, X., Zhao, W., Xia, W., and Yang, Y. (2022, January 23–27). Learning Quality-aware Dynamic Memory for Video Object Segmentation. Proceedings of the European Conference on Computer Vision, Tel Aviv, Israel.10.1007/978-3-031-19818-2_27 ↗
42. Hierarchical image saliency detection on extended CSSD10.1109/TPAMI.2015.2465960 ↗
43. Global contrast based salient region detection10.1109/TPAMI.2014.2345401 ↗
44. Oh, S.W., Lee, J.Y., Xu, N., and Kim, S.J. (November, January 27). Video object segmentation using space-time memory networks. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.
45. Rethinking space-time networks with improved memory coverage for efficient video object segmentation
46. Wang, L., Lu, H., Wang, Y., Feng, M., Wang, D., Yin, B., and Ruan, X. (2017, January 21–26). Learning to detect salient objects with image-level supervision. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.404 ↗
47. Zeng, Y., Zhang, P., Zhang, J., Lin, Z., and Lu, H. (November, January 27). Towards high-resolution salient object detection. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.
48. Cheng, H.K., Chung, J., Tai, Y.W., and Tang, C.K. (2020, January 13–19). Cascadepsp: Toward class-agnostic and very high-resolution segmentation via global and local refinement. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00891 ↗
49. Li, X., Wei, T., Chen, Y.P., Tai, Y.W., and Tang, C.K. (2020, January 13–19). Fss-1000: A 1000-class dataset for few-shot segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00294 ↗
50. Chang, A.X., Funkhouser, T., Guibas, L., Hanrahan, P., Huang, Q., Li, Z., Savarese, S., Savva, M., Song, S., and Su, H. (2015). Shapenet: An information-rich 3d model repository. arXiv.
51. The sketchy database: Learning to retrieve badly drawn bunnies10.1145/2897824.2925954 ↗
52. How do humans sketch objects?10.1145/2185520.2185540 ↗
53. Yu, Q., Liu, F., Song, Y.Z., Xiang, T., Hospedales, T.M., and Loy, C.C. (2016, January 27–30). Sketch me that shoe. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.93 ↗
54. Tripathi, A., Dani, R.R., Mishra, A., and Chakraborty, A. (2020, January 23–28). Sketch-guided object localization in natural images. Proceedings of the European Conference on Computer Vision, Glasgow, UK.10.1007/978-3-030-58539-6_32 ↗
55. Hu, C., Li, D., Song, Y.Z., Xiang, T., and Hospedales, T.M. (2018, January 18–23). Sketch-a-classifier: Sketch-based photo classifier generation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00952 ↗
56. Sketch-a-segmenter: Sketch-based photo segmenter generation10.1109/TIP.2020.3028292 ↗
57. Fine-grained instance-level sketch-based video retrieval10.1109/TCSVT.2020.3014491 ↗
58. Fine-grained video retrieval with scene sketches10.1109/TIP.2023.3278474 ↗
59. Caelles, S., Maninis, K.K., Pont-Tuset, J., Leal-Taixé, L., Cremers, D., and Van Gool, L. (2017, January 21–26). One-shot video object segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.565 ↗
60. Cheng, J., Tsai, Y.H., Hung, W.C., Wang, S., and Yang, M.H. (2018, January 18–23). Fast and accurate online video object segmentation via tracking parts. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00774 ↗
61. Cho, S., Lee, H., Lee, M., Park, C., Jang, S., Kim, M., and Lee, S. (2022, January 23–27). Tackling background distraction in video object segmentation. Proceedings of the European Conference on Computer Vision, Tel Aviv, Israel.10.1007/978-3-031-20047-2_26 ↗
62. Mao, Y., Wang, N., Zhou, W., and Li, H. (2021, January 10–17). Joint inductive and transductive learning for video object segmentation. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada.10.1109/ICCV48922.2021.00953 ↗
63. Lin, Z., Yang, T., Li, M., Wang, Z., Yuan, C., Jiang, W., and Liu, W. (2022, January 18–24). SWEM: Towards Real-Time Video Object Segmentation with Sequential Weighted Expectation-Maximization. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00142 ↗
64. Liu, Y., Yu, R., Wang, J., Zhao, X., Wang, Y., Tang, Y., and Yang, Y. (2022, January 23–27). Global Spectral Filter Memory Network for Video Object Segmentation. Proceedings of the European Conference on Computer Vision, Tel Aviv, Israel.10.1007/978-3-031-19818-2_37 ↗
65. Zhu, X., Su, W., Lu, L., Li, B., Wang, X., and Dai, J. (2020). Deformable detr: Deformable transformers for end-to-end object detection. arXiv.
66. The Hungarian method for the assignment problem10.1002/nav.3800020109 ↗
67. Ribeiro, L.S.F., Bui, T., Collomosse, J., and Ponti, M. (2020, January 13–19). Sketchformer: Transformer-based representation for sketched structure. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.
68. Bhunia, A.K., Gajjala, V.R., Koley, S., Kundu, R., Sain, A., Xiang, T., and Song, Y.Z. (2022, January 18–24). Doodle it yourself: Class incremental learning by drawing a few sketches. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00233 ↗
69. Woo, S., Jeon, S.Y., Park, J., Son, M., Lee, S., and Kim, C. (2024, January 3–8). Sketch-based Video Object Localization. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, Waikoloa, HI, USA.10.1109/WACV57701.2024.00829 ↗
70. Chowdhury, P.N., Sain, A., Bhunia, A.K., Xiang, T., Gryaditskaya, Y., and Song, Y.Z. (2022, January 23–27). Fs-coco: Towards understanding of freehand sketches of common objects in context. Proceedings of the European Conference on Computer Vision, Tel Aviv, Israel.10.1007/978-3-031-20074-8_15 ↗
71. Ha, D., Dai, A., and Le, Q.V. (2016). Hypernetworks. arXiv.
72. Vaswani, A. (2017, January 4–9). Attention is all you need. Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, CA USA.
73. Teed, Z., and Deng, J. (2020, January 23–28). RAFT: Recurrent all-pairs field transforms for optical flow. Proceedings of the European Conference on Computer Vision, Glasgow, UK.10.1007/978-3-030-58536-5_24 ↗
74. Oord, A.v.d., Li, Y., and Vinyals, O. (2018). Representation learning with contrastive predictive coding. arXiv.
75. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., and Fei-Fei, L. (2009, January 20–25). Imagenet: A large-scale hierarchical image database. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.10.1109/CVPR.2009.5206848 ↗
76. Ha, D., and Eck, D. (2017). A neural representation of sketch drawings. arXiv.
77. Caelles, S., Montes, A., Maninis, K.K., Chen, Y., Van Gool, L., Perazzi, F., and Pont-Tuset, J. (2018). The 2018 davis challenge on video object segmentation. arXiv.10.1109/CVPR.2017.565 ↗
78. Yang, Z., Wang, J., Tang, Y., Chen, K., Zhao, H., and Torr, P.H. (2022, January 18–24). Lavt: Language-aware vision transformer for referring image segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01762 ↗
79. Lin, D., Dai, J., Jia, J., He, K., and Sun, J. (2016, January 27–30). Scribblesup: Scribble-supervised convolutional networks for semantic segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.344 ↗
80. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗