Context-Based Structure Mining Methodology for Static Object Re-Identification in Broadcast Content
Source abstract
Abstract
Technological advancement, in addition to the pandemic, has given rise to an explosive increase in the consumption and creation of multimedia content worldwide. This has motivated people to enrich and publish their content in a way that enhances the experience of the user. In this paper, we propose a context-based structure mining pipeline that not only attempts to enrich the content, but also simultaneously splits it into shots and logical story units (LSU). Subsequently, this paper extends the structure mining pipeline to re-ID objects in broadcast videos such as SOAPs. We hypothesise the object re-ID problem of SOAP-type content to be equivalent to the identification of reoccurring contexts, since these contexts normally have a unique spatio-temporal similarity within the content structure. By implementing pre-trained models for object and place detection, the pipeline was evaluated using metrics for shot and scene detection on benchmark datasets, such as RAI. The object re-ID methodology was also evaluated on 20 randomly selected episodes from broadcast SOAP shows New Girl and Friends. We demonstrate, quantitatively, that the pipeline outperforms existing state-of-the-art methods for shot boundary detection, scene detection, and re-identification tasks.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Video Analysis and Summarization · Music and Audio Processing · Advanced Image and Video Retrieval Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
IDLab, Ghent University—imec, 9052 Ghent, Belgium
References (27)
Show source-provided references
1. Bagdanov, A.D., Bertini, M., Bimbo, A.D., Serra, G., and Torniai, C. (2007, January 17–19). Semantic annotation and retrieval of video events using multimedia ontologies. Proceedings of the International Conference on Semantic Computing (ICSC 2007), Irvine, CA, USA.10.1109/ICOSC.2007.4338414 ↗
2. Lu, Z., and Grauman, K. (2013, January 23–28). Story-driven summarization for egocentric video. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Portland, OR, USA.10.1109/CVPR.2013.350 ↗
3. Mahasseni, B., Lam, M., and Todorovic, S. (2017, January 21–26). Unsupervised Video Summarization with Adversarial LSTM Networks. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.318 ↗
4. Goyal, P., Hu, Z., Liang, X., Wang, C., and Xing, E.P. (2017). Nonparametric Variational Auto-encoders for Hierarchical Representation Learning. arXiv.10.1109/ICCV.2017.545 ↗
5. Han, J., Yang, L., Zhang, D., Chang, X., and Liang, X. (2018, January 18–23). Reinforcement Cutting-Agent Learning for Video Object Segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.10.1109/CVPR.2018.00946 ↗
6. Meng, J., Wang, H., Yuan, J., and Tan, Y. (2016, January 27–30). From Keyframes to Key Objects: Video Summarization by Representative Object Proposal Selection. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.118 ↗
7. Plummer, B.A., Brown, M., and Lazebnik, S. (2017, January 21–26). Enhancing Video Summarization via Vision-Language Embedding. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.118 ↗
8. Video content characterization and compaction for digital library applications10.1117/12.263440 ↗
9. Systematic evaluation of logical story unit segmentation10.1109/TMM.2002.802021 ↗
10. Petersohn, C. (2010). Temporal Video Segmentation, Jörg Vogt Verlag.
11. Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., and Torralba, A. (2016, January 27–30). Learning Deep Features for Discriminative Localization. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.319 ↗
12. Redmon, J., and Farhadi, A. (2018). YOLOv3: An Incremental Improvement. arXiv.
13. Altadmri, A., and Ahmed, A. (2009, January 18–19). Automatic semantic video annotation in wide domain videos based on similarity and commonsense knowledgebases. Proceedings of the 2009 IEEE International Conference on Signal and Image Processing Applications, Kuala Lumpur, Malaysia.10.1109/ICSIPA.2009.5478723 ↗
14. Using deep features for video scene detection and annotation10.1007/s11760-018-1244-6 ↗
15. A semantic-based video scene segmentation using a deep neural network10.1177/0165551518819964 ↗
16. Torralba, A., Murphy, K.P., Freeman, W.T., and Rubin, M.A. (2003, January 13–16). Context-based Vision System for Place and Object Recognition. Proceedings of the Ninth IEEE International Conference on Computer Vision, Nice, France.10.1109/ICCV.2003.1238354 ↗
17. Odobez, J.M., Gatica-Perez, D., and Guillemot, M. (2003). Spectral structuring of home videos. International Conference on Image and Video Retrieval, Springer.10.1007/3-540-45113-7_31 ↗
18. Kwon, Y.M., Song, C.J., and Kim, I.J. (August, January 30). A new approach for high level video structuring. Proceedings of the 2000 IEEE International Conference on Multimedia and Expo. ICME2000 Proceedings, Latest Advances in the Fast Changing World of Multimedia (Cat. No.00TH8532), New York, NY, USA.
19. Mitrović, D., Hartlieb, S., Zeppelzauer, M., and Zaharieva, M. (2010). Scene segmentation in artistic archive documentaries. Symposium of the Austrian HCI and Usability Engineering Group, Springer.10.1007/978-3-642-16607-5_27 ↗
20. Gygli, M. (2017). Ridiculously Fast Shot Boundary Detection with Fully Convolutional Neural Networks. arXiv.10.1109/CBMI.2018.8516556 ↗
21. State-of-the-art and future challenges in video scene detection: A survey10.1007/s00530-013-0306-4 ↗
22. Lin, T., Maire, M., Belongie, S.J., Bourdev, L.D., Girshick, R.B., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C.L. (2014). Microsoft COCO: Common Objects in Context. arXiv.10.1007/978-3-319-10602-1_48 ↗
23. Places: A 10 million Image Database for Scene Recognition10.1109/TPAMI.2017.2723009 ↗
24. Alhashim, I., and Wonka, P. (2019). High Quality Monocular Depth Estimation via Transfer Learning. arXiv.
25. Silberman, N., Hoiem, D., Kohli, P., and Fergus, R. (2012). Indoor Segmentation and Support Inference from RGBD Images. European Conference on Computer Vision, Springer.10.1007/978-3-642-33715-4_54 ↗
26. Temporal Video Segmentation to Scenes Using High-Level Audiovisual Features10.1109/TCSVT.2011.2138830 ↗
27. Baraldi, L., Grana, C., and Cucchiara, R. (2015). A Deep Siamese Network for Scene Detection in Broadcast Videos. arXiv.10.1145/2733373.2806316 ↗