A Survey of Data Representation for Multi-Modality Event Detection and Evolution
Source abstract
Abstract
The rapid growth of online data has made it very convenient for people to obtain information. However, it also leads to the problem of “information overload”. Therefore, how to detect hot events from the massive amount of information has always been a problem. With the development of multimedia platforms, event detection has gradually developed from traditional single modality detection to multi-modality detection and is receiving increasing attention. The goal of multi-modality event detection is to discover events from a huge amount of online data with different data structures, such as texts, images and videos. These data represent real-world events from different perspectives so that they can provide more information about an event. In addition, event evolution is also a meaningful research direction; it models how events change dynamically over time and has great significance for event analysis. This paper comprehensively reviews the existing research on event detection and evolution. We first give a series of necessary definitions of event detection and evolution. Next, we discuss the techniques of data representation for event detection, including textual, visual, and multi-modality content. Finally, we review event evolution under multi-modality data. Furthermore, we review several public datasets and compare their results. At the end of this paper, we provide a conclusion and discuss future work.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Video Analysis and Summarization · Advanced Text Analysis Techniques · Advanced Image and Video Retrieval Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Information, Renmin University of China, Beijing 100872, China
School of Artificial Intelligence, Beijing Technology and Business University, Beijing 100048, China
Funding
Humanity and Social Science Youth Foundation of Ministry of Education of China · National Natural Science Foundation of China · Open Project Program of National Engineering Laboratory for Agri-product Quality Traceability
References (100+)
Show source-provided references
1. Adaptive online event detection in news streams10.1016/j.knosys.2017.09.039 ↗
2. A graph analytical approach for topic detection10.1145/2542214.2542215 ↗
3. Spina, D., Gonzalo, J., and Amigó, E. (2014, January 6–11). Learning similarity functions for topic detection in online reputation monitoring. Proceedings of the 37th International ACM SIGIR Conference on Research & Development in Information Retrieval, Gold Coast, Australia.10.1145/2600428.2609621 ↗
4. Hot topic detection research of internet public opinion based on affinity propagation clustering10.1007/978-94-007-1839-5_28 ↗
5. Tweet analysis for real-time event detection and earthquake reporting system development10.1109/TKDE.2012.29 ↗
6. Hot Topic Extraction Based on Timeline Analysis and Multidimensional Sentence Modeling10.1109/TKDE.2007.1040 ↗
7. A review of approaches for topic detection in Twitter10.1080/0952813X.2020.1785019 ↗
8. A survey of techniques for event detection in twitter10.1111/coin.12017 ↗
9. Review on event detection techniques in social multimedia10.1108/OIR-08-2015-0281 ↗
10. Panagiotou, N., Katakis, I., and Gunopulos, D. (2016). Detecting events in online social networks: Definitions, trends and challenges. Solving Large Scale Learning Tasks. Challenges and Algorithms, Springer.10.1007/978-3-319-41706-6_2 ↗
11. A survey of event detection techniques in online social networks10.1007/s13278-016-0414-1 ↗
12. Event-based media processing and analysis: A survey of the literature10.1016/j.imavis.2016.05.005 ↗
13. Multimodal classification of events in social media10.1016/j.imavis.2015.12.004 ↗
14. A survey on trends of cross-media topic evolution map10.1016/j.knosys.2017.03.009 ↗
15. A survey of event analysis and mining from social multimedia10.1007/s11042-019-7567-7 ↗
16. A survey on multi-modal social event detection10.1016/j.knosys.2020.105695 ↗
17. Nallapati, R., Feng, A., Peng, F., and Allan, J. (2004, January 8–13). Event threading within news topics. Proceedings of the Thirteenth ACM International Conference on Information and Knowledge Management, Washington, DC, USA.10.1145/1031171.1031258 ↗
18. A Survey on Topic Detection and Tracking Methods in Social Media
19. James, A. (2012). Topic Detection and Tracking: Event Based Information Organization, Springer.
20. A vector space model for automatic indexing10.1145/361219.361220 ↗
21. Jones, K.S. (1972). A Statistical Interpretation of Term Specificity and Its Application in Retrieval. J. Doc., 28, Available online: http://www.staff.city.ac.uk/~sbrp622/idfpapers/ksj_orig.pdf.10.1108/eb026526 ↗
22. Term-weighting approaches in automatic text retrieval10.1016/0306-4573(88)90021-0 ↗
23. Zhang, K., Zi, J., and Wu, L.G. (2007, January 23–27). New event detection based on indexing-tree and named entity. Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Amsterdam, The Netherlands.10.1145/1277741.1277780 ↗
24. Mihalcea, R., and Tarau, P. (2004, January 25–26). Textrank: Bringing order into text. Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, Barcelona, Spain.
25. Yang, Y., Pierce, T., and Carbonell, J.G. (1998, January 24–28). A study of retrospective and on-line event detection. Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Melbourne, Australia.10.1145/290941.290953 ↗
26. Allan, J., Papka, R., and Lavrenko, V. (1998, January 24–28). On-line new event detection and tracking. Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Melbourne, Australia.10.1145/290941.290954 ↗
27. Petrović, S., Osborne, M., and Lavrenko, V. (2010, January 2–4). Streaming first story detection with application to twitter. Proceedings of the Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Los Angeles, CA, USA.
28. Sankaranarayanan, J., Samet, H., Teitler, B.E., Ieberman, M.D., and Sperling, J. (2009, January 4–6). Twitterstand: News in tweet. Proceedings of the 17th ACM Sigspatial International Conference on Advances in Geographic Information Systems, Seattle, WA, USA.10.1145/1653771.1653781 ↗
29. Vakali, A., Giatsoglou, M., and Antaris, S. (2012, January 16–20). Social networking trends and dynamics detection via a cloud-based framework design. Proceedings of the 21st International Conference on World Wide Web, Lyon, France.10.1145/2187980.2188263 ↗
30. Brants, T., Chen, F., and Farahat, A. (August, January 28). A system for new event detection. Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), Toronto, ON, Canada.
31. Liu, L., Kang, J., Yu, J., and Wang, Z. (November, January 30). A comparative study on unsupervised feature selection methods for text clustering. Proceedings of the 2005 International Conference on Natural Language Processing and Knowledge Engineering, Wuhan, China.
32. Latent dirichlet allocation
33. Indexing by latent semantic analysis10.1002/(SICI)1097-4571(199009)41:6<391::AID-ASI1>3.0.CO;2-9 ↗
34. Hofmann, T. (1999, January 15–19). Probabilistic latent semantic indexing. Proceedings of the 22nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Berkeley, CA USA.10.1145/312624.312649 ↗
35. Supervised topic models
36. Andrzejewski, D., Zhu, X., and Craven, M. (2009, January 14–18). Incorporating domain knowledge into topic modeling via Dirichlet forest priors. Proceedings of the 26th Annual International Conference on Machine Learning, Montreal, QC, Canada.10.1145/1553374.1553378 ↗
37. Blei, D.M., Griffiths, T.L., Jordan, M.I., and Tenenbaum, J.B. (2004). Hierarchical topic models and the nested Chinese restaurant process. Advances Neural Information Processing Systems 16, MIT Press. Available online: https://proceedings.neurips.cc/paper/2003/file/7b41bfa5085806dfa24b8c9de0ce567f-Paper.pdf.
38. Newsminer: Multifaceted news analysis for event search10.1016/j.knosys.2014.11.017 ↗
39. Li, Z., Wang, B., Li, M., and Ma, W. (2005, January 15–19). A probabilistic model for retrospective news event detection. Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Salvador, Brazil.10.1145/1076034.1076055 ↗
40. A hybrid term–term relations analysis approach for topic detection10.1016/j.knosys.2015.11.006 ↗
41. Semantic-based topic detection using markov decision processes10.1016/j.neucom.2017.02.020 ↗
42. Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013). Efficient estimation of word representations in vector space. arXiv.
43. A neural model for joint event detection and prediction10.1016/j.neucom.2020.05.023 ↗
44. Long short-term memory10.1162/neco.1997.9.8.1735 ↗
45. Bodrunova, S.S., Orekhov, A.V., Blekanov, I.S., Lyudkevich, N.S., and Tarasov, N.A. (2020). Topic detection based on sentence embeddings and agglomerative clustering with markov moment. Future Internet, 12.10.3390/fi12090144 ↗
46. Imran, N., Liu, J., Luo, J., and Shah, M. (2009, January 19–24). Event Recognition from Photo Collections via Pagerank. Proceedings of the 17th ACM International Conference on Multimedia, Beijing, China. Available online: https://dl.acm.org/doi/epdf/10.1145/1631272.1631371.10.1145/1631272.1631371 ↗
47. Distinctive image features from scale-invariant keypoints10.1023/B:VISI.0000029664.99615.94 ↗
48. Bossard, L., Guillaumin, M., and Van Gool, L. (2013, January 1–8). Event recognition in photo collections with a stopwatch HMM. Proceedings of the IEEE International Conference on Computer Vision, Washington, DC, USA.10.1109/ICCV.2013.151 ↗
49. Robust event discovery from photo collections using Signature Image Bases (SIBs)10.1007/s11042-012-1153-6 ↗
50. A scalable algorithm for extraction and clustering of event-related pictures10.1007/s11042-012-1087-z ↗
51. Cluster-based Landmark and Event Detection on Tagged Photo Collections10.1109/MMUL.2010.68 ↗
52. Temporal event clustering for digital photo collections10.1145/1083314.1083317 ↗
53. Guo, C., and Tian, X. (2015, January 19–21). Event recognition in personal photo collections using hierarchical model and multiple features. Proceedings of the 2015 IEEE 17th International Workshop on Multimedia Signal Processing (MMSP), Xiamen, China.
54. Multigranular event recognition of personal photo albums10.1109/TMM.2017.2777664 ↗
55. Event photo mining from twitter using keyword bursts and image clustering10.1016/j.neucom.2015.02.081 ↗
56. Zaharieva, M., Zeppelzauer, M., and Breiteneder, C. (2013, January 16–20). Automated social event detection in large photo collections. Proceedings of the 3rd ACM Conference on International Conference on Multimedia Retrieval, Dallas, TX, USA.10.1145/2461466.2461495 ↗
57. Hamrouni, A., Ghazzai, H., Frikha, M., and Massoud, Y. (2019, January 4–7). A photo-based mobile crowdsourcing framework for event reporting. Proceedings of the 2019 IEEE 62nd International Midwest Symposium on Circuits and Systems (MWSCAS), Dallas, TX, USA.10.1109/MWSCAS.2019.8884949 ↗
58. Tang, K., Yao, B., Fei-Fei, L., and Koller, D. (2013, January 1–8). Combining the right features for complex event recognition. Proceedings of the 2013 IEEE International Conference on Computer Vision, Sydney, Australia.10.1109/ICCV.2013.335 ↗
59. Performance evaluation of local colour invariants10.1016/j.cviu.2008.07.003 ↗
60. Lan, Z.Z., Yang, Y., Ballas, N., Yu, S., and Haputmann, A. (2014). Resource constrained multimedia event detection. International Conference on Multi Media Modeling, Springer.10.1007/978-3-319-04114-8_33 ↗
61. Tamrakar, A., Ali, S., Yu, Q., Liu, J., Javed, O., Divakaran, A., Cheng, H., and Sawhney, H. (2012, January 16–21). Evaluation of low-level features and their combinations for complex event detection in open source videos. Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition, Washington, DC, USA.10.1109/CVPR.2012.6248114 ↗
62. Ming, Y. (2013, January 8–14). Human activity recognition based on 3d mesh mosift feature descriptor. Proceedings of the 2013 International Conference on Social Computing, Washington, DC, USA.10.1109/SocialCom.2013.151 ↗
63. Yang, Y., Ma, Z., Xu, Z., Yan, S., and Hauptmann, A.G. (2013, January 1–8). How related exemplars help complex event detection in web videos?. Proceedings of the 2013 IEEE International Conference on Computer Vision, Sydney, Australia.10.1109/ICCV.2013.456 ↗
64. On space-time interest points10.1007/s11263-005-1838-7 ↗
65. Klaser, A., Marszałek, M., and Schmid, C. (2008, January 1–4). A spatio-temporal descriptor based on 3d-gradients. Proceedings of the BMVC 2008-19th British Machine Vision Conference. British Machine Vision Association, Norwich, UK.10.5244/C.22.99 ↗
66. Video event detection, classification and retrieval using ensemble feature selection10.1007/s10586-021-03308-1 ↗
67. Semantic model vectors for complex video event recognition10.1109/TMM.2011.2168948 ↗
68. Gkalelis, N., Mezaris, V., and Kompatsiaris, I. (2011, January 13–15). High-level event detection in video exploiting discriminant concepts. Proceedings of the 2011 9th International Workshop on Content-Based Multimedia Indexing (CBMI), Madrid, Spain.10.1109/CBMI.2011.5972525 ↗
69. Yu, Q., Liu, J., Cheng, H., Divakaran, A., and Sawhney, H. (November, January 29). Multimedia event recounting with concept based representation. Proceedings of the 20th ACM International Conference on Multimedia, Nara, Japan.
70. Gan, C., Wang, N., Yang, Y., Yeung, D.Y., and Hauptmann, A.G. (2015, January 7–12). Devnet: A deep event network for multimedia event detection and evidence recounting. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298872 ↗
71. Xu, Z., Yang, Y., and Hauptmann, A.G. (2015, January 7–12). A discriminative CNN video representation for event detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298789 ↗
72. Zha, S., Luisier, F., Andrews, W., Srivastava, N., and Salakhutdinov, R. (2015). Exploiting image-trained CNN architectures for unconstrained video classification. arXiv.10.5244/C.29.60 ↗
73. Ye, G., Li, Y., Xu, H., Liu, D., and Chang, S.F. (2015, January 26–30). Eventnet: A large scale structured concept library for complex event detection in video. Proceedings of the 23rd ACM International Conference on Multimedia, Brisbane, Australia.10.1145/2733373.2806221 ↗
74. Multimodal deep representation learning for video classification10.1007/s11280-018-0548-3 ↗
75. Automatic Video Event Detection for Imbalance Data Using Enhanced Ensemble Deep Learning10.1142/S1793351X17400050 ↗
76. Xu, H., Li, B., Ramanishka, V., Sigal, L., and Saenko, K. (2019, January 7–11). Joint event detection and description in continuous video streams. Proceedings of the 2019 IEEE winter conference on applications of computer vision (WACV), Waikoloa Village, HI, USA.10.1109/WACV.2019.00048 ↗
77. Video event classification based on two-stage neural network10.1007/s11042-019-08457-5 ↗
78. Schinas, M., Papadopoulos, S., Petkos, G., Kompatsiaris, Y., and Mitkas, P.A. (2015, January 26–30). Multimodal graph-based event detection and summarization in social media streams. Proceedings of the 23rd ACM international conference on Multimedia, Brisbane, Australia.10.1145/2733373.2809933 ↗
79. Brenner, M., and Izquierdo, E. (2014, January 2–4). Multimodal detection, retrieval and classification of social events in web photo collections. Proceedings of the ICMR 2014 Workshop on Social Events in Web Multimedia (SEWM), Glasgow, UK.
80. Sutanto, T.E., and Nayak, R. (2013, January 18–19). Admrg@ mediaeval 2013 social event detection. Proceedings of the MediaEval 2013 Multimedia Benchmark Workshop [CEUR Workshop Proceedings], Barcelona, Spain.
81. Automatic visual concept learning for social event understanding10.1109/TMM.2015.2393635 ↗
82. Mei, T., Wang, B., Hua, X.S., Zhou, H.Q., and Li, S. (2006, January 9–12). Probabilistic multimodality fusion for event based home photo clustering. Proceedings of the 2006 IEEE International Conference on Multimedia and Expo, Toronto, ON, Canada.10.1109/ICME.2006.262891 ↗
83. Singh, B., Han, X., Wu, Z., Morariu, V.I., and Davis, L.S. (2015, January 13–16). Selecting relevant web trained concepts for automated event retrieval. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.10.1109/ICCV.2015.518 ↗
84. Xu, Z., Tsang, I.W., Yang, Y., Ma, Z., and Hauptmann, A.G. (2014, January 23–28). Event detection using multi-level relevance labels and multiple features. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.20 ↗
85. Ma, Y., Li, Q., Yang, Z., Zheng, L., and Chan, A.B. (2015, January 13–17). An SVD-based Multimodal Clustering method for Social Event Detection. Proceedings of the 2015 31st IEEE International Conference on Data Engineering Workshops, Seoul, Korea.10.1109/ICDEW.2015.7129577 ↗
86. Gupta, S.K., Phung, D., Adams, B., Tran, T., and Venkatesh, S. (2010, January 25–28). Nonnegative shared subspace learning and its application to social media retrieval. Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Washington, DC, USA.10.1145/1835804.1835951 ↗
87. Multi-view concept learning for data representation10.1109/TKDE.2015.2448542 ↗
88. Zhu, F., Shao, L., and Yu, M. (2014, January 3–7). Cross-modality submodular dictionary learning for information retrieval. Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management, Shanghai, China.10.1145/2661829.2661926 ↗
89. Xu, X., Yang, Y., Shimada, A., Taniguchi, R.I., and He, L. (2015, January 26–30). Semi-supervised coupled dictionary learning for cross-modal retrieval in internet images and texts. Proceedings of the 23rd ACM International Conference on Multimedia, Brisbane, Australia.10.1145/2733373.2806346 ↗
90. Topic detection in cross-media: A semi-supervised co-clustering approach10.1007/s13735-014-0056-x ↗
91. Yu, Z., Wu, F., Yang, Y., Tian, Q., Luo, J., and Zhuang, Y. (2014, January 6–11). Discriminative coupled dictionary hashing for fast cross-media retrieval. Proceedings of the ACM SIGIR Conference Research and Development Information Retriev, Gold Coast, Australia.10.1145/2600428.2609563 ↗
92. Click prediction for web image reranking using multimodal sparse coding10.1109/TIP.2014.2311377 ↗
93. Sharma, A., Kumar, A., Daume, H., and Jacobs, D.W. (2012, January 16–21). Generalized multiview analysis: A discriminative latent space. Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA.10.1109/CVPR.2012.6247923 ↗
94. Multimedia summarization for social events in microblog stream10.1109/TMM.2014.2384912 ↗
95. Cai, H., Yang, Y., Li, X., and Zi, H. (2015, January 26–30). What are popular: Exploring twitter features for event detection, tracking and visualization. Proceedings of the 23rd ACM International Conference on Multimedia, Brisbane, Australia.10.1145/2733373.2806236 ↗
96. Social event classification via boosted multimodal supervised latent dirichlet allocation10.1145/2659521 ↗
97. Qian, S., Zhang, T., and Xu, C. (2014, January 10–12). Multi-modal supervised latent dirichlet allocation for event classification in social media. Proceedings of the International Conference on Internet Multimedia Computing and Service, Xiamen, China.10.1145/2632856.2632893 ↗
98. Multi-modal event topic model for social event analysis10.1109/TMM.2015.2510329 ↗
99. Bi-Level Semantic Representation Analysis for Multimedia Event Detection10.1109/TCYB.2016.2539546 ↗
100. Lv, J., Liang, J., and Yang, Z. (2020, January 9–11). HGE2MED: Heterogeneous Graph Embedding for Multi-domain Event Detection. Proceedings of the 2020 IEEE 32nd International Conference on Tools with Artificial Intelligence (ICTAI), Baltimore, MD, USA.10.1109/ICTAI50040.2020.00160 ↗