Action Detection for Wildlife Monitoring with Camera Traps Based on Segmentation with Filtering of Tracklets (SWIFT) and Mask-Guided Action Recognition (MAROON)
Source abstract
Abstract
Behavioral analysis of animals in the wild plays an important role for ecological research and conservation and has been mostly performed by researchers. We introduce an action detection approach that automates this process by detecting animals and performing action recognition on the detected animals in camera trap videos. Our action detection approach is based on SWIFT (segmentation with filtering of tracklets), which we have already shown to successfully detect and track animals in wildlife videos, and MAROON (mask-guided action recognition), an action recognition network that we are introducing here. The basic ideas of MAROON are the exploitation of the instance masks detected by SWIFT and a triple-stream network. The instance masks enable more accurate action recognition, especially if multiple animals appear in a video at the same time. The triple-stream approach extracts features for the motion and appearance of the animal. We evaluate the quality of our action recognition on two self-generated datasets, from an animal enclosure and from the wild. These datasets contain videos of red deer, fallow deer and roe deer, recorded both during the day and night. MAROON improves the action recognition accuracy compared to other state-of-the-art approaches by an average of 10 percentage points on all analyzed datasets and achieves an accuracy of 69.16% on the Rolandseck Daylight dataset, in which 11 different action classes occur. Our action detection system makes it possible todrasticallyreduce the manual work of ecologists and at the same time gain new insights through standardized results.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Human Pose and Action Recognition · Video Surveillance and Tracking Methods · Multimodal Machine Learning Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Science IV, University of Bonn, Friedrich-Hirzebruch-Allee 8, 53115 Bonn, Germany
Department of National Park Monitoring and Animal Management, Bavarian Forest National Park, Freyunger Str. 2, 94481 Grafenau, Germany
Forest Zoology, Institute of Forest Botany and Forest Zoology, Technical University of Dresden, Pienner Str. 7, 01737 Tharandt, Germany
Wildlife Sciences, Faculty of Forest Sciences and Forest Ecology, University of Göttingen, Büsgenweg 3, 37077 Göttingen, Germany
Faculty of Environment and Natural Resources, University of Freiburg, Tennenbacher Straße 4, 79106 Freiburg im Breisgau, Germany
Institute of Forestry and Wildlife Management, Inland Norway University of Applied Science, NO-2480 Koppang, Norway
Funding
German Federal Ministry of Education and Research (Bundesministerium für Bildung und Forschung (BMBF)), Bonn, Germany
References (64)
Show source-provided references
1. Integrating animal behavior and conservation biology: A conceptual framework10.1093/beheco/arq224 ↗
2. A review of camera trapping for conservation behaviour research10.1002/rse2.48 ↗
3. Changing use of camera traps in mammalian field research: Habitats, taxa and study types10.1111/j.1365-2907.2012.00216.x ↗
4. Snap happy: Camera traps are an effective sampling tool when compared with alternative methods10.1098/rsos.181748 ↗
5. Hongo, S., Nakashima, Y., Yajima, G., and Hongo, S. (2021). A practical guide for estimating animal density using camera traps: Focus on the REST model. bioRxiv.10.1101/2021.05.18.444583 ↗
6. Evaluating camera traps as an alternative to live trapping for estimating the density of snowshoe hares (Lepus americanus) and red squirrels (Tamiasciurus hudsonicus)10.1007/s10344-016-1064-3 ↗
7. Henrich, M., Burgueño, M., Hoyer, J., Haucke, T., Steinhage, V., Kühl, H.S., and Heurich, M. (2023). A semi-automated camera trap distance sampling approach for population density estimation. Remote Sens. Ecol. Conserv.10.1002/rse2.362 ↗
8. Further notes on the analysis of mammal inventory data collected with camera traps10.1111/j.1469-1795.2008.00181.x ↗
9. Estimating occupancy of a data deficient mammalian species living in tropical rainforests: Sun bears in the Kerinci Seblat region, Sumatra10.1016/j.biocon.2007.01.016 ↗
10. Investigating animal activity patterns and temporal niche partitioning using camera-trap data: Challenges and opportunities10.1002/rse2.60 ↗
11. An invasive-native mammalian species replacement process captured by camera trap survey random encounter models10.1002/rse2.11 ↗
12. Green, S.E., Rees, J.P., Stephens, P.A., Hill, R.A., and Giordano, A.J. (2020). Innovations in camera trapping technology and approaches: The integration of citizen science and artificial intelligence. Animals, 10.10.3390/ani10010132 ↗
13. Mitterwallner, V., Peters, A., Edelhoff, H., Mathes, G., Nguyen, H., Peters, W., Heurich, M., and Steinbauer, M.J. (2023). Automated visitor and wildlife monitoring with camera traps and machine learning. Remote Sens. Ecol. Conserv.10.1002/rse2.367 ↗
14. An evaluation of platforms for processing camera-trap data using artificial intelligence10.1111/2041-210X.14044 ↗
15. Two-stream convolutional networks for action recognition in videos
16. Human action recognition and prediction: A survey10.1007/s11263-022-01594-9 ↗
17. Identification of animals and recognition of their actions in wildlife videos using deep learning techniques10.1016/j.ecoinf.2021.101215 ↗
18. Sakib, F., and Burghardt, T. (2020). Visual recognition of great ape behaviours in the wild. arXiv.
19. Ng, X.L., Ong, K.E., Zheng, Q., Ni, Y., Yeo, S.Y., and Liu, J. (2022, January 18–24). Animal kingdom: A large and diverse dataset for animal behavior understanding. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01844 ↗
20. Brookes, O., Mirmehdi, M., Kühl, H., and Burghardt, T. (2023). Triple-stream Deep Metric Learning of Great Ape Behavioural Actions. arXiv.10.5220/0011798400003417 ↗
21. Instance segmentation and tracking of animals in wildlife videos: SWIFT-segmentation with filtering of tracklets10.1016/j.ecoinf.2022.101794 ↗
22. Feichtenhofer, C., Fan, H., Malik, J., and He, K. (November, January 27). Slowfast networks for video recognition. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.
23. Fan, H., Xiong, B., Mangalam, K., Li, Y., Yan, Z., Malik, J., and Feichtenhofer, C. (2021, January 11–17). Multiscale vision transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.10.1109/ICCV48922.2021.00675 ↗
24. Zhang, H.B., Zhang, Y.X., Zhong, B., Lei, Q., Yang, L., Du, J.X., and Chen, D.S. (2019). A comprehensive survey of vision-based human action recognition methods. Sensors, 19.10.3390/s19051005 ↗
25. Bhoi, A. (2019). Spatio-temporal action recognition: A survey. arXiv.
26. Liu, X., Bai, S., and Bai, X. (2022, January 18–24). An empirical study of end-to-end temporal action detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01938 ↗
27. Tran, D., Bourdev, L., Fergus, R., Torresani, L., and Paluri, M. (2015, January 7–13). Learning spatiotemporal features with 3d convolutional networks. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.10.1109/ICCV.2015.510 ↗
28. Carreira, J., and Zisserman, A. (2017, January 21–26). Quo vadis, action recognition? a new model and the kinetics dataset. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.502 ↗
29. Tran, D., Wang, H., Torresani, L., Ray, J., LeCun, Y., and Paluri, M. (2018, January 18–23). A closer look at spatiotemporal convolutions for action recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00675 ↗
30. Hara, K., Kataoka, H., and Satoh, Y. (2018, January 18–23). Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet?. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00685 ↗
31. Wang, X., Girshick, R., Gupta, A., and He, K. (2018, January 18–23). Non-local neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00813 ↗
32. Zhou, B., Andonian, A., Oliva, A., and Torralba, A. (2018, January 8–14). Temporal relational reasoning in videos. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01246-5_49 ↗
33. Temporal segment networks for action recognition in videos10.1109/TPAMI.2018.2868668 ↗
34. Feichtenhofer, C. (2020, January 13–19). X3d: Expanding architectures for efficient video recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00028 ↗
35. Spatiotemporal residual networks for video action recognition
36. Sheth, I. (2021). Three-stream network for enriched Action Recognition. arXiv.
37. Liang, D., Fan, G., Lin, G., Chen, W., Pan, X., and Zhu, H. (2019, January 16–17). Three-stream convolutional neural network with multi-task and ensemble learning for 3d action recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Long Beach, CA, USA.10.1109/CVPRW.2019.00123 ↗
38. Jiang, B., Wang, M., Gan, W., Wu, W., and Yan, J. (November, January 27). Stm: Spatiotemporal and motion encoding for action recognition. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.
39. Lin, J., Gan, C., and Han, S. (November, January 27). Tsm: Temporal shift module for efficient video understanding. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.
40. Li, Y., Ji, B., Shi, X., Zhang, J., Kang, B., and Wang, L. (2020, January 13–19). Tea: Temporal excitation and aggregation for action recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00099 ↗
41. Kwon, H., Kim, M., Kwak, S., and Cho, M. (2020, January 23–28). Motionsqueeze: Neural motion feature learning for video understanding. Proceedings of the Computer Vision—ECCV 2020: 16th European Conference, Glasgow, UK. Proceedings, Part XVI 16.10.1007/978-3-030-58517-4_21 ↗
42. Ryoo, M.S., Piergiovanni, A., Arnab, A., Dehghani, M., and Angelova, A. (2021). Tokenlearner: What can 8 learned tokens do for images and videos?. arXiv.
43. Chen, M., Wei, F., Li, C., and Cai, D. (2022, January 18–24). Frame-wise action representations for long videos via sequence contrastive learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01343 ↗
44. Attention is all you need
45. Yan, S., Xiong, X., Arnab, A., Lu, Z., Zhang, M., Sun, C., and Schmid, C. (2022, January 18–24). Multiview transformers for video recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00333 ↗
46. Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lučić, M., and Schmid, C. (2021, January 11–17). Vivit: A video vision transformer. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.10.1109/ICCV48922.2021.00676 ↗
47. Sushmit, A.S., Ghosh, P., Istiak, M.A., Rashid, N., Akash, A.H., and Hasan, T. (2020). SegCodeNet: Color-Coded Segmentation Masks for Activity Detection from Wearable Cameras. arXiv.
48. He, K., Gkioxari, G., Dollár, P., and Girshick, R. (2017, January 22–29). Mask r-cnn. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.322 ↗
49. Zaghbani, S., and Bouhlel, M.S. (2021). Proceedings of the 12th International Conference on Soft Computing and Pattern Recognition (SoCPaR 2020), Springer.
50. Hacker, L., Bartels, F., and Martin, P.E. (2023). Fine-Grained Action Detection with RGB and Pose Information using Two Stream Convolutional Networks. arXiv.
51. Tang, J., Xia, J., Mu, X., Pang, B., and Lu, C. (2020, January 23–28). Asynchronous interaction aggregation for action detection. Proceedings of the Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK. Proceedings, Part XV 16.10.1007/978-3-030-58555-6_5 ↗
52. Biswas, S., and Gall, J. (December, January 30). Discovering Multi-Label Actor-Action Association in a Weakly Supervised Setting. Proceedings of the Asian Conference on Computer Vision, Online.
53. Chen, L., Tong, Z., Song, Y., Wu, G., and Wang, L. (2023). Efficient Video Action Detection with Token Dropout and Context Refinement. arXiv.10.1109/ICCV51070.2023.00953 ↗
54. Yuan, L., Zhou, Y., Chang, S., Huang, Z., Chen, Y., Nie, X., Wang, T., Feng, J., and Yan, S. (2020, January 12–16). Toward accurate person-level action recognition in videos of crowed scenes. Proceedings of the 28th ACM International Conference on Multimedia, Seattle, WA, USA.10.1145/3394171.3416301 ↗
55. Dutta, A., and Zisserman, A. (2019, January 21–25). The VIA Annotation Software for Images, Audio and Video. Proceedings of the 27th ACM International Conference on Multimedia, MM ’19, Nice, France.10.1145/3343031.3350535 ↗
56. Sofiiuk, K., Petrov, I.A., and Konushin, A. (2022, January 16–19). Reviving iterative training with mask guidance for interactive segmentation. Proceedings of the 2022 IEEE International Conference on Image Processing (ICIP), Bordeaux, France.10.1109/ICIP46576.2022.9897365 ↗
57. Zhang, Y., Kang, B., Hooi, B., Yan, S., and Feng, J. (2023). Deep long-tailed learning: A survey. IEEE Trans. Pattern Anal. Mach. Intell.10.1109/TPAMI.2023.3268118 ↗
58. Gu, C., Sun, C., Ross, D.A., Vondrick, C., Pantofaru, C., Li, Y., Vijayanarasimhan, S., Toderici, G., Ricco, S., and Sukthankar, R. (2018, January 18–23). Ava: A video dataset of spatio-temporally localized atomic visual actions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00633 ↗
59. Liu, Q., Xu, Z., Bertasius, G., and Niethammer, M. (2023, January 2–3). Simpleclick: Interactive image segmentation with simple vision transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France.10.1109/ICCVW60793.2023.00084 ↗
60. Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., and Lo, W.Y. (2023). Segment anything. arXiv.10.1109/ICCV51070.2023.00371 ↗
61. Analysing sequences of behavioural events10.1016/0022-5193(70)90107-4 ↗
62. Bels, V.L., Pallandre, J.P., Pelle, E., and Kirchhoff, F. (2022). Studies of the Behavioral Sequences: The Neuroethological Morphology Concept Crossing Ethology and Functional Morphology. Animals, 12.10.3390/ani12111336 ↗
63. Fully flexible analysis of behavioural sequences based on parametric survival models with frailties—A tutorial10.1111/eth.13225 ↗
64. Keatley, D. (2018). Pathways in Crime: An Introduction to Behaviour Sequence Analysis, Springer.10.1007/978-3-319-75226-6 ↗