End to End Alignment Learning of Instructional Videos with Spatiotemporal Hybrid Encoding and Decoding Space Reduction
Source abstract
Abstract
We solve the problem of how to densely align actions in videos at frame level, with only the order of occurring actions available, in order to save the time-consuming efforts to accurately annotate the temporal boundaries of each action. We propose three task-specific innovations under this scenario: (1) To encode fine-grained spatiotemporal local features and long-range temporal patterns simultaneously, we test three popular backbones and compare their accuracy and training times: (i) a recurrent LSTM; (ii) a fully convolutional model; and (iii) the recently proposed Transformer model. (2) To address the absence of ground truth frame-by-frame labels during training, we apply connectionist temporal classification (CTC) on top of the temporal encoder to recursively collect all theoretically valid alignments, and further weight these alignments with frame-wise visual similarities, in order to avoid a significant number of degenerated paths and improve both recognition accuracy and computation efficiency. (3) To quantitatively assess the quality of the learned alignment, we apply a comprehensive set of frame-level, segment-level, and video-level evaluation measurements. Extensive evaluations verify the effectiveness of our proposal, with performance comparable to that of fully supervised approaches across four benchmarks of different difficulty and data scale.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Human Pose and Action Recognition · Multimodal Machine Learning Applications · Domain Adaptation and Few-Shot Learning
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer Science and Technology, University of Science and Technology of China, Hefei 230027, China
Funding
Fundamental Research Funds for the Central Universities · National Natural Science Foundation of China
References (51)
Show source-provided references
1. Richard, A., Kuehne, H., and Gall, J. (2017, January 21–26). Weakly supervised action learning with rnn based fine-to-coarse modeling. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.140 ↗
2. Ding, L., and Xu, C. (2018, January 18–22). Weakly-supervised action segmentation with iterative soft boundary assignment. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.
3. Li, J., Lei, P., and Todorovic, S. (November, January 27). Weakly supervised energy-based learning for action segmentation. Proceedings of the IEEE International Conference on Computer Vision, Seoul, Korea.
4. Learning from narrated instruction videos10.1109/TPAMI.2017.2749223 ↗
5. Tang, Y., Ding, D., Rao, Y., Zheng, Y., Zhang, D., Zhao, L., Lu, J., and Zhou, J. (2019, January 16–20). COIN: A large-scale dataset for comprehensive instructional video analysis. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00130 ↗
6. Fried, D., Alayrac, J.B., Blunsom, P., Dyer, C., Clark, S., and Nematzadeh, A. (2020). Learning to Segment Actions from Observation and Narration. arXiv.10.18653/v1/2020.acl-main.231 ↗
7. Graves, A. (2012). Connectionist Temporal Classification. Supervised Sequence Labelling with Recurrent Neural Networks, Springer.10.1007/978-3-642-24797-2 ↗
8. Pirsiavash, H., and Ramanan, D. (2014, January 23–28). Parsing videos of actions with segmental grammars. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.85 ↗
9. Richard, A., and Gall, J. (July, January 26). Temporal action detection using a statistical language model. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.
10. Leibe, B., Matas, J., Sebe, N., and Welling, M. (2016, January 11–14). Segmental Spatiotemporal CNNs for Fine-Grained Action Segmentation. Proceedings of the Computer Vision—ECCV 2016, Amsterdam, The Netherlands. Lecture Notes in Computer Science.10.1007/978-3-319-46493-0 ↗
11. Singh, B., Marks, T.K., Jones, M., Tuzel, O., and Shao, M. (July, January 26). A multi-stream bi-directional recurrent neural network for fine-grained action detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.
12. Lea, C., Flynn, M.D., Vidal, R., Reiter, A., and Hager, G.D. (2017, January 21–26). Temporal Convolutional Networks for Action Segmentation and Detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.113 ↗
13. Lei, P., and Todorovic, S. (2018, January 18–22). Temporal deformable residual networks for action segmentation in videos. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00705 ↗
14. Zhang, Y., Tang, S., Muandet, K., Jarvers, C., and Neumann, H. (2019, January 16–20). Local temporal bilinear pooling for fine-grained action parsing. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.01228 ↗
15. Ishikawa, Y., Kasai, S., Aoki, Y., and Kataoka, H. (2021, January 5–9). Alleviating Over-Segmentation Errors by Detecting Action Boundaries. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Waikola, HI, USA.10.1109/WACV48630.2021.00237 ↗
16. Ghoddoosian, R., Sayed, S., and Athitsos, V. (2021, January 5–9). Action Duration Prediction for Segment-Level Alignment of Weakly-Labeled Videos. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Waikola, HI, USA.10.1109/WACV48630.2021.00210 ↗
17. Recognizing Fine-Grained and Composite Activities Using Hand-Centric Features and Script Data10.1007/s11263-015-0851-8 ↗
18. Stein, S., and McKenna, S.J. (2013, January 8–12). Combining Embedded Accelerometers with Computer Vision for Recognizing Food Preparation Activities. Proceedings of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp 13), Zurich, Switzerland.10.1145/2493432.2493482 ↗
19. Fathi, A., Ren, X., and Rehg, J.M. (2011, January 20–25). Learning to recognize objects in egocentric activities. Proceedings of the CVPR 2011, Colorado Springs, CO, USA.10.1109/CVPR.2011.5995444 ↗
20. Gao, Y., Vedula, S.S., Reiley, C.E., Ahmidi, N., Varadarajan, B., Lin, H.C., Tao, L., Zappella, L., Béjar, B., and Yuh, D.D. (2014, January 14–18). Jhu-isi gesture and skill assessment working set (jigsaws): A surgical activity dataset for human motion modeling. Proceedings of the Modeling and Monitoring of Computer Assisted Interventions (M2CAI), MICCAI Workshop, Boston, MA, USA.
21. A dataset and benchmarks for segmentation and recognition of gestures in robotic surgery10.1109/TBME.2016.2647680 ↗
22. Kuehne, H., Arslan, A.B., and Serre, T. (2014, January 23–28). The Language of Actions: Recovering the Syntax and Semantics of Goal-Directed Human Activities. Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), Columbus, OH, USA.10.1109/CVPR.2014.105 ↗
23. Bojanowski, P., Lajugie, R., Grave, E., Bach, F., Laptev, I., Ponce, J., and Schmid, C. (2015, January 11–18). Weakly-supervised alignment of video with text. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.10.1109/ICCV.2015.507 ↗
24. Alayrac, J.B., Bojanowski, P., Agrawal, N., Sivic, J., Laptev, I., and Lacoste-Julien, S. (July, January 26). Unsupervised learning from narrated instruction videos. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.
25. Zhou, L., Xu, C., and Corso, J.J. (2018, January 2–7). Towards automatic learning of procedures from web instructional videos. Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.10.1609/aaai.v32i1.12342 ↗
26. Kukleva, A., Kuehne, H., Sener, F., and Gall, J. (2019, January 16–20). Unsupervised learning of action classes with continuous temporal embedding. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.01234 ↗
27. Oneata, D., Verbeek, J., and Schmid, C. (2013, January 2–8). Action and event recognition with fisher vectors on a compact feature set. Proceedings of the IEEE International Conference on Computer Vision, Sydney, NSW, Australia.10.1109/ICCV.2013.228 ↗
28. Wang, H., and Schmid, C. (2013, January 2–8). Action recognition with improved trajectories. Proceedings of the IEEE International Conference on Computer Vision, Sydney, NSW, Australia.10.1109/ICCV.2013.441 ↗
29. Simonyan, K., and Zisserman, A. (2014, January 8–13). Two-stream convolutional networks for action recognition in videos. Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada.
30. Tran, D., Bourdev, L., Fergus, R., Torresani, L., and Paluri, M. (2015, January 11–17). Learning spatiotemporal features with 3d convolutional networks. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.10.1109/ICCV.2015.510 ↗
31. Carreira, J., and Zisserman, A. (2017, January 21–26). Quo vadis, action recognition?. A new model and the kinetics dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.502 ↗
32. Piergiovanni, A., and Ryoo, M.S. (2019, January 16–20). Representation flow for action recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.01018 ↗
33. Malmaud, J., Huang, J., Rathod, V., Johnston, N., Rabinovich, A., and Murphy, K. (2015). What’s cookin’? interpreting cooking videos using text, speech and vision. arXiv.10.3115/v1/N15-1015 ↗
34. Simonyan, K., and Zisserman, A. (June, January June). Very deep convolutional networks for large-scale image recognition. Proceedings of the Computer Vision and Pattern Recognition, Columbus, OH, USA.
35. He, K., Zhang, X., Ren, S., and Sun, J. (July, January 26). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.
36. Ioffe, S., and Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv.
37. Feichtenhofer, C., Fan, H., Malik, J., and He, K. (November, January 27). Slowfast networks for video recognition. Proceedings of the IEEE International Conference on Computer Vision, Seoul, Korea.
38. Kuehne, H., Gall, J., and Serre, T. (2016, January 7–10). An end-to-end generative framework for video segmentation and recognition. Proceedings of the 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), Lake Placid, NY, USA.10.1109/WACV.2016.7477701 ↗
39. Graves, A., Fernández, S., and Schmidhuber, J. (2005, January 11–15). Bidirectional LSTM networks for improved phoneme classification and recognition. Proceedings of the International Conference on Artificial Neural Networks, Warsaw, Poland. Lecture Notes in Computer Science.10.1007/11550907_126 ↗
40. Chollet, F. (2017, January 21–26). Xception: Deep Learning with Depthwise Separable Convolutions. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.195 ↗
41. Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (2017). Attention is All you Need. Advances in Neural Information Processing Systems, Curran Associates, Inc.
42. Graves, A., Fernández, S., Gomez, F., and Schmidhuber, J. (2006, January 25–29). Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks. Proceedings of the 23rd International Conference on Machine Learning, Pittsburgh, PA, USA.10.1145/1143844.1143891 ↗
43. Li, A., Thotakuri, M., Ross, D.A., Carreira, J., Vostrikov, A., and Zisserman, A. (2020). The AVA-Kinetics Localized Human Actions Video Dataset. arXiv.
44. Bengio, Y., and LeCun, Y. (2015, January 7–9). Adam: A Method for Stochastic Optimization. Proceedings of the 23rd International Conference on Learning Representations (ICLR 2015), San Diego, CA, USA. Conference Track Proceedings.
45. Dropout: A Simple Way to Prevent Neural Networks from Overfitting
46. Gehring, J., Auli, M., Grangier, D., and Dauphin, Y. A Convolutional Encoder Model for Neural Machine Translation. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
47. Kaiser, L., Gomez, A.N., and Chollet, F. (May, January 30). Depthwise Separable Convolutions for Neural Machine Translation. Proceedings of the International Conference on Learning Representations, Vancouver, BC, Canada.
48. Zhang, Y., Pezeshki, M., Brakel, P., Zhang, S., Laurent, C., Bengio, Y., and Courville, A. (2016, January 8–12). Towards End-to-End Speech Recognition with Deep Convolutional Neural Networks. Proceedings of the Interspeech 2016, San Francisco, CA, USA.10.21437/Interspeech.2016-1446 ↗
49. Wang, Y., Deng, X., Pu, S., and Huang, Z. (2017). Residual Convolutional CTC Networks for Automatic Speech Recognition. arXiv.
50. Precup, D., and Teh, Y.W. (2017, January 6–11). Convolutional Sequence to Sequence Learning. Proceedings of the 34th International Conference on Machine Learning, Sydney, NSW, Australia.
51. Bai, S., Kolter, J.Z., and Koltun, V. (2018). An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling. arXiv.