Sequence-to-Sequence Video Prediction by Learning Hierarchical Representations
Source abstract
Abstract
Video prediction which maps a sequence of past video frames into realistic future video frames is a challenging task because it is difficult to generate realistic frames and model the coherent relationship between consecutive video frames. In this paper, we propose a hierarchical sequence-to-sequence prediction approach to address this challenge. We present an end-to-end trainable architecture in which the frame generator automatically encodes input frames into different levels of latent Convolutional Neural Network (CNN) features, and then recursively generates future frames conditioned on the estimated hierarchical CNN features and previous prediction. Our design is intended to automatically learn hierarchical representations of video and their temporal dynamics. Convolutional Long Short-Term Memory (ConvLSTM) is used in combination with skip connections so as to separately capture the sequential structures of multiple levels of hierarchy of features. We adopt Scheduled Sampling for training our recurrent network in order to facilitate convergence and to produce high-quality sequence predictions. We evaluate our method on the Bouncing Balls, Moving MNIST, and KTH human action dataset, and report favorable results as compared to existing methods.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Human Pose and Action Recognition · Advanced Vision and Imaging · Advanced Image Processing Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Science and Engineering, Korea University, Seoul 02841, Korea
References (56)
Show source-provided references
1. Li, C., Zhang, Z., Lee, W.S., and Lee, G.H. (2018, January 18–23). Convolutional Sequence to Sequence Model for Human Dynamics. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00548 ↗
2. Pedestrian tracking with shoe-mounted inertial sensors10.1109/MCG.2005.140 ↗
3. Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
4. Srivastava, N., Mansimov, E., and Salakhudinov, R. (2015, January 6–11). Unsupervised learning of video representations using LSTMs. Proceedings of the International Conference on Machine Learning, Lille, France.
5. Mathieu, M., Couprie, C., and LeCun, Y. (2016, January 2–4). Deep multi-scale video prediction beyond mean square error. Proceedings of the International Conference on Learning Representations, San Juan, PR, USA.
6. Action-conditional video prediction using deep networks in atari games
7. Kalchbrenner, N., van den Oord, A., Simonyan, K., Danihelka, I., Vinyals, O., Graves, A., and Kavukcuoglu, K. (2017, January 6–11). Video pixel networks. Proceedings of the 34th International Conference on Machine Learning, Sydney, Australia.
8. ImageNet Classification with Deep Convolutional Neural Networks10.1145/3065386 ↗
9. Simonyan, K., and Zisserman, A. (2015, January 7–9). Very deep convolutional networks for large-scale image recognition. Proceedings of the International Conference on Learning Representations, San Diego, CA, USA.
10. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
11. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015, January 7–12). Going deeper with convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298594 ↗
12. Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K.Q. (2017, January 21–26). Densely connected convolutional networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.243 ↗
13. Bidirectional recurrent neural networks10.1109/78.650093 ↗
14. Generative adversarial nets
15. Unsupervised learning of disentangled representations from video
16. Villegas, R., Yang, J., Hong, S., Lin, X., and Lee, H. (2017, January 24–26). Decomposing motion and content for natural video sequence prediction. Proceedings of the International Conference on Learning Representations, Toulon, France.
17. Villegas, R., Yang, J., Zou, Y., Sohn, S., Lin, X., and Lee, H. (2017, January 6–11). Learning to generate long-term future via hierarchical prediction. Proceedings of the 34th International Conference on Machine Learning, Sydney, Australia.
18. Walker, J., Marino, K., Gupta, A., and Hebert, M. (2017, January 22–29). The pose knows: Video forecasting by generating pose futures. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.361 ↗
19. Wichers, N., Villegas, R., Erhan, D., and Lee, H. (2018, January 10–15). Hierarchical Long-term Video Prediction without Supervision. Proceedings of the International Conference on Machine Learning, Stockholm, Sweden.
20. Ronneberger, O., Fischer, P., and Brox, T. (2015, January 5–9). U-net: Convolutional networks for biomedical image segmentation. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Munich, Germany.10.1007/978-3-319-24574-4_28 ↗
21. Newell, A., Yang, K., and Deng, J. (2016). Stacked hourglass networks for human pose estimation. European Conference on Computer Vision, Springer.10.1007/978-3-319-46484-8_29 ↗
22. Interaction networks for learning about objects, relations and physics
23. Chang, M.B., Ullman, T., Torralba, A., and Tenenbaum, J.B. (2017, January 24–26). A Compositional Object-Based Approach to Learning Physical Dynamics. Proceedings of the International Conference on Learning Representations, Toulon, France.
24. Fragkiadaki, K., Agrawal, P., Levine, S., and Malik, J. (2016, January 2–4). Learning visual predictive models of physics for playing billiards. Proceedings of the International Conference on Learning Representations, San Juan, PR, USA.
25. Schüldt, C., Laptev, I., and Caputo, B. (2004, January 18–23). Recognizing human actions: A local SVM approach. Proceedings of the 17th International Conference on Pattern Recognition ( ICPR), St. Petersburg, Russian.10.1109/ICPR.2004.1334462 ↗
26. Image quality assessment: From error visibility to structural similarity10.1109/TIP.2003.819861 ↗
27. Wang, Y., Jiang, L., Yang, M.H., Li, L.J., Long, M., and Fei-Fei, L. (2019, January 6–9). Eidetic 3D LSTM: A Model for Video Prediction and Beyond. Proceedings of the International Conference on Learning Representations, New Orleans, LA, USA.
28. Visual dynamics: Probabilistic future frame synthesis via cross convolutional networks
29. Babaeizadeh, M., Finn, C., Erhan, D., Campbell, R.H., and Levine, S. (May, January 30). Stochastic Variational Video Prediction. Proceedings of the International Conference on Learning Representations, Vancouver, BC, Canada.
30. PredRNN: Recurrent Neural Networks for Predictive Learning using Spatiotemporal LSTMs
31. Oliu, M., Selva, J., and Escalera, S. (2018). Folded Recurrent Neural Networks for Future Video Prediction. ECCV, Springer.10.1007/978-3-030-01264-9_44 ↗
32. Lakhal, M.I., Lanz, O., and Cavallaro, A. (November, January 27). View-LSTM: Novel-View Video Synthesis Through View Decomposition. Proceedings of the 2019 ICCV, Seoul, Korea.10.1109/ICCV.2019.00767 ↗
33. Gao, H., Xu, H., Cai, Q.Z., Wang, R., Yu, F., and Darrell, T. (November, January 27). Disentangling Propagation and Generation for Video Prediction. Proceedings of the 2019 ICCV, Seoul, Korea.10.1109/ICCV.2019.00910 ↗
34. Sequence to Sequence Learning with Neural Networks
35. Learning Precise Timing with LSTM Recurrent Networks
36. Unsupervised learning for physical interaction through video prediction
37. Lotter, W., Kreiman, G., and Cox, D. (2017, January 24–26). Deep predictive coding networks for video prediction and unsupervised learning. Proceedings of the International Conference on Learning Representations, Toulon, France.
38. Attend, infer, repeat: Fast scene understanding with generative models
39. Sequential attend, infer, repeat: Generative modelling of moving objects
40. Learning to decompose and disentangle representations for video prediction
41. Wang, Y., Gao, Z., Long, M., Wang, J., and Yu, P.S. (2018, January 10–15). PredRNN++: Towards A Resolution of the Deep-in-Time Dilemma in Spatiotemporal Predictive Learning. Proceedings of the International Conference on Machine Learning, Stockholm, Sweden.
42. Pan, J., Wang, C., Jia, X., Shao, J., Sheng, L., Yan, J., and Wang, X. (2019, January 16–20). Video Generation From Single Semantic Label Map. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00385 ↗
43. Wang, T.H., Cheng, Y.C., Lin, C.H., Chen, H.T., and Sun, M. (November, January 27). Point-to-Point Video Generation. Proceedings of the IEEE International Conference on Computer Vision, Seoul, South Korea.
44. Generating videos with scene dynamics
45. Tulyakov, S., Liu, M.Y., Yang, X., and Kautz, J. (2018, January 18–23). MoCoGAN: Decomposing Motion and Content for Video Generation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00165 ↗
46. Denton, E., and Fergus, R. (2018, January 10–15). Stochastic video generation with a learned prior. Proceedings of the International Conference on Machine Learning, Stockholm, Sweden.
47. Ohnishi, K., Yamamoto, S., Ushiku, Y., and Harada, T. (2018, January 2–7). Hierarchical video generation from orthogonal information: Optical flow and texture. Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.10.1609/aaai.v32i1.11881 ↗
48. Liang, X., Lee, L., Dai, W., and Xing, E.P. (2017, January 22–29). Dual Motion GAN for Future-Flow Embedded Video Prediction. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.194 ↗
49. Lee, A.X., Zhang, R., Ebert, F., Abbeel, P., Finn, C., and Levine, S. (May, January 30). Stochastic Adversarial Video Prediction. Proceedings of the International Conference on Learning Representations, Vancouver, BC, Canada.
50. Ioffe, S., and Szegedy, C. (2015, January 6–11). Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. Proceedings of the International Conference on Machine Learning, Lille, France.
51. Ba, J., Kiros, J.R., and Hinton, G.E. (2016). Layer Normalization. arXiv.
52. Isola, P., Zhu, J.Y., Zhou, T., and Efros, A.A. (2017, January 21–26). Image-to-Image Translation with Conditional Adversarial Networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.632 ↗
53. Scheduled sampling for sequence prediction with recurrent neural networks
54. Image analysis using mathematical morphology10.1109/TPAMI.1987.4767941 ↗
55. Srivastava, N., Mansimov, E., and Salakhudinov, R. (2019, May 06). Moving MNIST Download Link. Available online: http://www.cs.toronto.edu/~nitish/unsupervised_video/.
56. Kingma, D.P., and Ba, J. (2015, January 7–9). Adam: A method for stochastic optimization. Proceedings of the International Conference on Learning Representations, San Diego, CA, USA.