Video Frame Prediction by Joint Optimization of Direct Frame Synthesis and Optical-Flow Estimation
Source abstract
Abstract
Video prediction is the problem of generating future frames by exploiting the spatiotemporal correlation from the past frame sequence. It is one of the crucial issues in computer vision and has many real-world applications, mainly focused on predicting future scenarios to avoid undesirable outcomes. However, modeling future image content and object is challenging due to the dynamic evolution and complexity of the scene, such as occlusions, camera movements, delay and illumination. Direct frame synthesis or optical-flow estimation are common approaches used by researchers. However, researchers mainly focused on video prediction using one of the approaches. Both methods have limitations, such as direct frame synthesis, usually face blurry prediction due to complex pixel distributions in the scene, and optical-flow estimation, usually produce artifacts due to large object displacements or obstructions in the clip. In this paper, we constructed a deep neural network Frame Prediction Network (FPNet-OF) with multiple-branch inputs (optical flow and original frame) to predict the future video frame by adaptively fusing the future object-motion with the future frame generator. The key idea is to jointly optimize direct RGB frame synthesis and dense optical flow estimation to generate a superior video prediction network. Using various real-world datasets, we experimentally verify that our proposed framework can produce high-level video frame compared to other state-of-the-art framework.
Source: openalex_inverted_index · Quality: baseline
Research context
OpenAlex topics: Advanced Vision and Imaging · Advanced Image Processing Techniques · Image Enhancement Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
References (66)
Show source-provided references
1. Future frame prediction for anomaly detection-A new baseline
2. Looking ahead: Anticipating pedestrians crossing with future frames prediction
3. Multi-modal trajectory prediction of surrounding vehicles with maneuver based LSTMs
4. A survey of scene understanding by event reasoning in autonomous driving10.1007/s11633-018-1126-y ↗
5. Learning-based approach for online lane change intention prediction
6. Optimizing steering angle predictive convolutional neural network for autonomous car10.32604/cmc.2022.022726 ↗
7. Deep predictive video compression using mode-selective uni-and bi-directional predictions based on multi-frame hypothesis10.1109/ACCESS.2020.3046040 ↗
8. Deep predictive coding networks for video prediction and unsupervised learning
9. Robot operation monitoring for collision avoidance by image sequence analysis10.1016/0031-3203(92)90038-K ↗
10. Collision avoidance on unmanned aerial vehicles using neural network pipelines and flow clustering techniques10.3390/rs13132643 ↗
11. HARTIV: Human activity recognition using temporal information in videos10.32604/cmc.2022.020655 ↗
12. Visual forecasting by imitating dynamics in natural sequences
13. AI based traffic flow prediction model for connected and autonomous electric vehicles10.32604/cmc.2022.020197 ↗
14. City-wide traffic congestion prediction based on CNN, LSTM and transpose CNN10.1109/ACCESS.2020.2991462 ↗
15. Large-scale road network congestion pattern analysis and prediction using deep convolutional autoencoder10.3390/su13095108 ↗
16. Learning and recognizing human dynamics in video sequences
17. Coupled hidden markov models for complex action recognition
18. Efficient nonlinear markov models for human motion
19. Anomaly detection in crowded scenes
20. Rate-coded restricted boltzmann machines for face recognition
21. Bearing fault detection and diagnosis using case western reserve university dataset with deep learning approaches: A review10.1109/ACCESS.2020.2990528 ↗
22. A review on deep learning-based approaches for automatic sonar target recognition10.3390/electronics9111972 ↗
23. Deep learning-based content caching in the fog access points10.3390/electronics10040512 ↗
24. Optimal cache resource based on deep neural network for fog radio access networks
25. An uncertain future: Forecasting from static images using variational autoencoders
26. Video frame synthesis using deep voxel flow
27. DRAW: A recurrent neural network for image generation
28. Unsupervised learning of video representations using LSTMs
29. Learning latent global network for skeleton-based action prediction10.1109/TIP.2019.2937757 ↗
30. Early action prediction with generative adversarial networks10.1109/ACCESS.2019.2904857 ↗
31. Predicting future frames using retrospective cycle GAN
32. SegNet: A deep convolutional encoder-decoder architecture for image segmentation
33. Recent progress in semantic image segmentation10.1007/s10462-018-9641-3 ↗
34. Reduced-gate convolutional LSTM architecture for next-frame video prediction using predictive coding
35. Efficient and information-preserving future frame prediction and beyond
36. A Log-likelihood regularized KL divergence for video prediction with a 3D convolutional variational recurrent network
37. Future frame prediction using convolutional VRNN for anomaly detection
38. Deep multi-scale video prediction beyond mean square error
39. Action-conditional video prediction using deep networks in atari games
40. Video (language) modeling: A baseline for generative models of natural videos
41. Generating videos with scene dynamics
42. Decomposing motion and content for natural video sequence prediction
43. Visual dynamics: Probabilistic future frame synthesis via cross convolutional networks
44. Learning spatiotemporal features with 3D convolutional networks
45. Unsupervised learning for physical interaction through video prediction
46. Unsupervised learning of visual structure using predictive generative networks
47. ContextVP: Fully context-aware video prediction
48. Sift flow: Dense correspondence across scenes and its applications
49. EpicFlow: Edge-preserving interpolation of correspondences for optical flow
50. Mahajan · SIGGRAPH09: Special Intrest Group on Computer Graphics and Interactive Techniques Conf. · 2009 · 1
51. Unsupervised learning of long-term motion dynamics for videos
52. Determining optical flow10.1016/0004-3702(81)90024-2 ↗
53. FlowNet: Learning optical flow with convolutional networks
54. Dual motion GAN for futureflow embedded video prediction
55. Video frame prediction by deep multi-branch mask network10.1109/TCSVT.2020.2984783 ↗
56. Next-flow: Hybrid multi-tasking with next-frame prediction to boost optical-flow estimation in the wild
57. Deep residual learning for image recognition
58. An improved ResNet based on the adjustable shortcut connections10.1109/ACCESS.2018.2814605 ↗
59. Pedestrian detection: An evaluation of the state of the art10.1109/TPAMI.2011.155 ↗
60. UCF101: A dataset of 101 human action classes from videos in the wild
61. Abnormal event detection in videos using generative adversarial nets
62. A revisit of sparse coding based anomaly detection in stacked RNN framework
63. Performance of optical flow techniques10.1007/BF01420984 ↗
64. Optical flow using spatiotemporal filters10.1007/BF00133568 ↗
65. Computation of optical flow from the motion of edge features in image sequences10.1016/0262-8856(84)90001-5 ↗
66. Two-frame motion estimation based on polynomial expansion10.1007/3-540-45103-X_50 ↗