A Novel Real-Time Virtual 3D Object Composition Method for 360° Video
Source abstract
Abstract
As highly immersive virtual reality (VR) content, 360° video allows users to observe all viewpoints within the desired direction from the position where the video is recorded. In 360° video content, virtual objects are inserted into recorded real scenes to provide a higher sense of immersion. These techniques are called 3D composition. For a realistic 3D composition in a 360° video, it is important to obtain the internal (focal length) and external (position and rotation) parameters from a 360° camera. Traditional methods estimate the trajectory of a camera by extracting the feature point from the recorded video. However, incorrect results may occur owing to stitching errors from a 360° camera attached to several high-resolution cameras for the stitching process, and a large amount of time is spent on feature tracking owing to the high-resolution of the video. We propose a new method for pre-visualization and 3D composition that overcomes the limitations of existing methods. This system achieves real-time position tracking of the attached camera using a ZED camera and a stereo-vision sensor, and real-time stabilization using a Kalman filter. The proposed system shows high time efficiency and accurate 3D composition.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Vision and Imaging · Robotics and Sensor-Based Localization · Image and Video Stabilization
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Plasma Bio-Display, Kwangwoon University, Seoul 01897, Korea
Spatial Computing Convergence Center, Kwangwoon University, Seoul 01897, Korea
Graduate School of Smart Convergence, Kwangwoon University, Seoul 01897, Korea
Ingenium College, Kwangwoon University, Seoul 01897, Korea
References (32)
Show source-provided references
1. The virtual reality head-mounted display Oculus Rift induces motion sickness and is sexist in its effects10.1007/s00221-016-4846-7 ↗
2. Jung, S., and Whangbo, T. (2017, January 8–10). Study on inspecting VR motion sickness inducing factors. Proceedings of the 2017 4th International Conference on Computer Applications and Information Processing Technology (CAIPT), Kuta, Bali.10.1109/CAIPT.2017.8320660 ↗
3. Erica, H. (2010). The Art and Technique of Matchmoving: Solutions for the VFX Artist, Elsevier. [1st ed.].
4. Dobbert, T. (2005). The Matchmoving Process. Matchmoving: The Invisible Art of Camera Tracking, Sybex. [1st ed.].
5. Visual modeling with a hand-held camera10.1023/B:VISI.0000025798.50602.3a ↗
6. Davison, A.J. (2003, January 13–16). Real-Time Simultaneous Localisation and Mapping with a Single Camera. Proceedings of the Ninth IEEE International Conference on Computer Vision, Nice, France.10.1109/ICCV.2003.1238654 ↗
7. Klein, G., and Murray, D. (2007, January 13–16). Parallel Tracking and Mapping for Small AR Workspaces. Proceedings of the 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality, Nara, Japan.10.1109/ISMAR.2007.4538852 ↗
8. Davison, A.J. (2013, January 23–28). SLAM++: Simultaneous Localisation In addition, Mapping at the Level of Objects. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Portland, OR, USA.
9. Bao, S.Y., and Savarese, S. (2011, January 20–25). Semantic Structure from Motion. Proceedings of the IEEE CVPR 2011, Colorado Springs, CO, USA.10.1109/CVPR.2011.5995462 ↗
10. The effect of patterns on image-based modelling of texture-less objects10.24425/mms.2018.124883 ↗
11. Lee, J., Hafeez, J., Kim, K., Lee, S., and Kwon, S. (2019). A novel real-time match-moving method with HoloLens. Appl. Sci., 9.10.3390/app9142889 ↗
12. Huang, K.-C., Chien, P.-Y., Chien, C.-A., Chang, H.-C., and Guo, J.-I. (2014). A 360-degree panoramic video system design. Proceedings of the Technical Papers of 2014 International Symposium on VLSI Design, Automation and Test, Institute of Electrical and Electronics Engineers (IEEE).10.1109/VLSI-DAT.2014.6834863 ↗
13. Interactive Augmented Omnidirectional Video with Realistic Lighting
14. Huang, J., Chen, Z., Ceylan, D., and Jin, H. (2017, January 18–22). 6-DOF VR videos with a single 360-camera. Proceedings of the 2017 IEEE Virtual Reality (VR), Los Angeles, CA, USA.10.1109/VR.2017.7892229 ↗
15. Scale invariant feature transform on the sphere: Theory and applications10.1007/s11263-011-0505-4 ↗
16. SPHORB: A fast and robust binary feature on the sphere10.1007/s11263-014-0787-4 ↗
17. Keyframe-based visual–inertial odometry using nonlinear optimization10.1177/0278364914554813 ↗
18. Robust stereo visual inertial odometry for fast autonomous flight10.1109/LRA.2018.2793349 ↗
19. Vit, A., and Shani, G. (2018). Comparing RGB-D sensors for close range outdoor agricultural phenotyping. Sensors, 18.10.20944/preprints201810.0664.v1 ↗
20. (2020, October 10). ZED. Available online: https://www.stereolabs.com/zed/.
21. Ibragimov, I.Z., and Afanasyev, I.M. (2017, January 25–26). Comparison of ROS-based Visual SLAM Methods in Homogeneous Indoor Environment. Proceedings of the 2017 14th Workshop on Positioning, Navigation and Communications (WPNC), Bremen, Germany.10.1109/WPNC.2017.8250081 ↗
22. Alapetite, A., Wang, Z., Hansen, J.P., Zajączkowski, M., and Patalan, M. (2020). Comparison of three off-the-shelf visual odometry systems. Robotics, 9.10.3390/robotics9030056 ↗
23. Huang, F., and Sugimoto, A. (2016). Real-Time Image Based Lighting for 360-Degree Panoramic Video. Image and Video Technology—PSIVT 2015 Workshops, Springer. PSIVT 2015, Lecture Notes in Computer Science.10.1007/978-3-319-30285-0 ↗
24. MR360: Mixed reality rendering for 360° panoramic videos10.1109/TVCG.2017.2657178 ↗
25. Iorns, T., and Rhee, T.H. (2015, January 23–27). Real-Time Image Based Lighting for 360-Degree Panoramic Video. Proceedings of the PSIVT Workshops, Auckland, New Zealand.10.1007/978-3-319-30285-0_12 ↗
26. A study on motion graphics virtual camera using real-time position tracking in post-production
27. (2020, October 10). Z1. Available online: https://theta360.com/en/about/theta/z1.html.
28. A new approach to linear filtering and prediction problem10.1115/1.3662552 ↗
29. Welch, G., and Bishop, G. (2001). An Introduction to the Kalman Filter, University North Carolina. Lecture.
30. Prabhu, U., Seshadri, K., and Savvides, M. (2010, January 5–11). Automatic Facial Landmark Tracking in Video Sequences using Kalman Filter Assisted Active Shape Models. Proceedings of the European Conference on Computer Vision, Heraklion, Greece.
31. Kalman filter for robot vision: A survey10.1109/TIE.2011.2162714 ↗
32. Visual tracking: An experimental survey10.1109/TPAMI.2013.230 ↗