Reliability-Based View Synthesis for Free Viewpoint Video
Source abstract
Abstract
View synthesis is a crucial technique for free viewpoint video and multi-view video coding because of its capability to render an unlimited number of virtual viewpoints from adjacent captured texture images and corresponding depth maps. The accuracy of depth maps is very important to the rendering quality, since depth image–based rendering (DIBR) is the most widely used technology among synthesis algorithms. There are some issues due to the fact that stereo depth estimation is error-prone. In addition, filling occlusions is another challenge in producing desirable synthesized images. In this paper, we propose a reliability-based view synthesis framework. A depth refinement method is used to check the reliability of depth values and refine some of the unreliable pixels, and an adaptive background modeling algorithm is utilized to construct a background image aiming to fill the remaining empty regions after a proposed weighted blending process. Finally, the proposed approach is implemented and tested on test video sequences, and experimental results indicate objective and subjective improvements compared to previous view synthesis methods.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Vision and Imaging · Video Coding and Compression Technologies · Advanced Image Processing Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Faculty of Electronics and Information Engineering, Shenzhen Graduate School, Harbin Institute of Technology, Shenzhen 518055, China
References (27)
Show source-provided references
1. Overview of the multi-view and 3D extensions of high efficiency video coding10.1109/TCSVT.2015.2477935 ↗
2. Free-viewpoint TV10.1109/MSP.2010.939077 ↗
3. Fehn, C. (2004, January 19–22). Depth-image-based rendering (DIBR), compression, and transmission for a new approach on 3D-Tv. Proceedings of the Society of Photo-Optical Instrumentation Engineers (SPIE), San Jose, CA, USA.10.1117/12.524762 ↗
4. Farid, M.S., Lucenteforte, M., and Grangetto, M. (October, January 30). Depth image based rendering with inverse mapping. Proceedings of the IEEE 15th International Conference on Workshop Multimedia Signal Processing, Pula, Italy.
5. Virtual View Synthesis for Free Viewpoint Video and Multiview Video Compression using Gaussian Mixture Modeling10.1109/TIP.2017.2772858 ↗
6. Zhang, L., Tam, W.J., and Wang, D.M. (2004, January 24–27). Stereoscopic image generation based on depth images. Proceedings of the International Conference on Image Processing (ICIP), Singapore, Singapore.
7. Stereoscopic image generation based on depth images for 3D TV10.1109/TBC.2005.846190 ↗
8. Cheng, C.M., Lin, S.J., Lai, S.H., and Yang, J.C. (2008, January 8–11). Improved novel view synthesis from depth image with large baseline. Proceedings of the 19th International Conference on Pattern Recognition (ICPR), Tampa, FL, USA.
9. Non-geometric distortion smoothing approach for depth map preprocessing10.1109/TMM.2010.2100372 ↗
10. Region filling and object removal by exemplar-based image inpainting10.1109/TIP.2004.833105 ↗
11. A novel depth-based virtual view synthesis method for free viewpoint video10.1109/TBC.2013.2281658 ↗
12. Boundary artifact reduction in view synthesis of 3D video: From perspective of texture depth alignment10.1109/TBC.2011.2120730 ↗
13. Bertalmio, M., Bertozzi, A.L., and Sapiro, G. (2001, January 8–14). Navier-stokes, fluid dynamics, and image and video inpainting. Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), Kauai, HI, USA.
14. Strong-continuation, contrast-invariant inpainting with a third-order optimal PDE10.1109/TIP.2006.877067 ↗
15. Rahaman, D.M.M., and Paul, M. (2016, January 11–15). Hole-filling for single-view plus depth based rendering with temporal texture synthesis. Proceedings of the IEEE International Conference on Workshop Multimedia Expo Workshops (ICMEW), Seattle, WA, USA.10.1109/ICMEW.2016.7574740 ↗
16. Rahaman, D.M.M., and Paul, M. (2015, January 23–24). Free view-point video synthesis using Gaussian mixture modeling. Proceedings of the IEEE Conference on Image and Vision Computing, Auckland, New Zealand.
17. Li, S., Zhu, C., and Sun, M.T. (2018). Hole Filling with Multiple Reference Views in DIBR View Synthesis. IEEE Trans. Multimed.10.1109/TMM.2018.2791810 ↗
18. Schmeing, M., and Jiang, X. (2010, January 7–9). Depth image based rendering: A faithful approach for the disocclusion problem. Proceedings of the 3DTV-Conference: The True Vision—Capture, Transmission and Display of 3D Video, Tampere, Finland.10.1109/3DTV.2010.5506596 ↗
19. Chen, K.Y., Tsung, P.K., Lin, P.C., Yang, H.J., and Chen, L.G. (2010, January 7–9). Hybrid motion/depth-oriented inpainting for virtual view synthesis in multiview applications. Proceedings of the 3DTV-Conference: The True Vision—Capture, Transmission and Display of 3D Video, Tampere, Finland.10.1109/3DTV.2010.5506309 ↗
20. Köppel, M., Ndjiki-Nya, P., Doshkov, D., Lakshman, H., Merkle, P., Müller, K., and Wiegand, T. (2010, January 26–29). Temporally consistent handling of disocclusions with texture synthesis for depth-image-based rendering. Proceedings of the IEEE International Conference on Image Processing, Hong Kong, China.10.1109/ICIP.2010.5652138 ↗
21. Depth image-based rendering with advanced texture synthesis for 3-D video10.1109/TMM.2011.2128862 ↗
22. Bosc, E., Köppel, M., Pépion, R., Pressigout, M., Morin, L., Ndjiki-Nya, P., and Le Callet, P. (2011, January 11–14). Can 3D synthesized views be reliably assessed through usual subjective and objective evaluation protocols?. Proceedings of the 18th IEEE International Conference on Image Processing, Brussels, Belgium.10.1109/ICIP.2011.6116196 ↗
23. Depth map driven hole filling algorithm exploiting temporal correlation information10.1109/TBC.2014.2321671 ↗
24. Luo, G., Zhu, Y., Li, Z., and Zhang, L. (2016, January 27–10). A hole filling approach based on background reconstruction for view synthesis in 3D video. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.197 ↗
25. Stauffer, C., and Grimson, W.E.L. (1999, January 23–25). Adaptive background mixture models for real-time tracking. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Fort Collins, CO, USA.
26. Deng, Z.M., and Wang, M.J. (2017, January 24–25). Hybrid Temporal Correlation Based on Gaussian Mixture Model Framework for View Synthesis. Proceedings of the 18th International Conference on Computers and Communication Networks, Boston, MA, USA.
27. Tanimoto, M., Fujii, T., and Suzuki, K. (2018, May 20). Available online: http://wg11.sc29.org/svn/repos/MPEG-4/test/trunk/3D/view_synthesis/VSRS.