journal article·2019-05-03Open access

Artifact Handling Based on Depth Image for View Synthesis

Xiaodong Chen, Haitao Liang, Huaiyuan Xu, Siyu Ren, Huaiyu Cai, Yi Wang

Applied Sciences· Volume 9· Issue 9

Abstract

The depth image based rendering (DIBR) is a popular technology for 3D video and free viewpoint video (FVV) synthesis, by which numerous virtual views can be generated from a single reference view and its depth image. However, some artifacts are produced in the DIBR process and reduce the visual quality of virtual view. Due to the diversity of artifacts, effectively handling them becomes a challenging task. In this paper, an artifact handling method based on depth image is proposed. The reference image and its depth image are extended to fill the holes that belong to the out-of-field regions. A depth image preprocessing method is applied to project the ghosts to their correct place. The 3D warping process is optimized by an adaptive one-to-four method to deal with the cracks and pixel overlapping. For disocclusions, we calculate depth and background terms of the filling priority based on depth information. The search for the best matching patch is performed simultaneously in the reference image and the virtual image. Moreover, adaptive patch size is used in all hole-filling processes. Experimental results demonstrate the effectiveness of the proposed method, which has better performance compared with previous methods in subjective and objective evaluation.
Source: crossref · Quality: baseline

Research context

OpenAlex topics: Advanced Vision and Imaging · Advanced Image Processing Techniques · Video Coding and Compression Technologies

SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.

Affiliations

School of Precision Instrument & Opto-Electronics Engineering, Tianjin University, Key Laboratory of Opto-Electronics Information Technology (Tianjin University), Ministry of Education, Tianjin 300072, China

Funding

the National Major Project of Scientific and Technical Supporting Programs of China during the 13th Five-year Plan Period

References (31)

Show source-provided references
1. Overview of the Multiview and 3D Extensions of High Efficiency Video Coding10.1109/TCSVT.2015.2477935 ↗
2. Overview of free viewpoint television10.1016/j.image.2006.03.009 ↗
3. Luo, G.B., Zhu, Y.S., Li, Z.T., and Zhang, L.M. (2016, January 27–30). A Hole Filling Approach based on Background Reconstruction for View Synthesis in 3D Video. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, New York, NY, USA.10.1109/CVPR.2016.197 ↗
4. Fehn, C. (2004). Depth-image-based rendering (DIBR), compression and transmission for a new approach on 3D-TV. Stereoscopic Displays and Virtual Reality Systems XI, Spie-Int Soc Optical Engineering.10.1117/12.524762 ↗
5. An Asymmetric Edge Adaptive Filter for Depth Generation and Hole Filling in 3DTV10.1109/TBC.2010.2053971 ↗
6. View generation with 3D warping using depth information for FTV10.1016/j.image.2008.10.013 ↗
7. Virtual view synthesis using layered depth image generation and depth-based inpainting for filling disocclusions and translucent disocclusions10.1016/j.jvcir.2016.02.017 ↗
8. Zhang, L., Tam, W.J., and Wang, D.M. (2004, January 24–27). Stereoscopic image generation based on depth images. Proceedings of the 2004 International Conference on Image Processing, Singapore.
9. Stereoscopic image generation based on depth images for 3D TV10.1109/TBC.2005.846190 ↗
10. Chen, W.Y., Chang, Y.L., Lin, S.F., Ding, L.F., and Chen, L.G. (2005, January 6). Efficient depth image-based rendering with edge dependent depth filter and interpolation. Proceedings of the 2005 IEEE International Conference on Multimedia and Expo, Amsterdam, The Netherlands.
11. Lu, X., Wei, F., and Chen, F. (2012, January 9–13). Foreground-Object-Protected Depth Map Smoothing for DIBR. Proceedings of the 2012 IEEE International Conference on Multimedia and Expo, Melbourne, VIC, Australia.10.1109/ICME.2012.116 ↗
12. A divide-and-conquer hole-filling method for handling disocclusion in single-view rendering10.1007/s11042-016-3413-3 ↗
13. Liu, W., Ma, L.Y., Qiu, B., Cui, M.Y., and Ding, J.W. (2017). An efficient depth map preprocessing method based on structure-aided domain transform smoothing for 3D view generation. PLoS ONE, 12.10.1371/journal.pone.0175910 ↗
14. Region filling and object removal by exemplar-based image inpainting10.1109/TIP.2004.833105 ↗
15. A Novel Inpainting-Based Layered Depth Video for 3DTV10.1109/TBC.2011.2125110 ↗
16. Oliveira, A., Fickel, G., Walter, M., and Jung, C. (2015, January 19–24). Selective hole-filling for depth-image based rendering. Proceedings of the 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brisbane, QLD, Australia.10.1109/ICASSP.2015.7178157 ↗
17. Stereoscopic image generation with depth image-based rendering. Multimed10.1007/s11042-016-3733-3 ↗
18. Fast MRF-Based Hole Filling for View Synthesis10.1109/LSP.2017.2720182 ↗
19. Foreground Removal Approach for Hole Filling in 3D Video and FVV Synthesis10.1109/TCSVT.2016.2583978 ↗
20. View synthesis using foreground object extraction for disparity control and image inpainting10.1016/j.jvcir.2018.10.004 ↗
21. Efficient dense stereo with occlusions for new view-synthesis by four-state dynamic programming10.1007/s11263-006-8525-1 ↗
22. Depth Map Driven Hole Filling Algorithm Exploiting Temporal Correlation Information10.1109/TBC.2014.2321671 ↗
23. Sun, W.X., Au, O.C., Xu, L.F., Li, Y.J., and Hu, W. (October, January 30). Novel temporal domain hole filling based on background modeling for view synthesis. Proceedings of the 2012 IEEE International Conference on Image Processing, Orlando, FL, USA.10.1109/ICIP.2012.6467461 ↗
24. Deng, Z., and Wang, M. (2018). Reliability-Based View Synthesis for Free Viewpoint Video. Appl. Sci., 8.10.3390/app8050823 ↗
25. Depth map creation and image-based rendering for advanced 3DTV services providing interoperability and scalability10.1016/j.image.2006.11.013 ↗
26. Depth Image Based View Synthesis: New Insights and Perspectives on Hole Generation and Filling10.1109/TBC.2015.2475697 ↗
27. Hole Filling for View Synthesis Using Depth Guided Global Optimization10.1109/ACCESS.2018.2847312 ↗
28. Hole Filling with Multiple Reference Views in DIBR View Synthesis10.1109/TMM.2018.2791810 ↗
29. A Real-Time High-Quality Complete System for Depth Image-Based Rendering on FPGA10.1109/TCSVT.2018.2825022 ↗
30. High-quality video view interpolation using a layered representation10.1145/1015706.1015766 ↗
31. Image quality assessment: From error visibility to structural similarity10.1109/TIP.2003.819861 ↗