High-Precision Depth Estimation Networks Using Low-Resolution Depth and RGB Image Sensors for Low-Cost Mixed Reality Glasses
Source abstract
Abstract
In recent years, with the booming development of the three-dimensional and mixed reality (MR) industries, depth estimation tools have become increasingly important to support many visual problems. Intrinsically, the restriction of computational resources hardly lets complex depth completion methods be implemented on MR glasses. In this paper, we propose a competitive high-precision depth estimation network, which integrates a dual-path autoencoder and adaptive bin depth estimator together. The proposed network with different types of models can aptly fuse a pair of an RGB image and low-resolution low-quality depth map to generate a high-precision depth map. The simplest model of the proposed network has only 2.28 M weights and 1.150 G MACs. It is very lightweight, such that this model could be easily implemented into the platform of low-cost MR glasses to support hand-gesture controls realized in edge devices in real time.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Vision and Imaging · Industrial Vision Systems and Defect Detection · Optical measurement and interference techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Electrical Engineering, National Kaohsiung Normal University, Kaohsiung 824004, Taiwan
Institute of Computer and Communication Engineering, Department of Electrical Engineering, National Cheng Kung University, Tainan 701401, Taiwan
Department of Computer Science and Information Engineering, National Chiayi University, Chiayi 600355, Taiwan
Funding
National Science and Technology Council, Taiwan
References (35)
Show source-provided references
1. Real-time computation of distance to dynamic obstacles with multiple depth sensors10.1109/LRA.2016.2535859 ↗
2. Depth map creation and image-based rendering for advanced 3DTV services providing interoperability and scalability10.1016/j.image.2006.11.013 ↗
3. Depth-image-based rendering (DIBR), compression, and transmission for a new approach on 3d-tv10.1117/12.524762 ↗
4. An assigned color depth packing method with centralized texture depth packing formats for 3D VR broadcasting services10.1109/JETCAS.2018.2881968 ↗
5. End-to-end autonomous driving with semantic depth cloud mapping and multi-agent10.1109/TIV.2022.3185303 ↗
6. Suarez, J., and Murphy, R.R. (2012, January 9–13). Hand gesture recognition with depth images: A review. Proceedings of the 2012 IEEE RO-MAN: The 21st IEEE International Symposium on Robot and Human Interactive Communication, Paris, France.10.1109/ROMAN.2012.6343787 ↗
7. Stereo processing by semiglobal matching and mutual information10.1109/TPAMI.2007.1166 ↗
8. Eigen, D., Puhrsch, C., and Fergus, R. (2014). Depth map prediction from a single image using a multi-scale deep network. Advances in Neural Information Processing Systems, The MIT Press.
9. Alhashim, I., and Wonka, P. (2018). High quality monocular depth estimation via transfer learning. arXiv.
10. Wofk, D., Ma, F., Yang, T.-J., Karaman, S., and Sze, V. (2019, January 20–24). Fastdepth: Fast monocular depth estimation on embedded systems. Proceedings of the IEEE/CVF International Conference on Robotics and Automation (ICRA), Montreal, QC, Canada.10.1109/ICRA.2019.8794182 ↗
11. Laina, I., Rupprecht, C., Belagiannis, V., Tombari, F., and Navab, N. (2016, January 25–28). Deeper depth prediction with fully convolutional residual networks. Proceedings of the IEEE International Conference on 3D vision (3DV), Stanford, CA, USA.10.1109/3DV.2016.32 ↗
12. Bhat, S.F., Alhashim, I., and Wonka, P. (2021, January 20–25). Adabins: Depth estimation using adaptive bins. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.
13. Godard, C., Aodha, O.M., and Brostow, G.J. (2017, January 21–26). Unsupervised monocular depth estimation with left-right consistency. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.699 ↗
14. Godard, C., Aodha, O.M., Firman, M., and Brostow, G.J. (November, January 27). Digging into self-supervised monocular depth estimation. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.
15. Zama Ramirez, P., Poggi, M., Tosi, F., Mattoccia, S., and Di Stefano, L. (2019). Geometry meets semantics for semi-supervised monocular depth estimation. Computer Vision—ACCV 2018, Springer.10.1007/978-3-030-20893-6_19 ↗
16. Kuznietsov, Y., Stuckler, J., and Leibe, B. (2017, January 21–26). Semi-supervised deep learning for monocular depth map prediction. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.238 ↗
17. A review of sensor technologies for perception in automated driving10.1109/MITS.2019.2907630 ↗
18. Lock-in time-of-flight (ToF) cameras: A Survey10.1109/JSEN.2010.2101060 ↗
19. Jorjin (2023, August 01). J7EF Plus AR Glasses. Available online: https://www.jorjin.com/products/ar-vr-glasses/j-reality/j7ef/.
20. Ma, F., and Karaman, S. (2018, January 21–25). Sparse-to-dense: Depth prediction from sparse depth samples and a single image. Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), Brisbane, Australia.10.1109/ICRA.2018.8460184 ↗
21. Learning guided convolutional network for depth completion10.1109/TIP.2020.3040528 ↗
22. Cheng, X., Wang, P., and Yang, R. (2018, January 8–14). Depth estimation via affinity learned with convolutional spatial propagation network. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01270-0_7 ↗
23. Liu, S., De Mello, S., Gu, J., Zhong, G., Yang, M.-H., and Kautz, J. (2017). Learning affinity via spatial propagation networks. Advances in Neural Information Processing Systems (NIPS), The MIT Press.
24. HTC-DC Net: Monocular height estimation from single remote sensing images10.1109/TGRS.2023.3321255 ↗
25. Miclea, V.-C., and Nedevschi, S. (2024, January 4–6). SemanticAdaBins—Using semantics to improve depth estimation based on adaptive bins in aerial scenarios. Proceedings of the 4th International Conference on Electrical, Computer, Communications and Mechatronics Engineering (ICECCME), Male, Maldives.10.1109/ICECCME62383.2024.10797010 ↗
26. Yang, X., Yuan, L., Wilber, K., Sharma, A., Gu, X., and Qiao, S. (2024, January 3–8). PolyMaX: General dense prediction with mask transformer. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Waikoloa, HI, USA.10.1109/WACV57701.2024.00109 ↗
27. Jaisawal, P.K., and Papakonstantinou, S. (October, January 29). Monocular fisheye depth estimation for UAV applications with segmentation feature integration. Proceedings of the AIAA DATC/IEEE 43rd Digital Avionics Systems Conference (DASC), San Diego, CA, USA.
28. Improving monocular depth estimation through knowledge distillation: Better visual quality and efficiency10.1109/ACCESS.2024.3523912 ↗
29. Bhat, S.F., Birkl, R., Wofk, D., Wonka, P., and Muller, M. (2023, January 17–24). ZoeDepth: Zero-shot transfer by combining relative and metric depth. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada.
30. Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. (2017). Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv.
31. Fan, H., Su, H., and Guibas, L. (2017, January 21–26). A point set generation network for 3d object reconstruction from a single image. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.264 ↗
32. Silberman, N., Hoiem, D., Kohli, P., and Fergus, R. (2012, January 7–13). Indoor segmentation and support inference from rgbd images. Proceedings of the European Conference on Computer Vision (ECCV), Florence, Italy.10.1007/978-3-642-33715-4_54 ↗
33. Egogesture: A new dataset and benchmark for egocentric hand gesture recognition10.1109/TMM.2018.2808769 ↗
34. Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., and Antiga, L. (2019). Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems (NIPS), The MIT Press.
35. (2023, August 01). EPSON MOVERIO BO-IC400 User’s Guide. Available online: https://download3.ebz.epson.net/dsc/f/03/00/16/15/46/95ef7d99574b1d519141f71c055ea50600b6b390/UsersGuide_BO-IC400_EN_Rev04.pdf.