Tracking and Registration Technology Based on Panoramic Cameras
Source abstract
Abstract
Augmented reality (AR) has become a research focus in computer vision and graphics, with growing applications driven by advances in artificial intelligence and the emergence of the metaverse. Panoramic cameras offer new opportunities for AR due to their wide field of view but also pose significant challenges for camera pose estimation because of severe distortion and complex scene textures. To address these issues, this paper proposes a lightweight, unsupervised deep learning model for panoramic camera pose estimation. The model consists of a depth estimation sub-network and a pose estimation sub-network, both optimized for efficiency using network compression, multi-scale rectangular convolutions, and dilated convolutions. A learnable occlusion mask is incorporated into the pose network to mitigate errors caused by complex relative motion. Furthermore, a panoramic view reconstruction model is constructed to obtain effective supervisory signals from the predicted depth, pose information, and corresponding panoramic images and is trained using a designed spherical photometric consistency loss. The experimental results demonstrate that the proposed method achieves competitive accuracy while maintaining high computational efficiency, making it well-suited for real-time AR applications with panoramic input.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Robotics and Sensor-Based Localization · 3D Surveying and Cultural Heritage · Advanced Image and Video Retrieval Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer Science and Technology, Changchun University of Science and Technology, Changchun 130022, China
School of Information Engineering, Changchun College of Electronic Technology, Changchun 130114, China
Funding
Natural Science Foundation of Jilin of China
References (35)
Show source-provided references
1. Kato, H., and Billinghurst, M. (1999, January 20–21). Marker Tracking and HMD Calibration for a Video-Based Augmented Reality Conferencing System. Proceedings of the 2nd IEEE and ACM International Workshop on Augmented Reality (IWAR’99), San Francisco, CA, USA.
2. Wagner, D., Schmalstieg, D., and Bischof, H. (2009, January 19–22). Multiple target detection and tracking with guaranteed framerates on mobile phones. Proceedings of the 8th IEEE International Symposium on Mixed and Augmented Reality, Orlando, FL, USA.10.1109/ISMAR.2009.5336497 ↗
3. Klein, G., and Murray, D. (2007, January 13–16). Parallel tracking and mapping for small AR workspaces. Proceedings of the 6th IEEE and ACM International Symposium on Mixed and Augmented Reality, Nara, Japan.10.1109/ISMAR.2007.4538852 ↗
4. ORB-SLAM2: An open-source SLAM system for monocular, stereo, and RGB-D cameras10.1109/TRO.2017.2705103 ↗
5. Newcombe, R.A., Lovegrove, S.J., and Davison, A.J. (2011, January 6–13). DTAM: Dense tracking and mapping in real-time. Proceedings of the 2011 International Conference on Computer Vision (ICCV), Barcelona, Spain.10.1109/ICCV.2011.6126513 ↗
6. Engel, J., Schöps, T., and Cremers, D. (2014, January 6–12). LSD-SLAM: Large-scale direct monocular SLAM. Proceedings of the European Conference on Computer Vision (ECCV), Zurich, Switzerland.10.1007/978-3-319-10605-2_54 ↗
7. Direct sparse odometry10.1109/TPAMI.2017.2658577 ↗
8. Forster, C., Pizzoli, M., and Scaramuzza, D. (June, January 31). SVO: Fast semi-direct monocular visual odometry. Proceedings of the 2014 IEEE International Conference on Robotics and Automation (ICRA), Hong Kong, China.10.1109/ICRA.2014.6906584 ↗
9. Omnidirectional DSO: Direct Sparse Odometry with Fisheye Cameras10.1109/LRA.2018.2855443 ↗
10. Seok, H., and Lim, J. (2019, January 20–24). ROVO: Robust Omnidirectional Visual Odometry for Wide-Baseline Wide-FoV Camera Systems. Proceedings of the 2019 International Conference on Robotics and Automation (ICRA), Montreal, QC, Canada.10.1109/ICRA.2019.8793758 ↗
11. Panoramic SLAM from a Multiple Fisheye Camera Rig10.1016/j.isprsjprs.2019.11.014 ↗
12. Huang, H., and Yeung, S.-K. (2022, January 23–27). 360VO: Visual Odometry Using a Single 360 Camera. Proceedings of the 2022 International Conference on Robotics and Automation (ICRA), Philadelphia, PA, USA.10.1109/ICRA46639.2022.9812203 ↗
13. Sumikura, S., Shibuya, M., and Sakurada, K. (2019, January 21–25). OpenVSLAM: A Versatile Visual SLAM Framework. Proceedings of the 27th ACM International Conference on Multimedia, Nice, France.10.1145/3343031.3350539 ↗
14. Kendall, A., Grimes, M., and Cipolla, R. (2015, January 7–13). Posenet: A Convolutional Network for Real-Time 6-DoF Camera Relocalization. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.10.1109/ICCV.2015.336 ↗
15. Wang, S., Clark, R., Wen, H., and Trigoni, N. (June, January 29). DeepVO: Towards End-to-End Visual Odometry with Deep Recurrent Convolutional Neural Networks. Proceedings of the 2017 IEEE International Conference on Robotics and Automation (ICRA), Singapore.10.1109/ICRA.2017.7989236 ↗
16. Zhou, T., Brown, M., Snavely, N., and Lowe, D.G. (2017, January 21–26). Unsupervised Learning of Depth and Ego-Motion from Video. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.700 ↗
17. Yin, Z., and Shi, J. (2018, January 18–22). GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.10.1109/CVPR.2018.00212 ↗
18. Wang, F.-E., Hu, H.-N., Cheng, H.-T., Lin, J.-T., Yang, S.-T., Shih, M.-L., Chu, H.-K., and Sun, M. (2018, January 2–6). Self-Supervised Learning of Depth and Camera Motion from 360 Videos. Proceedings of the Asian Conference on Computer Vision, Perth, Australia.10.1007/978-3-030-20873-8_4 ↗
19. Sharma, A., and Ventura, J. (2019, January 9–11). Unsupervised Learning of Depth and Ego-Motion from Cylindrical Panoramic Video. Proceedings of the 2019 IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR), San Diego, CA, USA.10.1109/AIVR46125.2019.00018 ↗
20. Pano-SfMLearner: Self-Supervised Multi-Task Learning of Depth and Semantics in Panoramic Videos10.1109/LSP.2021.3073627 ↗
21. Hasegawa, Y., Ikehata, S., and Aizawa, K. (2022). Distortion-Aware Self-Supervised 360° Depth Estimation from a Single Equirectangular Projection Image. arXiv.
22. Bifuse++: Self-Supervised and Efficient Bi-Projection Fusion for 360° Depth Estimation
23. Survey of Vision-Based Augmented Reality 3D Registration Technology
24. Review of Visual Odometry Methods Based on Deep Learning
25. Zhang, Z., Rebecq, H., Forster, C., and Scaramuzza, D. (2016, January 16–21). Benefit of Large Field-of-View Cameras for Visual Odometry. Proceedings of the 2016 IEEE International Conference on Robotics and Automation (ICRA), Stockholm, Sweden.
26. ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual–Inertial, and Multimap SLAM10.1109/TRO.2021.3075644 ↗
27. Caruso, D., Engel, J., and Cremers, D. (October, January 28). Large-Scale Direct SLAM for Omnidirectional Cameras. Proceedings of the 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Hamburg, Germany.10.1109/IROS.2015.7353366 ↗
28. Zioulis, N., Karakottas, A., Zarpalas, D., and Daras, P. (2018, January 8–14). Omnidepth: Dense Depth Estimation for Indoors Spherical Panoramas. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01231-1_28 ↗
29. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
30. Song, S., Yu, F., Zeng, A., Chang, A.X., Savva, M., and Funkhouser, T. (2017, January 21–26). Semantic Scene Completion from a Single Depth Image. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.28 ↗
31. Kinga, D., and Ba, J. (2015, January 7–9). Adam: A Method for Stochastic Optimization. Proceedings of the International Conference on Learning Representations (ICLR), San Diego, CA, USA.
32. Glorot, X., and Bengio, Y. (2010, January 13–15). Understanding the Difficulty of Training Deep Feedforward Neural Networks. Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics (AISTATS), Sardinia, Italy.
33. Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. (2009, January 20–25). ImageNet: A Large-Scale Hierarchical Image Database. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Miami, FL, USA.10.1109/CVPR.2009.5206848 ↗
34. Eigen, D., Puhrsch, C., and Fergus, R. (2014, January 8–13). Depth Map Prediction from a Single Image Using a Multi-Scale Deep Network. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Montreal, QC, Canada.
35. Sturm, J., Engelhard, N., Endres, F., Burgard, W., and Cremers, D. (2012, January 7–12). A Benchmark for the Evaluation of RGB-D SLAM Systems. Proceedings of the 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Vilamoura, Portugal.10.1109/IROS.2012.6385773 ↗