Research on Indoor 3D Semantic Mapping Based on ORB-SLAM2 and Multi-Object Tracking
Source abstract
Abstract
The integration of semantic simultaneous localization and mapping (SLAM) with 3D object detection in indoor scenes is a significant challenge in the field of robot perception. Existing methods typically rely on expensive sensors and lack robustness and accuracy in complex environments. To address this, this paper proposes a novel 3D semantic SLAM framework that integrates Oriented FAST and Rotated BRIEF-SLAM2 (ORB-SLAM2), 3D object detection, and multi-object tracking (MOT) techniques to achieve efficient and robust semantic environment modeling. Specifically, we employ an improved 3D object detection network to extract semantic information and enhance detection accuracy through category balancing strategies and optimized loss functions. Additionally, we introduce MOT algorithms to filter and track 3D bounding boxes, enhancing stability in dynamic scenes. Finally, we deeply integrate 3D semantic information into the SLAM system, achieving high-precision 3D semantic map construction. Experiments were conducted on the public dataset SUNRGBD and two self-collected datasets (robot navigation and XR glasses scenes). The results show that, compared with the current state-of-the-art methods, our method demonstrates significant advantages in detection accuracy, localization accuracy, and system robustness, providing an effective solution for low-cost, high-precision indoor semantic SLAM.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Robotics and Sensor-Based Localization · 3D Surveying and Cultural Heritage · Remote Sensing and LiDAR Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Institute of Space Optoelectronic Technology, Changchun University of Science and Technology, Changchun 130022, China
OPPO Artificial Intelligence Center, Beijing 100026, China
Funding
the National Key R&D Program of China
References (64)
Show source-provided references
1. Wu, H., Liu, Y., Wang, C., and Wei, Y. (2025). An Effective 3D Instance Map Reconstruction Method Based on RGBD Images for Indoor Scene. Remote Sens., 17.10.3390/rs17010139 ↗
2. Perfetti, L., Fassi, F., and Vassena, G. (2024). Ant3D—A Fisheye Multi-Camera System to Survey Narrow Spaces. Sensors, 24.10.20944/preprints202405.1466.v1 ↗
3. Open Source, Open Hardware Hand-Held Mobile Mapping System for Large Scale Surveys10.1016/j.softx.2023.101618 ↗
4. Gao, X., Yang, R., Chen, X., Tan, J., Liu, Y., Wang, Z., Tan, J., and Liu, H. (2024). A New Framework for Generating Indoor 3D Digital Models from Point Clouds. Remote Sens., 16.10.3390/rs16183462 ↗
5. Huang, Y., Xie, F., Zhao, J., Gao, Z., Chen, J., Zhao, F., and Liu, X. (2024). ULG-SLAM: A Novel Unsupervised Learning and Geometric Feature-Based Visual SLAM Algorithm for Robot Localizability Estimation. Remote Sens., 16.10.3390/rs16111968 ↗
6. SG-SLAM: A Real-Time RGB-D Visual SLAM Toward Dynamic Scenes with Semantic and Geometric Information10.1109/TIM.2022.3228006 ↗
7. Ji, T., Wang, C., and Xie, L. (June, January 30). Towards Real-time Semantic RGB-D SLAM in Dynamic Environments. Proceedings of the 2021 IEEE International Conference on Robotics and Automation (ICRA), Xi’an, China.10.1109/ICRA48506.2021.9561743 ↗
8. Hu, X. (2023). Multi-level map construction for dynamic scenes. arXiv.
9. Wang, T., Zhu, X., Pang, J., and Lin, D. (2021, January 11–17). Fcos3d: Fully convolutional one-stage monocular 3d object detection. Proceedings of the IEEE/CVF International Conference on Computer Vision, Online.10.1109/ICCVW54120.2021.00107 ↗
10. Tian, Z., Shen, C., Chen, H., and Chen, H. (November, January 27). Fcos: Fully convolutional one-stage object detection. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.
11. Park, D., Ambrus, R., Guizilini, V., Li, J., and Gaidon, A. (2021, January 11–17). Is pseudo-lidar needed for monocular 3d object detection?. Proceedings of the IEEE/CVF International Conference on Computer Vision, Online.10.1109/ICCV48922.2021.00313 ↗
12. Lin, T.Y., Dollár, P., Girshick, R., He, K., Hariharan, B., and Belongie, S. (2017, January 21–26). Feature pyramid networks for object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.106 ↗
13. Philion, J., and Fidler, S. (2020, January 23–28). Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. Proceedings of the European Conference on Computer Vision, Glasgow, UK.10.1007/978-3-030-58568-6_12 ↗
14. Reading, C., Harakeh, A., Chae, J., and Waslander, S.L. (2021, January 19–25). Categorical depth distribution network for monocular 3d object detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Online.10.1109/CVPR46437.2021.00845 ↗
15. Huang, J., Huang, G., Zhu, Z., Ye, Y., and Du, D. (2021). Bevdet: High-performance multi-camera 3d object detection in bird-eye-view. arXiv.
16. Zhang, R., Qiu, H., Wang, T., Guo, Z., Cui, Z., Qiao, Y., Li, H., and Gao, P. (2023, January 2–6). MonoDETR: Depth-guided transformer for monocular 3D object detection. Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France.10.1109/ICCV51070.2023.00840 ↗
17. Yin, Z., Wen, H., Nie, W., and Zhou, M. (2023). Localization of Mobile Robots Based on Depth Camera. Remote Sens., 15.10.3390/rs15164016 ↗
18. Wang, T., Lian, Q., Zhu, C., Zhu, X., and Zhang, W. (2022). Mv-fcos3d++: Multi-view camera-only 4d object detection with pretrained monocular backbones. arXiv.
19. Yan, L., Yan, P., Xiong, S., Xiang, X., and Tan, Y. (2024, January 17–21). Monocd: Monocular 3d object detection with complementary depths. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR52733.2024.00976 ↗
20. Li, Z., Xu, X., Lim, S., and Zhao, H. (2024, January 17–21). Unimode: Unified monocular 3d object detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR52733.2024.01567 ↗
21. Barreiros, M.O., Dantas, D.O., Silva, L.C.O., Ribeiro, S., and Barros, A.K. (2021). Zebrafish tracking using YOLOv2 and Kalman filter. Sci. Rep., 11.10.1038/s41598-021-81997-9 ↗
22. Using Scene-Flow to Improve Predictions of Road Users in Motion with Respect to an Ego-Vehicle10.1049/itr2.70010 ↗
23. 3D multi-object tracking with adaptive cubature Kalman filter for autonomous driving10.1109/TIV.2022.3158419 ↗
24. SMART-TRACK: A Novel Kalman Filter-Guided Sensor Fusion for Robust UAV Object Tracking in Dynamic Environments10.1109/JSEN.2024.3505939 ↗
25. Pang, Z., Li, J., Tokmakov, P., Chen, D., Zagoruyko, S., and Wang, Y.-X. (2023, January 11–15). Standing between past and future: Spatio-temporal modeling for multi-camera 3d multi-object tracking. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR52729.2023.01719 ↗
26. Wang, Y.H. (2022). Smiletrack: Similarity learning for multiple object tracking. arXiv.
27. Gu, J., Wu, B., Fan, L., Huang, J., Cao, S., Xiang, Z., and Hua, X.S. (2022, January 29–24). Homography loss for monocular 3d object detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00115 ↗
28. Efficient deep learning infrastructures for embedded computing systems: A comprehensive survey and future envision10.1145/3701728 ↗
29. Graft: Efficient Inference Serving for Hybrid Deep Learning with SLO Guarantees via DNN Re-Alignment10.1109/TPDS.2023.3340518 ↗
30. Tian, P., and Li, H. (2024). Visual SLAMMOT Considering Multiple Motion Models. arXiv.
31. Krishna, G.S., Supriya, K., and Baidya, S. (2023). 3ds-slam: A 3d object detection based semantic slam towards dynamic indoor environments. arXiv.
32. Tufte, E.R., and Graves-Morris, P.R. (1983). The Visual Display of Quantitative Information, Graphics Press.
33. Ware, C. (2019). Information Visualization: Perception for Design, Morgan Kaufmann.
34. SOF-SLAM: A semantic visual SLAM for dynamic environments10.1109/ACCESS.2019.2952161 ↗
35. 3D semantic map construction using improved ORB-SLAM2 for mobile robot in edge computing environment10.1109/ACCESS.2020.2983488 ↗
36. Dynamic scene semantics SLAM based on semantic segmentation10.1109/ACCESS.2020.2977684 ↗
37. Yuan, X., and Chen, S. (2020–24, January 24). Sad-slam: A visual slam based on semantic and depth information. Proceedings of the 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Las Vegas, NV, USA.10.1109/IROS45743.2020.9341180 ↗
38. Vins-mono: A robust and versatile monocular visual-inertial state estimator10.1109/TRO.2018.2853729 ↗
39. ORB-SLAM: A Versatile Accurate Monocular SLAM system10.1109/TRO.2015.2463671 ↗
40. Orb-slam2: An open-source slam system for monocular, stereo, and rgb-d cameras10.1109/TRO.2017.2705103 ↗
41. Orb-slam3: An accurate open-source library for visual, visual–inertial, and multimap slam10.1109/TRO.2021.3075644 ↗
42. Engel, J., Schöps, T., and Cremers, D. (2014, January 6–12). LSD-SLAM: Large-scale direct monocular SLAM. Proceedings of the European Conference on Computer Vision, Zurich, Switzerland.10.1007/978-3-319-10605-2_54 ↗
43. Zhang, J., Henein, M., Mahony, R., and Ila, V. (2020). VDO-SLAM: A visual dynamic object-aware SLAM system. arXiv.
44. Tateno, K., Tombari, F., Laina, I., and Navab, N. (2017, January 21–26). Cnn-slam: Real-time dense monocular slam with learned depth prediction. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.695 ↗
45. Li, X., and Belaroussi, R. (2016). Semi-dense 3d semantic mapping from monocular slam. arXiv.10.1109/ITSC.2017.8317942 ↗
46. Ran, Y., Xu, X., Luo, M., Yang, J., and Chen, Z. (2024). Scene Classification Method Based on Multi-Scale Convolutional Neural Network with Long Short-Term Memory and Whale Optimization Algorithm. Remote Sens., 16.10.3390/rs16010174 ↗
47. Bowman, S.L., Atanasov, N., Daniilidis, K., and Pappas, G.J. (June, January 29). Probabilistic data association for semantic slam. Proceedings of the 2017 IEEE International Conference on Robotics and Automation (ICRA), Singapore.10.1109/ICRA.2017.7989203 ↗
48. Lianos, K.N., Schonberger, J.L., Pollefeys, M., and Sattler, T. (2018, January 8–14). Vso: Visual semantic odometry. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01225-0_15 ↗
49. Civera, J., Gálvez-López, D., Riazuelo, L., Tardos, J.D., and Montiel, J.M.M. (2011, January 25–30). Towards semantic SLAM using a monocular camera. Proceedings of the 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems, San Francisco, CA, USA.10.1109/IROS.2011.6048293 ↗
50. Rukhovich, D., Vorontsova, A., and Konushin, A. (2022, January 4–8). Imvoxelnet: Image to voxels projection for monocular and multi-view general-purpose 3d object detection. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, Waikoloa, HI, USA.10.1109/WACV51458.2022.00133 ↗
51. Qi, C.R., Chen, X., Litany, O., and Guibas, L.J. (2020, January 14–19). Imvotenet: Boosting 3d object detection in point clouds with image votes. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Online.10.1109/CVPR42600.2020.00446 ↗
52. Ding, Z., Han, X., and Niethammer, M. (2019, January 13–17). Votenet: A deep learning label fusion method for multi-atlas segmentation. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Shenzhen, China.10.1007/978-3-030-32248-9_23 ↗
53. Comparing of target-tracking performances of EKF, UKF and PF
54. Gupta, S.D., Yu, J.Y., Mallick, M., Coates, M., and Morelande, M. (2015, January 6–9). Comparison of angle-only filtering algorithms in 3D using EKF, UKF, PF, PFF, and ensemble KF. Proceedings of the 2015 18th International Conference on Information Fusion (Fusion), Washington, DC, USA.
55. Weng, X., Wang, J., Held, D., and Kitani, K. (2020). Ab3dmot: A baseline for 3d multi-object tracking and new evaluation metrics. arXiv.10.1109/IROS45743.2020.9341164 ↗
56. Song, S., Lichtenberg, S.P., and Xiao, J. (2015, January 8–10). Sun rgb-d: A rgb-d scene understanding benchmark suite. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298655 ↗
57. Wang, Y., Ye, T.Q., Cao, L., Huang, W., Sun, F., He, F., and Tao, D. (2022, January 19–24). Bridged transformer for vision and point cloud 3d object detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01180 ↗
58. He, K., Zhang, X., Ren, S., and Sun, J. (July, January 26). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.
59. Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. (July, January 26). You only look once: Unified, real-time object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.
60. Huang, S., Qi, S., Zhu, Y., Xiao, Y., Xu, Y., and Zhu, S.C. (2018, January 8–14). Holistic 3d scene parsing and reconstruction from a single rgb image. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01234-2_12 ↗
61. Cooperative holistic scene understanding: Unifying 3d object, layout, and camera pose estimation
62. Nie, Y., Han, X., Guo, S., Zheng, Y., and Zhang, J.J. (2020, January 14–19). Total3dunderstanding: Joint layout, object pose and mesh reconstruction for indoor scenes from a single image. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Online.10.1109/CVPR42600.2020.00013 ↗
63. Qi, C.R., Liu, W., Wu, C., Su, H., and Guibas, L.J. (2018, January 18–22). Frustum pointnets for 3d object detection from rgb-d data. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00102 ↗
64. Zhang, Z., Sun, B., Yang, H., and Huang, Q. (2020, January 23–28). H3dnet: 3d object detection using hybrid geometric primitives. Proceedings of the European Conference on Computer Vision, Glasgow, UK.10.1007/978-3-030-58610-2_19 ↗