MAC-I2P: I2P Registration with Modality Approximation and Cone–Block–Point Matching
Source abstract
Abstract
The misaligned geometric representation between images and point clouds and the different data densities limit the performance of I2P registration. The former hinders the learning of cross-modal features, and the latter leads to low-quality 2D–3D matching. To address these challenges, we propose a novel I2P registration framework called MAC-I2P, which is composed of a modality approximation module and a cone–block–point matching strategy. By generating pseudo-RGBD images, the module mitigates geometrical misalignment and converts 2D images into 3D space. In addition, it voxelizes the point cloud so that the features of the image and the point cloud can be processed in a similar way, thereby enhancing the repeatability of cross-modal features. Taking into account the different data densities and perception ranges between images and point clouds, the cone–block–point matching relaxes the strict one-to-one matching criterion by gradually refining the matching candidates. As a result, it effectively improves the 2D–3D matching quality. Notably, MAC-I2P is supervised by multiple matching objectives and optimized in an end-to-end manner, which further strengthens the cross-modal representation capability of the model. Extensive experiments conducted on KITTI Odometry and Oxford Robotcar demonstrate the superior performance of our MAC-I2P. Our approach surpasses the current state-of-the-art (SOTA) by 8∼63.2% in relative translation error (RTE) and 19.3∼38.5% in relative rotation error (RRE). The ablation experiments also confirm the effectiveness of each proposed component.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Robotics and Sensor-Based Localization · Advanced Image and Video Retrieval Techniques · Anomaly Detection Techniques and Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer Science and Technology, Tongji University, Shanghai 201804, China
Funding
National Natural Science Foundation of China · Fundamental Research Funds for the Central Universities
References (47)
Show source-provided references
1. Multiple-robot simultaneous localization and mapping: A review10.1002/rob.21620 ↗
2. VINS-Mono: A robust and versatile monocular visual-inertial state estimator10.1109/TRO.2018.2853729 ↗
3. Zhang, J., and Singh, S. (2014, January 12–16). LOAM: LiDAR Odometry and Mapping in Real-time. Proceedings of the Robotics: Science and Systems, Berkeley, CA, USA.10.15607/RSS.2014.X.007 ↗
4. Global Localization in Large-Scale Point Clouds via Roll-Pitch-Yaw Invariant Place Recognition and Low-Overlap Global Registration10.1109/TCSVT.2023.3323498 ↗
5. R2LIVE: A robust, real-Time, LiDAR-inertial-visual tightly-coupled state estimator and mapping10.1109/LRA.2021.3095515 ↗
6. Lin, J., and Zhang, F. (2022, January 23–27). R3LIVE: A robust, real-time, RGB-colored, LiDAR-inertial-visual tightly-coupled state estimation and mapping package. Proceedings of the IEEE International Conference on Robotics and Automation, Philadelphia, PA, USA.10.1109/ICRA46639.2022.9811935 ↗
7. Wang, Y., Sun, R., Luo, N., Pan, Y., and Zhang, T. (2024, January 16–22). Image-to-Image Matching via Foundation Models: A New Perspective for Open-Vocabulary Semantic Segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR52733.2024.00379 ↗
8. Huang, S., Gojcic, Z., Usvyatsov, M., Wieser, A., and Schindler, K. (2021, January 20–25). PREDATOR: Registration of 3D point clouds with low overlap. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.00425 ↗
9. Zhou, J., Ma, B., Zhang, W., Fang, Y., Liu, Y.S., and Han, Z. (2023, January 10–16). Differentiable Registration of Images and LiDAR Point Clouds with VoxelPoint-to-Pixel Matching. Proceedings of the Advances in Neural Information Processing Systems, New Orleans, LA, USA.
10. Feng, M., Hu, S., Ang, M.H., and Lee, G.H. (2019, January 20–24). 2D3D-Matchnet: Learning To Match Keypoints Across 2D Image And 3D Point Cloud. Proceedings of the IEEE International Conference on Robotics and Automation, Montreal, QC, Canada.10.1109/ICRA.2019.8794415 ↗
11. Li, J., and Hee Lee, G. (2021, January 20–25). DeepI2P: Image-to-Point Cloud Registration via Deep Classification. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.01570 ↗
12. CorrI2P: Deep Image-to-Point Cloud Registration via Dense Correspondence10.1109/TCSVT.2022.3208859 ↗
13. Geiger, A., Lenz, P., and Urtasun, R. (2012, January 16–21). Are we ready for autonomous driving? The KITTI vision benchmark suite. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA.10.1109/CVPR.2012.6248074 ↗
14. 1 Year, 1000 km: The Oxford RobotCar dataset10.1177/0278364916679498 ↗
15. Potje, G., Cadar, F., Araujo, A., Martins, R., and Nascimento, E.R. (2024, January 16–22). XFeat: Accelerated Features for Lightweight Image Matching. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR52733.2024.00259 ↗
16. Distinctive Image Features from Scale-Invariant Keypoints10.1023/B:VISI.0000029664.99615.94 ↗
17. EPnP: An accurate O(n) solution to the PnP problem10.1007/s11263-008-0152-6 ↗
18. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
19. DeTone, D., Malisiewicz, T., and Rabinovich, A. (2018, January 18–22). SuperPoint: Self-Supervised Interest Point Detection and Description. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshop, Salt Lake City, UT, USA.10.1109/CVPRW.2018.00060 ↗
20. Sarlin, P., DeTone, D., Malisiewicz, T., and Rabinovich, A. (2020, January 14–19). SuperGlue: Learning Feature Matching with Graph Neural Networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00499 ↗
21. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017, January 4–9). Attention is all you need. Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA.
22. Sun, J., Shen, Z., Wang, Y., Bao, H., and Zhou, X. (2021, January 20–25). LoFTR: Detector-Free Local Feature Matching with Transformers. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.00881 ↗
23. Wang, Y., He, X., Peng, S., Tan, D., and Zhou, X. (2024, January 16–22). Efficient LoFTR: Semi-Dense Local Feature Matching with Sparse-Like Speed. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR52733.2024.02047 ↗
24. Modeling Fabric-Type Actuator Using Point Clouds by Deep Learning10.1109/ACCESS.2022.3204652 ↗
25. Choy, C., Park, J., and Koltun, V. (November, January 27). Fully Convolutional Geometric Features. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.
26. Rusu, R.B., Blodow, N., and Beetz, M. (2009, January 12–17). Fast Point Feature Histograms (FPFH) for 3D registration. Proceedings of the IEEE International Conference on Robotics and Automation, Kobe, Japan.10.1109/ROBOT.2009.5152473 ↗
27. Aoki, Y., Goforth, H., Srivatsan, R.A., and Lucey, S. (2019, January 15–20). PointNetLK: Robust & Efficient Point Cloud Registration Using PointNet. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00733 ↗
28. Sarode, V., Li, X., Goforth, H., Aoki, Y., Srivatsan, R.A., Lucey, S., and Choset, H. (2019). PCRNet: Point Cloud Registration Network using PointNet Encoding. arXiv.
29. GeoTransformer: Fast and Robust Point Cloud Registration With Geometric Transformer10.1109/TPAMI.2023.3259038 ↗
30. Mu, J., Bie, L., Du, S., and Gao, Y. (2024, January 16–22). ColorPCR: Color Point Cloud Registration with Multi-Stage Geometric-Color Fusion. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR52733.2024.01990 ↗
31. Yuan, Y., Wu, Y., Fan, X., Gong, M., Miao, Q., and Ma, W. (2024, January 16–22). Inlier Confidence Calibration for Point Cloud Registration. Proceedings of the Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR52733.2024.00508 ↗
32. Zhang, X., Yang, J., Zhang, S., and Zhang, Y. (2023, January 18–22). 3D Registration with Maximal Cliques. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.01702 ↗
33. EFGHNet: A Versatile Image-to-Point Cloud Registration Network for Extreme Outdoor Environment10.1109/LRA.2022.3183899 ↗
34. Zhong, Y. (2009, January 27). Intrinsic shape signatures: A shape descriptor for 3D object recognition. Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, Kyoto, Japan.10.1109/ICCVW.2009.5457637 ↗
35. Wang, B., Chen, C., Cui, Z., Qin, J., Lu, C.X., Yu, Z., Zhao, P., Dong, Z., Zhu, F., and Trigoni, N. (2021, January 10–17). P2-Net: Joint Description and Detection of Local Features for Pixel and Point Matching. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada.10.1109/ICCV48922.2021.01570 ↗
36. Bhunia, A., Li, C., and Bilen, H. (2024, January 16–22). Looking 3D: Anomaly Detection with 2D-3D Alignment. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR52733.2024.01634 ↗
37. Pixel-Level Extrinsic Self Calibration of High Resolution LiDAR and Camera in Targetless Environments10.1109/LRA.2021.3098923 ↗
38. Koide, K., Oishi, S., Yokozuka, M., and Banno, A. (June, January 29). General, Single-shot, Target-less, and Automatic LiDAR-Camera Extrinsic Calibration Toolbox. Proceedings of the IEEE International Conference on Robotics and Automation, London, UK.
39. Targetless Extrinsic Calibration of Camera and LiDAR via Identifying True 2-D–3-D Line Matching Among Unknown Line Correspondences of Structured Environments
40. Li, M., Qin, Z., Gao, Z., Yi, R., Zhu, C., Guo, Y., and Xu, K. (2023, January 1–6). 2D3D-MATR: 2D-3D Matching Transformer for Detection-Free Registration Between Images and Point Clouds. Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France.10.1109/ICCV51070.2023.01299 ↗
41. Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography10.1145/358669.358692 ↗
42. Yang, L., Kang, B., Huang, Z., Xu, X., Feng, J., and Zhao, H. (2024, January 16–22). Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR52733.2024.00987 ↗
43. Qi, C.R., Yi, L., Su, H., and Guibas, L.J. (2017, January 4–9). PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space. Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA.
44. PVNAS: 3D Neural Architecture Search with Point-Voxel Convolution
45. Sun, Y., Cheng, C., Zhang, Y., Zhang, C., Zheng, L., Wang, Z., and Wei, Y. (2020, January 14–19). Circle Loss: A Unified Perspective of Pair Similarity Optimization. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00643 ↗
46. Kim, M., Koo, J., and Kim, G. (2023, January 1–6). EP2P-Loc: End-to-End 3D Point to 2D Pixel Localization for Large-Scale Visual Localization. Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France.10.1109/ICCV51070.2023.01968 ↗
47. Sun, Y., Zhang, L., Wang, Z., Chen, Y., Zhao, S., and Zhou, Y. (2024). I2P Registration by Learning the Underlying Alignment Feature Space from Pixel-to-Point Similarities. ACM Transactions on Multimedia Computing, Communications and Applications, Association for Computing Machinery.10.1145/3697839 ↗