RSCS6D: Keypoint Extraction-Based 6D Pose Estimation
Source abstract
Abstract
In this work, we propose an improved network, RSCS6D, for 6D pose estimation from RGB-D images by extracting keypoint-based point clouds. Our key insight is that keypoint cloud can reduce data redundancy in 3D point clouds and accelerate the convergence of convolutional neural networks. First, we employ a semantic segmentation network on the RGB image to obtain mask images containing positional information and per-pixel labels. Next, we introduce a novel keypoint cloud extraction algorithm that combines RGB and depth images to detect 2D keypoints and convert them into 3D keypoints. Specifically, we convert the RGB image to grayscale and use the Sobel edge detection operator to identify 2D edge keypoints. Additionally, we compute the Curvature matrix from the depth image and apply the Sobel operator to extract keypoints critical for 6D pose estimation. Finally, the extracted 3D keypoint cloud is fed into the 6D pose estimation network to predict both translation and rotation.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Robot Manipulation and Learning · Robotics and Sensor-Based Localization · Image and Object Detection Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Academy for Advanced Interdisciplinary Studies, Northeast Normal University, No. 5268 Renmin Avenue, Changchun 130024, China
Funding
Department of Science, Technology and Information of the Ministry of Education
References (42)
Show source-provided references
1. Yun, W.-H., Lee, J., Lee, J.-H., and Kim, J. (July, January 28). Object recognition and pose estimation for modular manipulation system: Overview and initial results. Proceedings of the 2017 14th International Conference on Ubiquitous Robots and Ambient Intelligence (URAI), Jeju, Republic of Korea.10.1109/URAI.2017.7992711 ↗
2. Tremblay, J., To, T., Sundaralingam, B., Xiang, Y., Fox, D., and Birchfield, S. (2018). Deep Object Pose Estimation for Semantic Robotic Grasping of Household Objects. arXiv.
3. Chen, X., Ma, H., Wan, J., Li, B., and Xia, T. (2017, January 21–26). Multi-view 3D Object Detection Network for Autonomous Driving. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.691 ↗
4. Geiger, A., Lenz, P., and Urtasun, R. (2012, January 16–21). Are we ready for autonomous driving? The KITTI vision benchmark suite. Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA.10.1109/CVPR.2012.6248074 ↗
5. Pose Estimation for Augmented Reality: A Hands-On Survey10.1109/TVCG.2015.2513408 ↗
6. An Improved Method for Model-Based Training, Detection and Pose Estimation of Texture-Less 3D Objects in Occlusion Scenes10.1016/j.procir.2019.04.115 ↗
7. Lowe, D.G. (1999, January 20–27). Object recognition from local scale-invariant features. Proceedings of the Seventh IEEE International Conference on Computer Vision, Kerkyra, Greece.10.1109/ICCV.1999.790410 ↗
8. Amini, A., Periyasamy, A.S., and Behnke, S. (2021). T6D-Direct: Transformers for Multi-object 6D Pose Direct Regression. Pattern Recognition, Springer International Publishing.10.1007/978-3-030-92659-5_34 ↗
9. Di, Y., Manhardt, F., Wang, G., Ji, X., Navab, N., and Tombari, F. (2021, January 10–17). SO-Pose: Exploiting Self-Occlusion for Direct 6D Pose Estimation. Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada.10.1109/ICCV48922.2021.01217 ↗
10. Hodan, T., Barath, D., and Matas, J. (2020, January 13–19). EPOS: Estimating 6D Pose of Objects With Symmetries. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR42600.2020.01172 ↗
11. Li, Z., Wang, G., and Ji, X. (2019, January 15–20). CDPN: Coordinates-Based Disentangled Pose Network for Real-Time RGB-Based 6-DoF Object Pose Estimation. Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Long Beach, CA, USA.10.1109/ICCV.2019.00777 ↗
12. Wang, G., Manhardt, F., Tombari, F., and Ji, X. (2021, January 20–25). GDR-Net: Geometry-Guided Direct Regression Network for Monocular 6D Object Pose Estimation. Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.10.1109/CVPR46437.2021.01634 ↗
13. Amini, A., Selvam Periyasamy, A., and Behnke, S. (2023). YOLOPose: Transformer-Based Multi-object 6D Pose Estimation Using Keypoint Regression. Intelligent Autonomous Systems 17, Springer Nature.10.1007/978-3-031-22216-0_27 ↗
14. PVNet: Pixel-Wise Voting Network for 6DoF Object Pose Estimation10.1109/TPAMI.2020.3047388 ↗
15. He, Y., Sun, W., Huang, H., Liu, J., Fan, H., and Sun, J. (2020, January 13–19). PVN3D: A Deep Point-Wise 3D Keypoints Voting Network for 6DoF Pose Estimation. Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR42600.2020.01165 ↗
16. He, Y., Huang, H., Fan, H., Chen, Q., and Sun, J. (2021, January 20–25). FFB6D: A Full Flow Bidirectional Fusion Network for 6D Pose Estimation. Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.10.1109/CVPR46437.2021.00302 ↗
17. Wu, Y., Zand, M., Etemad, A., and Greenspan, M. (2022). Vote from the Center: 6 DoF Pose Estimation in RGB-D Images by Radial Keypoint Voting. Computer Vision—ECCV 2022, Springer Nature.10.1007/978-3-031-20080-9_20 ↗
18. Zhou, J., Chen, K., Xu, L., Dou, Q., and Qin, J. (2023, January 1–6). Deep Fusion Transformer Network with Weighted Vector-Wise Keypoints Voting for Robust 6D Object Pose Estimation. Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision (ICCV), Paris, France.10.1109/ICCV51070.2023.01284 ↗
19. Su, Y., Saleh, M., Fetzer, T., Rambach, J., Navab, N., Busam, B., Stricker, D., and Tombari, F. (2022, January 18–24). ZebraPose: Coarse to Fine Surface Encoding for 6DoF Object Pose Estimation. Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.00662 ↗
20. Wang, C., Xu, D., Zhu, Y., Martín-Martín, R., Lu, C., Fei-Fei, L., and Savarese, S. (2019, January 15–20). DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusion. Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00346 ↗
21. Mo, N., Gan, W., Yokoya, N., and Chen, S. (2022, January 18–24). ES6D: A Computation Efficient and Symmetry-Aware 6D Pose Regression Framework. Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.00660 ↗
22. Jiang, X., Li, D., Chen, H., Zheng, Y., Zhao, R., and Wu, L. (2022, January 18–24). Uni6D: A Unified CNN Framework without Projection Breakdown for 6D Pose Estimation. Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.01089 ↗
23. Sun, M., Zheng, Y., Bao, T., Chen, J., Jin, G., Wu, L., Zhao, R., and Jiang, X. (2022, January 28–30). Uni6Dv2: Noise Elimination for 6D Pose Estimation. Proceedings of the International Conference on Artificial Intelligence and Statistics, Virtual.
24. Chen, S., Li, X., Wang, Z., and Prisacariu, V.A. (2022). DFNet: Enhance Absolute Pose Regression with Direct Feature Matching. Computer Vision—ECCV 2022, Springer Nature.10.1007/978-3-031-20080-9_1 ↗
25. Coarse-to-Fine Multi-Scene Pose Regression With Transformers10.1109/TPAMI.2023.3310929 ↗
26. Sattler, T., Zhou, Q., Pollefeys, M., and Leal-Taixe, L. (2019, January 13–19). Understanding the Limitations of CNN-Based Absolute Camera Pose Regression. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR.2019.00342 ↗
27. Chen, W., Jia, X., Chang, H.J., Duan, J., and Leonardis, A. (2020, January 13–19). G2L-Net: Global to Local Network for Real-Time 6D Pose Estimation With Embedding Vector Features. Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR42600.2020.00429 ↗
28. Xu, J., Xiong, Z., and Bhattacharyya, S.P. (2023, January 17–24). PIDNet: A Real-Time Semantic Segmentation Network Inspired by PID Controllers. Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada.10.1109/CVPR52729.2023.01871 ↗
29. Liu, Z., Li, J., Shen, Z., Huang, G., Yan, S., and Zhang, C. (2017, January 22–29). Learning Efficient Convolutional Networks Through Network Slimming. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Venice, Italy.10.1109/ICCV.2017.298 ↗
30. Zhong, Y. (October, January 27). Intrinsic shape signatures: A shape descriptor for 3D object recognition. Proceedings of the 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops, Kyoto, Japan.10.1109/ICCVW.2009.5457637 ↗
31. Harris 3D: A robust extension of the Harris operator for interest point detection on 3D meshes10.1007/s00371-011-0610-y ↗
32. Comparison of 3D interest point detectors and descriptors for point cloud fusion
33. Feature extraction of point clouds based on region clustering segmentation10.1007/s11042-019-08512-1 ↗
34. A Novel Approach to the Extraction of Key Points From 3-D Rigid Point Cloud Using 2-D Images Transformation
35. Least-Squares Fitting of Two 3-D Point Sets10.1109/TPAMI.1987.4767965 ↗
36. Fischler, M.A., and Bolles, R.C. (1987). Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography. Readings in Computer Vision, Morgan Kaufmann.10.1016/B978-0-08-051581-6.50070-2 ↗
37. Mean shift: A robust approach toward feature space analysis10.1109/34.1000236 ↗
38. Efficient plane extraction using normal estimation and RANSAC from 3D point cloud10.1016/j.csi.2021.103608 ↗
39. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., and Fei-Fei, L. (2009, January 20–25). ImageNet: A large-scale hierarchical image database. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.10.1109/CVPR.2009.5206848 ↗
40. Kingma, D.P., and Ba, J. (2015, January 7–9). Adam: A Method for Stochastic Optimization. Proceedings of the 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA.
41. Hinterstoisser, S., Holzer, S., Cagniart, C., Ilic, S., Konolige, K., Navab, N., and Lepetit, V. (2011, January 6–13). Multimodal templates for real-time detection of texture-less objects in heavily cluttered scenes. Proceedings of the 2011 International Conference on Computer Vision, Barcelona, Spain.10.1109/ICCV.2011.6126326 ↗
42. Xiang, Y., Schmidt, T., Narayanan, V., and Fox, D. (2018, January 26–30). PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes. Proceedings of the Robotics: Science and Systems, Pittsburgh, PA, USA.10.15607/RSS.2018.XIV.019 ↗