Neural-Based Detection and Segmentation of Articulated Objects for Robotic Interaction in a Household Environment
Source abstract
Abstract
Robots operating in household environments should detect and estimate the properties of articulated objects to efficiently perform tasks given by human operators. This paper presents the design and implementation of a system for estimating a point-cloud-based model of a scene, enhanced with information about articulated objects, based on a single RGB-D image. This article describes the neural method used to detect handles, detect and extract fronts, detect rotational joints, and build a point cloud model. It compares various architectures of neural networks to detect handles, the fronts of drawers and cabinets, and estimate rotational joints. In the end, the results are merged to build a 3D model of articulated objects in the environment.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Neural Network Applications · Industrial Vision Systems and Defect Detection · Robotics and Sensor-Based Localization
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Institute of Robotics and Machine Intelligence, Poznan University of Technology, Ul. Piotrowo 3A, 60-965 Poznan, Poland
Funding
National Science Centre, Poland
References (30)
Show source-provided references
1. Hausman, K., Niekum, S., Osentoski, S., and Sukhatme, G.S. (2015, January 26–30). Active articulation model estimation through interactive perception. Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), Seattle, WA, USA.10.1109/ICRA.2015.7139655 ↗
2. Ma, L., Meng, J., Liu, S., Chen, W., Xu, J., and Chen, R. (2023). Sim2Real2: Actively Building Explicit Physics Model for Precise Articulated Object Manipulation. arXiv.10.1109/ICRA48891.2023.10160370 ↗
3. Staszak, R., Molska, M., Mlodzikowski, K., Ataman, J., and Belter, D. (2020, January 8–11). Kinematic Structures Estimation on the RGB-D Images. Proceedings of the 25th IEEE International Conference on Emerging Technologies and Factory Automation (ETFA), Vienna, Austria.10.1109/ETFA46521.2020.9211967 ↗
4. SSD: Single Shot MultiBox Detector
5. The RBO dataset of articulated objects and interactions10.1177/0278364919844314 ↗
6. Desingh, K., Lu, S., Opipari, A., and Jenkins, O.C. (2019, January 20–24). Factored Pose Estimation of Articulated Objects using Efficient Nonparametric Belief Propagation. Proceedings of the 2019 International Conference on Robotics and Automation (ICRA), Montreal, QC, Canada.10.1109/ICRA.2019.8793973 ↗
7. Młodzikowski, K., and Belter, D. (2022, January 11–13). CNN-based Joint State Estimation During Robotic Interaction with Articulated Objects. Proceedings of the 17th International Conference on Control, Automation, Robotics and Vision (ICARCV), Singapore.10.1109/ICARCV57592.2022.10004277 ↗
8. Hsu, C.-C., Jiang, Z., and Zhu, Y. (2023). Ditto in the House: Building Articulation Models of Indoor Scenes through Interactive Perception. arXiv.10.1109/ICRA48891.2023.10161431 ↗
9. Jiang, Z., Hsu, C.-C., and Zhu, Y. (2022, January 18–24). Ditto: Building digital twins of articulated objects from interaction. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00553 ↗
10. Girshick, R., Donahue, J., Darrell, T., and Malik, J. (2014, January 23–28). Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation. Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.81 ↗
11. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks10.1109/TPAMI.2016.2577031 ↗
12. Redmon, J., and Farhadi, A. (2018). YOLOv3: An Incremental Improvement. arXiv.
13. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
14. Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Hartwig, A. (2017). MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. arXiv.
15. Tan, M., and Le, Q.V. (2019, January 9–15). EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. Proceedings of the 36th International Conference on Machine Learning, ICML, Long Beach, CA, USA.
16. Tan, M., and Le, Q.V. (2021, January 18–24). EfficientNetV2: Smaller Models and Faster Training. Proceedings of the 38th International Conference on Machine Learning, Online.
17. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2021, January 3–7). An Image is Worth 16×16 Words: Transformers for Image Recognition at Scale. Proceedings of the 9th International Conference on Learning Representations ICLR, Vienna, Austria.
18. Bao, H., Dong, L., Piao, S., and Wei, F. (2022, January 25–29). BEiT: BERT Pre-Training of Image Transformers. Proceedings of the 10th International Conference on Learning Representations, ICLR, Online.
19. Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and Jegou, H. (2021, January 18–24). Training data-efficient image transformers & distillation through attention. Proceedings of the 38th International Conference on Machine Learning, Online.
20. Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. (2021, January 10–17). Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, BC, Canada.10.1109/ICCV48922.2021.00986 ↗
21. Evolution of Image Segmentation using Deep Convolutional Neural Network: A Survey10.1016/j.knosys.2020.106062 ↗
22. SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation10.1109/TPAMI.2016.2644615 ↗
23. Navab, N., Hornegger, J., Wells, W.M., and Frangi, A. (2015). Medical Image Computing and Computer-Assisted Intervention—MICCAI 2015, Springer International Publishing.10.1007/978-3-319-24571-3 ↗
24. DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs10.1109/TPAMI.2017.2699184 ↗
25. Wu, H., Zhang, J., Huang, K., Liang, K., and Yizhou, Y. (2019). FastFCN: Rethinking Dilated Convolution in the Backbone for Semantic Segmentation. arXiv.
26. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. (2016, January 24–30). Rethinking the Inception Architecture for Computer Vision. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.308 ↗
27. Zhou, X., Wang, D., and Krähenbühl, P. (2019). Objects as Points. arXiv.
28. Focal Loss for Dense Object Detection10.1109/TPAMI.2018.2858826 ↗
29. Loshchilov, I., and Hutter, F. (2016). SGDR: Stochastic Gradient Descent with Restarts. arXiv.
30. Lin, T.-Y., Maire, M., Belongie, S.J., Bourdev, L.D., Girshick, R.B., Hays, J., and Zitnick, C.L. Proceedings of the Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, 6–12 September 2014, Springer International Publishing.