Image-Based Tactile Deformation Simulation and Pose Estimation for Robot Skill Learning
Source abstract
Abstract
The TacTip is a cost-effective, 3D-printed optical tactile sensor commonly used in deep learning and reinforcement learning for robotic manipulation. However, its specialized structure, which combines soft materials of varying hardnesses, makes it challenging to simulate the distribution of numerous printed markers on pins. This paper aims to create an interpretable, AI-applicable simulation of the deformation of TacTip under varying pressures and interactions with different objects, addressing the black-box nature of learning and simulation in haptic manipulation. The research focuses on simulating the TacTip sensor’s shape using a fully tunable, chain-based mathematical model, refined through comparisons with real-world measurements. We integrated the WRS system with our theoretical model to evaluate its effectiveness in object pose estimation. The results demonstrated that the prediction accuracy for all markers across a variety of contact scenarios exceeded 92%.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Robot Manipulation and Learning
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Bristol Robotics Laboratory, Bristol BS16 1QY, UK
Harada Laboratory, Graduate School of Engineering Science, Osaka University, Osaka 565-0871, Japan
College of Intelligent Systems Science and Engineering, Harbin Engineering University, Harbin 150001, China
Department of Computer Science, University of Liverpool, Liverpool L3 5TR, UK
Funding
Research Grants of The Royal Society · New Faculty Research Initiation Program Project
References (26)
Show source-provided references
1. Yuan, W., Dong, S., and Adelson, E.H. (2017). GelSight: High-resolution robot tactile sensors for estimating geometry and force. Sensors, 17.10.3390/s17122762 ↗
2. Photometric method for determining surface orientation from multiple images10.1117/12.7972479 ↗
3. The TacTip family: Soft optical tactile sensors with 3D-printed biomimetic morphologies10.1089/soro.2017.0052 ↗
4. Church, A., Lloyd, J., and Lepora, N.F. (2022, January 14–18). Tactile sim-to-real policy transfer via real-to-sim image translation. Proceedings of the Conference on Robot Learning, Auckland, New Zealand.
5. Lin, Y., Comi, M., Church, A., Zhang, D., and Lepora, N.F. (2023, January 1–5). Attention for Robot Touch: Tactile Saliency Prediction for Robust Sim-to-Real Tactile Control. Proceedings of the 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Detroit, MI, USA.10.1109/IROS55552.2023.10341888 ↗
6. DigiTac: A DIGIT-TacTip hybrid tactile sensor for comparing low-cost high-resolution robot touch10.1109/LRA.2022.3190641 ↗
7. Digit: A novel design for a low-cost compact high-resolution tactile sensor with application to in-hand manipulation10.1109/LRA.2020.2977257 ↗
8. In-Hand Pose Estimation Using Hand-Mounted RGB Cameras and Visuotactile Sensors10.1109/ACCESS.2023.3244552 ↗
9. 9dtact: A compact vision-based tactile sensor for accurate 3D shape reconstruction and generalizable 6D force estimation10.1109/LRA.2023.3339397 ↗
10. Lin, C., Lin, Z., Wang, S., and Xu, H. (June, January 29). Dtact: A vision-based tactile sensor that measures high-resolution 3D geometry directly from darkness. Proceedings of the 2023 IEEE International Conference on Robotics and Automation (ICRA), London, UK.10.1109/ICRA48891.2023.10160796 ↗
11. Lloyd, J., and Lepora, N. (2023). A pose and shear-based tactile robotic system for object tracking, surface following and object pushing. arXiv.
12. Ding, Z., Lepora, N.F., and Johns, E. (August, January 31). Sim-to-real transfer for optical tactile sensing. Proceedings of the 2020 IEEE International Conference on Robotics and Automation (ICRA), Paris, France.10.1109/ICRA40945.2020.9197512 ↗
13. Generation of GelSight tactile images for sim2real learning10.1109/LRA.2021.3063925 ↗
14. Tacto: A fast, flexible, and open-source simulator for high-resolution vision-based tactile sensors10.1109/LRA.2022.3146945 ↗
15. Isola, P., Zhu, J.-Y., Zhou, T., and Efros, A.A. (2017, January 21–26). Image-to-image translation with conditional adversarial networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.632 ↗
16. Tactile Gym 2.0: Sim-to-real deep reinforcement learning for comparing low-cost high-resolution robot touch10.1109/LRA.2022.3195195 ↗
17. Bi-touch: Bimanual tactile manipulation with sim-to-real deep reinforcement learning10.1109/LRA.2023.3295991 ↗
18. Touchsdf: A deepsdf approach for 3d shape reconstruction using vision-based tactile sensing10.1109/LRA.2024.3396054 ↗
19. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python10.1038/s41592-019-0686-2 ↗
20. A fast and elitist multiobjective genetic algorithm: NSGA-II10.1109/4235.996017 ↗
21. Single- and multipoint aerodynamic shape optimization using multifidelity models and manifold mapping10.2514/1.C035297 ↗
22. A Pareto optimal study for the multi-objective oil platform location problem with NSGA-II10.1016/j.petrol.2018.05.037 ↗
23. Planning grasps with suction cups and parallel grippers using superimposed segmentation of object meshes10.1109/TRO.2020.3014036 ↗
24. A regrasp planning component for object reorientation10.1007/s10514-018-9781-y ↗
25. Domain-Invariant Feature Learning via Margin and Structure Priors for Robotic Grasping10.1109/LRA.2024.3520437 ↗
26. Robotic grasp detection using structure prior attention and multiscale features10.1109/TSMC.2024.3446841 ↗