journal article·2024-11Open access

A Multifunctional Tactile Sensory System for Robotic Intelligent Identification and Manipulation Perception

Yue Jiang, Lin Fan, Xilong Sun, Zehe Luo, Herong Wang, Rucong Lai, Jie Wang, Qiyang Gan, Ning Li, Jindong Tian

Advanced Science· Volume 11· Issue 41

Abstract

Abstract Humans recognize and manipulate objects relying on the multidimensional force features captured by the tactile sense of skin during the manipulation. Since the current sensors integrated in robots cannot support the robots to sense the multiple interaction states between manipulator and objects, achieving human‐like perception and analytical capabilities remains a major challenge for service robots. Prompted by the tactile perception involved in robots performing complex tasks, a multimodal tactile sensory system is presented to provide in situ simultaneous sensing for robots when approaching, touching, and manipulating objects. The system comprises a capacitive sensor owning the high sensitivity of 1.11E‐2 pF mm−1, a triboelectricity nanogenerator with the fast response speed of 30 ms, and a pressure sensor array capable of 3D force detection. By Combining transfer learning models, which fuses multimodal tactile information to achieve high‐precision (up to 95%) recognition of the multi‐featured targets such as random hardness and texture information under random sampling conditions, including random grasp force and velocity. This sensory system is expected to enhance the intelligent recognition and behavior‐planning capabilities of autonomous robots when performing complex tasks in undefined surrounding environments.
Source: crossref · Quality: baseline

Research context

OpenAlex topics: Advanced Sensor and Energy Harvesting Materials · Tactile and Sensory Interactions · Muscle activation and electromyography studies

SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.

Affiliations

Key Laboratory of Optoelectronic Devices and Systems of Ministry of Education and Guangdong Province College of Physics and Optoelectronic Engineering Shenzhen University Shenzhen 518060 China

College of Computer Science and Software Engineering Shenzhen University Shenzhen 518060 China

Guangdong Laboratory of Artificial Intelligence and Digital Economy (Shenzhen) Shenzhen University Shenzhen 518132 China

Institute of Applied Physics and Materials Engineering University of Macau Macao 999078 China

Funding

National Natural Science Foundation of China · Natural Science Foundation of Shenzhen Municipality

References (62)

Show source-provided references
1. 10.1126/scirobotics.abf431710.1126/scirobotics.abf4317 ↗
2. 10.1126/scirobotics.adk674310.1126/scirobotics.adk6743 ↗
3. 10.1038/s41467-020-19059-310.1038/s41467-020-19059-3 ↗
4. 10.1016/j.apmt.2024.10212210.1016/j.apmt.2024.102122 ↗
5. 10.1177/1729880622109597410.1177/17298806221095974 ↗
6. 10.1109/LRA.2022.315522510.1109/LRA.2022.3155225 ↗
7. 10.1021/acsnano.2c0643210.1021/acsnano.2c06432 ↗
8. 10.1038/s41467-019-14054-910.1038/s41467-019-14054-9 ↗
9. 10.1038/s41467-020-19531-010.1038/s41467-020-19531-0 ↗
10. 10.1002/adma.20190213310.1002/adma.201902133 ↗
11. 10.1126/sciadv.abb908310.1126/sciadv.abb9083 ↗
12. 10.1002/adma.20210085910.1002/adma.202100859 ↗
13. 10.1038/s41467-020-17298-y10.1038/s41467-020-17298-y ↗
14. 10.1126/scirobotics.aau691410.1126/scirobotics.aau6914 ↗
15. 10.1002/adma.20200597010.1002/adma.202005970 ↗
16. a)W.Huang P.Abbeel D.Pathak I.Mordatch inProc. 39th Int. Conf. Machine Learning2022 164 9118;
17. 10.1109/ACCESS.2023.332937010.1109/ACCESS.2023.3329370 ↗
18. Kroemer O. · J. Mach. Learn. Res. · 2021 · 22 · 1
19. 10.1038/s42256-021-00439-310.1038/s42256-021-00439-3 ↗
20. 10.1038/s41928-020-0422-z10.1038/s41928-020-0422-z ↗
21. 10.1126/scirobotics.abl775510.1126/scirobotics.abl7755 ↗
22. 10.1038/s41562-021-01194-610.1038/s41562-021-01194-6 ↗
23. 10.1038/s41586-019-1234-z10.1038/s41586-019-1234-z ↗
24. 10.1073/pnas.201098911710.1073/pnas.2010989117 ↗
25. 10.1016/j.inffus.2023.10184710.1016/j.inffus.2023.101847 ↗
26. 10.1109/TIE.2022.317063110.1109/TIE.2022.3170631 ↗
27. 10.1016/j.cej.2022.14089010.1016/j.cej.2022.140890 ↗
28. 10.1109/TRO.2020.300323010.1109/TRO.2020.3003230 ↗
29. 10.1109/LRA.2023.329525510.1109/LRA.2023.3295255 ↗
30. 10.1038/s41928-023-01115-710.1038/s41928-023-01115-7 ↗
31. c)L.Ouyang J.Wu X.Jiang D.Almeida C.Wainwright P.Mishkin C.Zhang S.Agarwal K.Slama A.Ray in36th Conf. Neural Informat. Proces. Systems2022 35 27730;
32. 10.1016/j.matt.2022.02.01610.1016/j.matt.2022.02.016 ↗
33. 10.1038/s41928-021-00585-x10.1038/s41928-021-00585-x ↗
34. 10.1002/adfm.20230624910.1002/adfm.202306249 ↗
35. 10.1021/acsnano.0c0901510.1021/acsnano.0c09015 ↗
36. 10.1021/acsami.2c1486310.1021/acsami.2c14863 ↗
37. 10.1007/s40820-020-00525-y10.1007/s40820-020-00525-y ↗
38. 10.1016/j.carbpol.2022.12010910.1016/j.carbpol.2022.120109 ↗
39. 10.1002/adfm.20190589810.1002/adfm.201905898 ↗
40. 10.1007/s10570-021-03815-910.1007/s10570-021-03815-9 ↗
41. 10.1016/j.compositesa.2021.10670210.1016/j.compositesa.2021.106702 ↗
42. 10.1002/adfm.20221156310.1002/adfm.202211563 ↗
43. 10.1016/j.carbpol.2020.11758010.1016/j.carbpol.2020.117580 ↗
44. 10.1016/j.cej.2020.12479610.1016/j.cej.2020.124796 ↗
45. 10.1021/acsnano.8b0099710.1021/acsnano.8b00997 ↗
46. 10.1016/j.cej.2020.12643810.1016/j.cej.2020.126438 ↗
47. 10.1016/j.cej.2023.14230110.1016/j.cej.2023.142301 ↗
48. 10.1016/j.nanoen.2022.10815110.1016/j.nanoen.2022.108151 ↗
49. 10.1016/j.mtphys.2024.10133110.1016/j.mtphys.2024.101331 ↗
50. 10.1021/acsami.3c1403610.1021/acsami.3c14036 ↗
51. 10.1016/j.nanoen.2024.10927810.1016/j.nanoen.2024.109278 ↗
52. 10.1016/j.nanoen.2023.10817810.1016/j.nanoen.2023.108178 ↗
53. 10.1021/acsnano.8b0155710.1021/acsnano.8b01557 ↗
54. 10.1016/j.nanoen.2022.10732410.1016/j.nanoen.2022.107324 ↗
55. 10.1038/nmat338010.1038/nmat3380 ↗
56. 10.1002/adfm.20230979210.1002/adfm.202309792 ↗
57. 10.1038/s41467-019-09461-x10.1038/s41467-019-09461-x ↗
58. 10.1002/inf2.1236010.1002/inf2.12360 ↗
59. 10.1021/acsnano.1c1039610.1021/acsnano.1c10396 ↗
60. 10.1126/sciadv.abq252110.1126/sciadv.abq2521 ↗
61. 10.1016/j.nanoen.2023.10841810.1016/j.nanoen.2023.108418 ↗
62. 10.1021/acsami.3c0659710.1021/acsami.3c06597 ↗