A Meal-Assistance Robot System for Asian Food and Its Food Acquisition Point Estimation and User Interface Based on Face Recognition
Source abstract
Abstract
Various meal-assistance robot (MAR) systems are being studied, and several products have already been commercialized to alleviate the imbalance between the rising demand and diminishing supply of meal care services. However, several challenges remain. First, most of these services can serve limited types of western food using a predefined route. Additionally, their spoon or fork sometimes makes it difficult to acquire Asian food that is easy to handle with chopsticks. In addition, their limited user interface, requiring physical contact, makes it difficult for people with severe disabilities to use MARs alone. This paper proposes an MAR system that is suitable for the diet of Asians who use chopsticks. This system uses Mask R-CNN to recognize the food area on the plate and estimates the acquisition points for each side dish. The points become target points for robot motion planning. Depending on which food the user selects, the robot uses chopsticks or a spoon to obtain the food. In addition, a non-contact user interface based on face recognition was developed for users with difficulty physically manipulating the interface. This interface can be operated on the user’s Android OS tablet without the need for a separate dedicated display. A series of experiments verified the proposed system’s effectiveness and feasibility.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Social Robot Interaction and HRI · IoT-based Smart Home Systems · Robotics and Automated Systems
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
AI Robot R&D Department, Korea Institute of Industrial Technology (KITECH), Ansan 15588, Republic of Korea
Mechanical Engineering, Sungkyunkwan University, Suwon 16419, Republic of Korea
Korea Electronics Technology Institute (KETI), Bucheon 14502, Republic of Korea
Funding
Ministry of Trade, Industry & Energy (MOTIE, Korea) · Korea Institute of Industrial Technology (KITECH)
References (73)
Show source-provided references
1. World Health Organization (2022, December 08). Available online: https://www.who.int/news-room/fact-sheets/detail/disability-and-health/.
2. Naotunna, I., Perera, C.J., Sandaruwan, C., Gopura, R., and Lalitharatne, T.D. (2015, January 11–13). Meal assistance robots: A review on current status, challenges and future directions. Proceedings of the 2015 IEEE/SICE International Symposium on System Integration (SII), Nagoya, Japan.10.1109/SII.2015.7404980 ↗
3. Towards robotic feeding: Role of haptics in fork-based food manipulation10.1109/LRA.2019.2894592 ↗
4. An electric wheelchair mounted robotic arm—A survey of potential users10.3109/03091909009083051 ↗
5. Devices for assisting manipulation: A summary of user task priorities10.1109/86.340872 ↗
6. Climbing control of autonomous mobile robot with estimation of wheel slip and wheel-ground contact angle10.1007/s12206-022-0142-6 ↗
7. Pico, N., Park, S.H., Yi, J.S., and Moon, H. (2022). Six-Wheel Robot Design Methodology and Emergency Control to Prevent the Robot from Falling down the Stairs. Appl. Sci., 12.10.3390/app12094403 ↗
8. Obi (2022, December 08). Available online: https://meetobi.com/meet-obi/.
9. (2022, December 08). Meal Buddy. Available online: https://www.performancehealth.com/meal-buddy-systems.
10. (2022, December 08). Specializing in Assistive Eating and Assistive Drinking Equipment for Individuals with Disabilities. Available online: https://mealtimepartners.com/.
11. (2022, December 08). Neater. Available online: https://www.neater.co.uk/.
12. (2022, December 08). Bestic Eating Assistive Device. Available online: https://at-aust.org/items/13566.
13. (2022, December 08). Automation My Spoon through Image Processing. Available online: https://www.secom.co.jp/isl/e2/research/mw/report04/.
14. (2022, December 08). CareMeal. Available online: http://www.ntrobot.net/myboard/product.
15. Song, W.K., and Kim, J. (2012). Robotic Systems-Applications, Control and Programming, Intech Open.
16. Song, K., and Cha, Y. (2020). Chopstick Robot Driven by X-shaped Soft Actuator. Actuators, 9.10.3390/act9020032 ↗
17. Oka, T., Solis, J., Lindborg, A.L., Matsuura, D., Sugahara, Y., and Takeda, Y. (2020). Kineto-Elasto-Static design of underactuated chopstick-type gripper mechanism for meal-assistance robot. Robotics, 9.10.3390/robotics9030050 ↗
18. Koshizaki, T., and Masuda, R. (2010, January 7–9). Control of a meal assistance robot capable of using chopsticks. Proceedings of the ISR 2010 (41st International Symposium on Robotics) and ROBOTIK 2010 (6th German Conference on Robotics), Munich, Germany.
19. Yamazaki, A., and Masuda, R. (2012, January 21–22). Autonomous foods handling by chopsticks for meal assistant robot. Proceedings of the ROBOTIK 2012, 7th German Conference on Robotics, Munich, Germany.
20. Ohshima, Y., Kobayashi, Y., Kaneko, T., Yamashita, A., and Asama, H. (2013, January 15–17). Meal support system with spoon using laser range finder and manipulator. Proceedings of the 2013 IEEE Workshop on Robot Vision (WORV), Clearwater Beach, FL, USA.10.1109/WORV.2013.6521918 ↗
21. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
22. Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K.Q. (2017, January 21–26). Densely connected convolutional networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.243 ↗
23. Lin, T.Y., Dollár, P., Girshick, R., He, K., Hariharan, B., and Belongie, S. (2017, January 21–26). Feature pyramid networks for object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.106 ↗
24. Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. (July, January 26). You only look once: Unified, real-time object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.
25. Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.Y., and Berg, A.C. (2016, January 8–16). Ssd: Single shot multibox detector. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.10.1007/978-3-319-46448-0_2 ↗
26. Lin, T.Y., Goyal, P., Girshick, R., He, K., and Dollár, P. (2017, January 22–29). Focal loss for dense object detection. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.324 ↗
27. Li, Z., Peng, C., Yu, G., Zhang, X., Deng, Y., and Sun, J. (2018). Detnet: A backbone network for object detection. arXiv.10.1007/978-3-030-01240-3_21 ↗
28. Segnet: A deep convolutional encoder-decoder architecture for image segmentation10.1109/TPAMI.2016.2644615 ↗
29. Jégou, S., Drozdzal, M., Vazquez, D., Romero, A., and Bengio, Y. (2017, January 21–26). The one hundred layers tiramisu: Fully convolutional densenets for semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Honolulu, HI, USA.10.1109/CVPRW.2017.156 ↗
30. He, K., Gkioxari, G., Dollár, P., and Girshick, R. (2017, January 22–29). Mask r-cnn. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.322 ↗
31. Dvornik, N., Shmelkov, K., Mairal, J., and Schmid, C. (2017, January 22–29). Blitznet: A real-time deep network for scene understanding. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.447 ↗
32. Yanai, K., and Kawano, Y. (2022, January 18–22). Food image recognition using deep convolutional network with pre-training and fine-tuning. Proceedings of the 2015 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), Turin, Italy.10.1109/ICMEW.2015.7169816 ↗
33. Singla, A., Yuan, L., and Ebrahimi, T. (2016, January 16). Food/non-food image classification and food categorization using pre-trained googlenet model. Proceedings of the 2nd International Workshop on Multimedia Assisted Dietary Management, Amsterdam, The Netherlands.10.1145/2986035.2986039 ↗
34. Hassannejad, H., Matrella, G., Ciampolini, P., De Munari, I., Mordonini, M., and Cagnoni, S. (2016, January 16). Food image recognition using very deep convolutional networks. Proceedings of the 2nd International Workshop on Multimedia Assisted Dietary Management, Amsterdam, The Netherlands.10.1145/2986035.2986042 ↗
35. Solving the production transportation problem via a deterministic annealing neural network method10.1016/j.amc.2021.126518 ↗
36. An approximation algorithm for graph partitioning via deterministic annealing neural network10.1016/j.neunet.2019.05.010 ↗
37. Gallenberger, D., Bhattacharjee, T., Kim, Y., and Srinivasa, S.S. (2019, January 11–14). Transfer depends on acquisition: Analyzing manipulation strategies for robotic feeding. Proceedings of the 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI), Daegu, Korea.10.1109/HRI.2019.8673309 ↗
38. Ren, S., He, K., Girshick, R., and Sun, J. (2015). Faster r-cnn: Towards real-time object detection with region proposal networks. Adv. Neural Inf. Process. Syst., 28.
39. Feng, R., Kim, Y., Lee, G., Gordon, E.K., Schmittle, M., Kumar, S., Bhattacharjee, T., and Srinivasa, S.S. (2019, January 6–10). Robot-Assisted Feeding: Generalizing Skewering Strategies Across Food Items on a Plate. Proceedings of the International Symposium of Robotics Research, Hanoi, Vietnam.
40. Gordon, E.K., Meng, X., Bhattacharjee, T., Barnes, M., and Srinivasa, S.S. (January, January 24). Adaptive robot-assisted feeding: An online learning framework for acquiring previously unseen food items. Proceedings of the 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Las Vegas, NV, USA.10.1109/IROS45743.2020.9341359 ↗
41. Gordon, E.K., Roychowdhury, S., Bhattacharjee, T., Jamieson, K., and Srinivasa, S.S. (5, January 30). Leveraging Post Hoc Context for Faster Learning in Bandit Settings with Applications in Robot-Assisted Feeding. Proceedings of the 2021 IEEE International Conference on Robotics and Automation (ICRA), Xi’an, China.10.1109/ICRA48506.2021.9561520 ↗
42. Review of interfaces for industrial human-robot interaction10.1007/s43154-020-00005-6 ↗
43. Mahmud, S., Lin, X., and Kim, J.H. (2020, January 6–8). Interface for human machine interaction for assistant devices: A review. Proceedings of the 2020 10th Annual Computing and Communication Workshop and Conference (CCWC), Las Vegas, NV, USA.10.1109/CCWC47524.2020.9031244 ↗
44. Porcheron, M., Fischer, J.E., Reeves, S., and Sharples, S. (2018, January 21–26). Voice interfaces in everyday life. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, New York, NY, USA.10.1145/3173574.3174214 ↗
45. Suzuki, R., Ogino, K., Nobuaki, K., Kogure, K., and Tanaka, K. (2013, January 19–21). Development of meal support system with voice input interface for upper limb disabilities. Proceedings of the 2013 IEEE 8th Conference on Industrial Electronics and Applications (ICIEA), Melbourne, Australia.10.1109/ICIEA.2013.6566460 ↗
46. A systematic review of Augmented Reality interfaces for collaborative industrial robots10.1016/j.cie.2020.106806 ↗
47. Review on existing VR/AR solutions in human–robot collaboration10.1016/j.procir.2020.05.259 ↗
48. Chamola, V., Vineet, A., Nayyar, A., and Hossain, E. (2020). Brain-computer interface-based humanoid control: A review. Sensors, 20.10.3390/s20133620 ↗
49. Perera, C.J., Naotunna, I., Sadaruwan, C., Gopura, R.A.R.C., and Lalitharatne, T.D. (2016, January 9–12). SSVEP based BMI for a meal assistance robot. Proceedings of the 2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC), Budapest, Hungary.10.1109/SMC.2016.7844580 ↗
50. Perera, C.J., Lalitharatne, T.D., and Kiguchi, K. (June, January 29). EEG-controlled meal assistance robot with camera-based automatic mouth position tracking and mouth open detection. Proceedings of the 2017 IEEE international conference on robotics and automation (ICRA), Singapore.10.1109/ICRA.2017.7989208 ↗
51. Ha, J., Park, S., Im, C.H., and Kim, L. (2021). A hybrid brain–computer interface for real-life meal-assist robot control. Sensors, 21.10.3390/s21134578 ↗
52. Meal assistance system operated by electromyogram (EMG) signals: Movement onset detection with adaptive threshold10.1007/s12555-010-0226-4 ↗
53. Bhattacharjee, T., Gordon, E.K., Scalise, R., Cabrera, M.E., Caspi, A., Cakmak, M., and Srinivasa, S.S. (2020, January 23–26). Is more autonomy always better? Exploring preferences of users with mobility impairments in robot-assisted feeding. Proceedings of the 2020 15th ACM/IEEE International Conference on Human-Robot Interaction (HRI), Cambridge, UK.10.1145/3319502.3374818 ↗
54. Design of Meal-Assisting Manipulator AI via Open-Mouth Target Detection
55. Meal Assistance Robot Operated by Detecting Voluntary Closing eye
56. Robot-assisted feeding: A technical application that combines learning from demonstration and visual interaction10.3233/THC-202527 ↗
57. Huang, X., Wang, L., Fan, X., Zhao, P., and Ji, K. (2022, January 1–3). Facial Gesture Controled Low-Cost Meal Assistance Manipulator System with Real-Time Food Detection. Proceedings of the International Conference on Intelligent Robotics and Applications, Harbin, China.10.1007/978-3-031-13822-5_20 ↗
58. Visual and gyroscope sensor for head movement controller system on meal-assistance application10.29252/mjee.14.3.4 ↗
59. Lopes, P., Lavoie, R., Faldu, R., Aquino, N., Barron, J., Kante, M., and Magfory, B. Icraft-eye-controlled robotic feeding arm technology. Tech. Rep., 2012.
60. Cuong, N.H., and Hoang, H.T. (2010, January 7–10). Eye-gaze detection with a single WebCAM based on geometry features extraction. Proceedings of the 2010 11th International Conference on Control Automation Robotics & Vision, Singapore.10.1109/ICARCV.2010.5707319 ↗
61. TabletGaze: Dataset and analysis for unconstrained appearance-based gaze estimation in mobile tablets10.1007/s00138-017-0852-4 ↗
62. (2022, December 08). WebCam Eye-Tracking Accuracy. Available online: https://gazerecorder.com/webcam-eye-tracking-accuracy/.
63. Imagenet classification with deep convolutional neural networks10.1145/3065386 ↗
64. George, A., and Routray, A. (2016, January 12–15). Real-time eye gaze direction classification using convolutional neural network. Proceedings of the 2016 International Conference on Signal Processing and Communications (SPCOM), Bangalore, India.10.1109/SPCOM.2016.7746701 ↗
65. A Novel Integrated Spoon-chopsticks Mechanism for a Meal Assistant Robotic System10.1007/s12555-021-0520-3 ↗
66. Song, H., Jung, B.J., Kim, T.K., Cho, C.N., Jeong, H.S., and Hwang, J.H. (2020, January 29–31). Development of Flexible Control System for the Meal Assistant Robot. Proceedings of the KSME Conference, Daejeon, Republic of Korea.
67. Development of Meal Assistance Robot for Generalization of Robot Care Service Using Deep Learning for the User of Meal Assistant Robot
68. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., and Fei-Fei, L. (2009, January 20–25). Imagenet: A large-scale hierarchical image database. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.10.1109/CVPR.2009.5206848 ↗
69. Meal Intention Recognition System based on Gaze Direction Estimation using Deep Learning for the User of Meal Assistant Robot10.5302/J.ICROS.2021.21.0019 ↗
70. Bochkovskiy, A., Wang, C.Y., and Liao, H.Y.M. (2020). Yolov4: Optimal speed and accuracy of object detection. arXiv.
71. Vrânceanu, R., Florea, C., Florea, L., and Vertan, C. (2013, January 27–29). NLP EAC recognition by component separation in the eye region. Proceedings of the International Conference on Computer Analysis of Images and Patterns, York, UK.10.1007/978-3-642-40246-3_28 ↗
72. Florea, L., Florea, C., Vrânceanu, R., and Vertan, C. (2013, January 9–13). Can Your Eyes Tell Me How You Think? A Gaze Directed Estimation of the Mental Activity. Proceedings of the BMVC, Bristol, UK.10.5244/C.27.60 ↗
73. (2022, December 08). TensorFlow. Available online: https://www.tensorflow.org/lite/.