A Review of Target Recognition Technology for Fruit Picking Robots: From Digital Image Processing to Deep Learning
Source abstract
Abstract
Machine vision technology has dramatically improved the efficiency, speed, and quality of fruit-picking robots in complex environments. Target recognition technology for fruit is an integral part of the recognition systems of picking robots. The traditional digital image processing technology is a recognition method based on hand-designed features, which makes it difficult to achieve better recognition as it results in dealing with the complex and changing orchard environment. Numerous pieces of literature have shown that extracting special features by training data with deep learning has significant advantages for fruit recognition in complex environments. In addition, to realize fully automated picking, reconstructing fruits in three dimensions is a necessary measure. In this paper, we systematically summarize the research work on target recognition techniques for picking robots in recent years, analyze the technical characteristics of different approaches, and conclude their development history. Finally, the challenges and future development trends of target recognition technology for picking robots are pointed out.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Smart Agriculture and AI · Date Palm Research Studies · Industrial Vision Systems and Defect Detection
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
College of Automotive Engineering, Foshan Polytechnic, Foshan 528100, China
College of Engineering, South China Agricultural University, Guangzhou 510642, China
Guangdong Industrial Robot Integration and Application Engineering Technology Research Center, Guangzhou Institute of Technology, Guangzhou 510540, China
Funding
Guangdong Province Modern Agricultural Industry Technology System Innovation Team Construction Project (Tea)—Tea Industry Innovation Team Facility and Mechanization Post Expert · The “14th Five-Year Plan” Guangdong Province Agricultural Science and Technology Innovation in the Ten main Directions “Unveiling the List of Hanging” Project—Lingnan Characteristic Fruit Intelligent Harvesting Technology · Guangdong Province Special fund for scientific and technological innovation strategy (“Climbing Plan” Special fund)
References (92)
Show source-provided references
1. Recognition and localization methods for vision-based fruit picking robots: A review10.3389/fpls.2020.00510 ↗
2. Zhou, Y., Tang, Y., Zou, X., Wu, M., Tang, W., Meng, F., Zhang, Y., and Kang, H. (2022). Adaptive Active Positioning of Camellia oleifera Fruit Picking Points: Classical Image Processing and YOLOv7 Fusion Algorithm. Appl. Sci., 12.10.3390/app122412959 ↗
3. Basic considerations in mechanizing citrus harvest10.13031/2013.39405 ↗
4. Pictorial pattern recognition applied to fruit harvesting10.13031/2013.35657 ↗
5. Date fruit classification for robotic harvesting in a natural environment using deep learning10.1109/ACCESS.2019.2936536 ↗
6. Deep learning in agriculture: A survey10.1016/j.compag.2018.02.016 ↗
7. Intelligent robots for fruit harvesting: Recent developments and future challenges10.1007/s11119-022-09913-3 ↗
8. Detection of fruit-bearing branches and localization of litchi clusters for vision-based harvesting robots10.1109/ACCESS.2020.3005386 ↗
9. Location of apples in trees using stereoscopic vision10.1016/j.compag.2015.01.010 ↗
10. A visual detection method for nighttime litchi fruits and fruiting stems10.1016/j.compag.2019.105192 ↗
11. Li, B., Wang, M., and Wang, N. (2010, January 20–23). Development of a real-time fruit recognition system for pineapple harvesting robots. Proceedings of the 2010 Pittsburgh, Pittsburgh, PA, USA.
12. AE—Automation and emerging technologies: A segmentation algorithm for the automatic recognition of Fuji apples at harvest10.1006/bioe.2002.0132 ↗
13. Using colour features of cv. ‘Gala’ apple fruits in an orchard in image processing to predict yield10.1007/s11119-012-9269-2 ↗
14. Fruit location in a partially occluded image10.13031/2013.30444 ↗
15. A machine vision algorithm combining adaptive segmentation and shape analysis for orange fruit detection
16. Zhang, J., Tan, T., and Ma, L. (2002, January 11–15). Invariant texture segmentation via circular Gabor filters. Proceedings of the 2002 International Conference on Pattern Recognition, Quebec City, QC, Canada.
17. Green citrus detection using ‘eigenfruit’, color and circular Gabor texture features under natural outdoor conditions10.1016/j.compag.2011.07.001 ↗
18. Texture-based fruit detection10.1007/s11119-014-9361-x ↗
19. Estimation of mango crop yield using image analysis–segmentation method10.1016/j.compag.2012.11.009 ↗
20. Estimating mango crop yield using image analysis using fruit at ‘stone hardening’stage and night time imaging10.1016/j.compag.2013.11.011 ↗
21. Girshick, R., Donahue, J., Darrell, T., and Malik, J. (2014, January 23–28). Rich feature hierarchies for accurate object detection and semantic segmentation. Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.81 ↗
22. Imagenet classification with deep convolutional neural networks10.1145/3065386 ↗
23. Selective search for object recognition10.1007/s11263-013-0620-5 ↗
24. Spatial pyramid pooling in deep convolutional networks for visual recognition10.1109/TPAMI.2015.2389824 ↗
25. Girshick, R. (2015, January 11–18). Fast r-cnn. Proceedings of the 2015 IEEE International Conference on Computer Vision, Santiago, Chile.10.1109/ICCV.2015.169 ↗
26. Recognition method of tomato key organs based on dual convolution Fast R-CNN
27. Faster r-cnn: Towards real-time object detection with region proposal networks10.1109/TPAMI.2016.2577031 ↗
28. Machine vision assessment of mango orchard flowering10.1016/j.compag.2018.06.040 ↗
29. Multi-class fruit-on-plant detection for apple in SNAP system using Faster R-CNN10.1016/j.compag.2020.105634 ↗
30. Kiwifruit detection in field images using Faster R-CNN with VGG1610.1016/j.ifacol.2019.12.500 ↗
31. Passion fruit detection and counting based on multiple scale faster R-CNN using RGB-D images10.1007/s11119-020-09709-3 ↗
32. Kiwifruit detection in field images using Faster R-CNN with ZFNet10.1016/j.ifacol.2018.08.059 ↗
33. Detection of maturity stages of coconuts in complex background using Faster R-CNN model10.1016/j.biosystemseng.2020.12.002 ↗
34. Sun, J., He, X., Ge, X., Wu, X., Shen, J., and Song, Y. (2018). Detection of key organs in tomato based on deep migration learning in a complex background. Agriculture, 8.10.20944/preprints201810.0524.v1 ↗
35. Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. (2016, January 27–30). You only look once: Unified, real-time object detection. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.91 ↗
36. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015, January 7–12). Going deeper with convolutions. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298594 ↗
37. Redmon, J., and Farhadi, A. (2017, January 21–26). YOLO9000: Better, faster, stronger. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.690 ↗
38. Ioffe, S., and Szegedy, C. (2015, January 6–11). Batch normalization: Accelerating deep network training by reducing internal covariate shift. Proceedings of the 32nd International Conference on Machine Learning, Lille, France.
39. Visual detection of green mangoes by an unmanned aerial vehicle in orchards based on a deep learning method10.1016/j.biosystemseng.2020.04.006 ↗
40. Immature mango detection based on improved YOLOv2
41. Redmon, J., and Farhadi, A. (2018). Yolov3: An incremental improvement. arXiv.
42. Lin, T.Y., Dollár, P., Girshick, R., He, K., Hariharan, B., and Belongie, S. (2017, January 21–26). Feature pyramid networks for object detection. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.106 ↗
43. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
44. Apple detection during different growth stages in orchards using the improved YOLO-V3 model10.1016/j.compag.2019.01.012 ↗
45. Liu, G., Nouaze, J.C., Touko Mbouembe, P.L., and Kim, J.H. (2020). YOLO-tomato: A robust algorithm for tomato detection based on YOLOv3. Sensors, 20.10.3390/s20072145 ↗
46. Fast and accurate detection of kiwifruit in orchard using improved YOLOv3-tiny model10.1007/s11119-020-09754-y ↗
47. Light-YOLOv3: Fast method for detecting green mangoes in complex scenes using picking robots10.1007/s10489-020-01818-w ↗
48. Bochkovskiy, A., Wang, C.Y., and Liao, H.Y.M. (2020). Yolov4: Optimal speed and accuracy of object detection. arXiv.
49. Improved mosaic: Algorithms for more complex images10.1088/1742-6596/1684/1/012094 ↗
50. Wang, C.Y., Liao, H.Y.M., Wu, Y.H., Chen, P.Y., Hsieh, J.W., and Yeh, I.H. (2020, January 14–19). CSPNet: A new backbone that can enhance learning capability of CNN. Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Seattle, WA, USA.10.1109/CVPRW50498.2020.00203 ↗
51. Liu, S., Qi, L., Qin, H., Shi, J., and Jia, J. (2018, January 18–23). Path aggregation network for instance segmentation. Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00913 ↗
52. Fast detection of banana bunches and stalks in the natural environment based on deep learning10.1016/j.compag.2022.106800 ↗
53. Research on tomato detection in natural environment based on RC-YOLOv410.1016/j.compag.2022.107029 ↗
54. Gai, R., Chen, N., and Yuan, H. (2021). A detection algorithm for cherry fruits based on the improved YOLO-v4 model. Neural Comput. Appl., 1–12.10.1007/s00521-021-06029-z ↗
55. Using channel pruning-based YOLO v4 deep learning algorithm for the real-time and accurate detection of apple flowers in natural environments10.1016/j.compag.2020.105742 ↗
56. Fruit detection and positioning technology for a Camellia oleifera C. Abel orchard based on improved YOLOv4-tiny model and binocular stereo vision10.1016/j.eswa.2022.118573 ↗
57. Real-time strawberry detection using deep neural networks on embedded system (rtsd-net): An edge AI application10.1016/j.compag.2021.106586 ↗
58. Qi, X., Dong, J., Lan, Y., and Zhu, H. (2022). Method for identifying litchi picking position based on YOLOv5 and PSPNet. Remote Sens., 14.10.3390/rs14092004 ↗
59. Citrus fruits maturity detection in natural environments based on convolutional neural networks and visual saliency map10.1007/s11119-022-09895-2 ↗
60. Apple stem/calyx real-time recognition using YOLO-v5 algorithm for fruit automatic loading system10.1016/j.postharvbio.2021.111808 ↗
61. Lyu, S., Li, R., Zhao, Y., Li, Z., Fan, R., and Liu, S. (2022). Green citrus detection and counting in orchards based on YOLOv5-CS and AI edge system. Sensors, 22.10.3390/s22020576 ↗
62. Yan, B., Fan, P., Lei, X., Liu, Z., and Yang, F. (2021). A real-time apple targets detection method for picking robot based on improved YOLOv5. Remote Sens., 13.10.3390/rs13091619 ↗
63. Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.Y., and Berg, A.C. (2016, January 11–14). Ssd: Single shot multibox detector. Proceedings of the Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands. Proceedings, Part I 14.10.1007/978-3-319-46448-0_2 ↗
64. General improved SSD model for picking object recognition of multiple fruits in natural environment
65. Lightweight object detection method for Lingwu long jujube images based on improved SSD
66. Li, Z., and Zhou, F. (2017). FSSD: Feature fusion single shot multibox detector. arXiv.
67. Litchi detection based on multiple feature enhancement and feature fusion SSD
68. Lin, T.Y., Goyal, P., Girshick, R., He, K., and Dollár, P. (2017, January 22–29). Focal loss for dense object detection. Proceedings of the 2017 IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.324 ↗
69. Green Apple Detection Based on Optimized FCOS in Orchards
70. Apple detection in complex orchard environment based on improved RetinaNet
71. Apple fruit recognition in complex orchard environment based on improved YOLOv3
72. Image recognition of Rosa roxburghii fruit by imoroved RetinaNet
73. Camellia oleifera Fruit Detection in Natural Scene Based on YOLO v5s
74. Long, J., Shelhamer, E., and Darrell, T. (2015, January 7–12). Fully convolutional networks for semantic segmentation. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298965 ↗
75. Lin, G., Tang, Y., Zou, X., Xiong, J., and Li, J. (2019). Guava detection and pose estimation using a low-cost RGB-D sensor in the field. Sensors, 19.10.3390/s19020428 ↗
76. Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., and Yuille, A.L. (2014). Semantic image segmentation with deep convolutional nets and fully connected crfs. arXiv.
77. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs10.1109/TPAMI.2017.2699184 ↗
78. Chen, L.C., Papandreou, G., Schroff, F., and Adam, H. (2017). Rethinking atrous convolution for semantic image segmentation. arXiv.
79. Chen, L.C., Zhu, Y., Papandreou, G., Schroff, F., and Adam, H. (2018, January 8–14). Encoder-decoder with atrous separable convolution for semantic image segmentation. Proceedings of the 2018 European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01234-2_49 ↗
80. Semantic segmentation of litchi branches using DeepLabV3+ model10.1109/ACCESS.2020.3021739 ↗
81. Segnet: A deep convolutional encoder-decoder architecture for image segmentation10.1109/TPAMI.2016.2644615 ↗
82. Apple tree trunk and branch segmentation for automatic trellis training using convolutional neural network based semantic segmentation10.1016/j.ifacol.2018.08.064 ↗
83. He, K., Gkioxari, G., Dollár, P., and Girshick, R. (2017, January 22–29). Mask r-cnn. Proceedings of the 2017 IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.322 ↗
84. Fruit detection for strawberry harvesting robot in non-structural environment based on Mask-RCNN10.1016/j.compag.2019.06.001 ↗
85. Detection and segmentation of overlapped fruits based on optimized mask R-CNN application in apple harvesting robot10.1016/j.compag.2020.105380 ↗
86. Fusion of Mask RCNN and attention mechanism for instance segmentation of apples under complex background10.1016/j.compag.2022.106864 ↗
87. Visual recognition of cherry tomatoes in plant factory based on improved deep instance segmentation10.1016/j.compag.2022.106991 ↗
88. Li, T., Fang, W., Zhao, G., Gao, F., Wu, Z., Li, R., Fu, L., and Dhupia, J. (2021). An improved binocular localization method for apple based on fruit detection using deep learning. Inf. Process. Agric.10.1016/j.inpa.2021.12.003 ↗
89. Localisation of litchi in an unstructured environment using binocular stereo vision10.1016/j.biosystemseng.2016.02.004 ↗
90. Automatic apple recognition based on the fusion of color and 3D feature for robotic fruit picking10.1016/j.compag.2017.09.019 ↗
91. Three-dimensional reconstruction of guava fruits and branches using instance segmentation and geometry analysis10.1016/j.compag.2021.106107 ↗
92. Li, T., Feng, Q., Qiu, Q., Xie, F., and Zhao, C. (2022). Occluded Apple Fruit Detection and localization with a frustum-based point-cloud-processing approach for robotic harvesting. Remote Sens., 14.10.3390/rs14030482 ↗