A Review of Convolutional Neural Network Applied to Fruit Image Processing
Source abstract
Abstract
Agriculture has always been an important economic and social sector for humans. Fruit production is especially essential, with a great demand from all households. Therefore, the use of innovative technologies is of vital importance for the agri-food sector. Currently artificial intelligence is one very important technological tool widely used in modern society. Particularly, Deep Learning (DL) has several applications due to its ability to learn robust representations from images. Convolutional Neural Networks (CNN) is the main DL architecture for image classification. Based on the great attention that CNNs have had in the last years, we present a review of the use of CNN applied to different automatic processing tasks of fruit images: classification, quality control, and detection. We observe that in the last two years (2019–2020), the use of CNN for fruit recognition has greatly increased obtaining excellent results, either by using new models or with pre-trained networks for transfer learning. It is worth noting that different types of images are used in datasets according to the task performed. Besides, this article presents the fundamentals, tools, and two examples of the use of CNNs for fruit sorting and quality control.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Smart Agriculture and AI · Spectroscopy and Chemometric Analyses · Remote Sensing in Agriculture
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Laboratory of Technological Research in Pattern Recognition (LITRP), Universidad Católica del Maule, 3480112 Talca, Chile
Department of Agricultural Science, Universidad Católica del Maule, 3480112 Talca, Chile
Department of Economy and Administration, Universidad Católica del Maule, 3480112 Talca, Chile
Funding
Regional Government of Maule, Chile
References (100+)
Show source-provided references
1. Abdullahi, H.S., Sheriff, R., and Mahieddine, F. (2017, January 12–13). Convolution neural network in precision agriculture for plant image recognition and classification. Proceedings of the IEEE 2017 Seventh International Conference on Innovative Computing Technology (Intech), Porto, Portugal.10.1109/INTECH.2017.8102436 ↗
2. Annabel, L.S.P., Annapoorani, T., and Deepalakshmi, P. (2019, January 4–6). Machine Learning for Plant Leaf Disease Detection and Classification–A Review. Proceedings of the IEEE 2019 International Conference on Communication and Signal Processing (ICCSP), Chennai, India.
3. Agarwal, M., Kaliyar, R.K., Singal, G., and Gupta, S.K. (2019, January 18). FCNN-LDA: A Faster Convolution Neural Network model for Leaf Disease identification on Apple’s leaf dataset. Proceedings of the IEEE 2019 12th International Conference on Information & Communication Technology and System (ICTS), Surabaya, Indonesia.10.1109/ICTS.2019.8850964 ↗
4. Flexible system of multiple RGB-D sensors for measuring and classifying fruits in agri-food Industry10.1016/j.compag.2017.05.014 ↗
5. Automatic fruit and vegetable classification from images10.1016/j.compag.2009.09.002 ↗
6. Capizzi, G., Sciuto, G.L., Napoli, C., Tramontana, E., and Woźniak, M. (2015, January 13–16). Automatic classification of fruit defects based on co-occurrence matrix and neural networks. Proceedings of the IEEE 2015 Federated Conference on Computer Science and Information Systems (FedCSIS), Lodz, Poland.10.15439/2015F258 ↗
7. Rachmawati, E., Supriana, I., and Khodra, M.L. (2017, January 19–21). Toward a new approach in fruit recognition using hybrid RGBD features and fruit hierarchy property. Proceedings of the 2017 IEEE 4th International Conference on Electrical Engineering, Computer Science and Informatics (EECSI), Yogyakarta, Indonesia.10.1109/EECSI.2017.8239110 ↗
8. Automatic apple recognition based on the fusion of color and 3D feature for robotic fruit picking10.1016/j.compag.2017.09.019 ↗
9. Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning, MIT Press.
10. Coppin, B. (2004). Artificial Intelligence Illuminated, Jones & Bartlett Learning.
11. Machine learning: Trends, perspectives, and prospects10.1126/science.aaa8415 ↗
12. Deep Learning10.1038/nature14539 ↗
13. Artificial intelligence, machine learning, automation, robotics, future of work and future of humanity: A review and research agenda10.4018/JDM.2019010104 ↗
14. Some studies in machine learning using the game of checkers10.1147/rd.441.0206 ↗
15. A survey of deep neural network architectures and their applications10.1016/j.neucom.2016.12.038 ↗
16. Gewali, U.B., Monteiro, S.T., and Saber, E. (2018). Machine learning based hyperspectral image analysis: A survey. arXiv.
17. Femling, F., Olsson, A., and Alonso-Fernandez, F. (2018, January 26–29). Fruit and Vegetable Identification Using Machine Learning for Retail Applications. Proceedings of the IEEE 2018 14th International Conference on Signal-Image Technology & Internet-Based Systems (SITIS), Las Palmas de Gran Canaria, Spain.10.1109/SITIS.2018.00013 ↗
18. Application of extreme learning machine method for time series analysis
19. Gradient-based learning applied to document recognition10.1109/5.726791 ↗
20. Deep learning for visual understanding: A review10.1016/j.neucom.2015.09.116 ↗
21. Fleet, D., Pajdla, T., Schiele, B., and Tuytelaars, T. (2014). Visualizing and Understanding Convolutional Networks. Computer Vision—ECCV 2014, Springer International Publishing.10.1007/978-3-319-10578-9 ↗
22. Krizhevsky, A., Sutskever, I., and Hinton, G.E. (2012). ImageNet Classification with Deep Convolutional Neural Networks. Proceedings of the 25th International Conference on Neural Information Processing Systems—Volume 1, Curran Associates Inc.. NIPS’12.
23. Imagenet large scale visual recognition challenge10.1007/s11263-015-0816-y ↗
24. Lu, Y. (2019). Food image recognition by using convolutional neural networks (CNNs). arXiv.
25. Image based fruit category classification by 13-layer deep convolutional neural network and data augmentation10.1007/s11042-017-5243-3 ↗
26. Hyperspectral fruit and vegetable classification using convolutional neural networks10.1016/j.compag.2019.04.019 ↗
27. Counting apples and oranges with deep learning: A data-driven approach10.1109/LRA.2017.2651944 ↗
28. Bargoti, S., and Underwood, J. (June, January 29). Deep fruit detection in orchards. Proceedings of the 2017 IEEE International Conference on Robotics and Automation (ICRA), Marina Bay Sands Singapore, Singapore.10.1109/ICRA.2017.7989417 ↗
29. Liu, F., Snetkov, L., and Lima, D. (2017, January 29–30). Summary on fruit identification methods: A literature review. Proceedings of the 2017 3rd International Conference on Economics, Social Science, Arts, Education and Management Engineering (ESSAEME 2017), Huhhot, China.10.2991/essaeme-17.2017.338 ↗
30. Fruit classification using computer vision and feedforward neural network10.1016/j.jfoodeng.2014.07.001 ↗
31. Fruit classification by biogeography-based optimization and feedforward neural network10.1111/exsy.12146 ↗
32. Fruit classification by wavelet-entropy and feedforward neural network trained by fitness-scaled chaotic ABC and biogeography-based optimization10.3390/e17085711 ↗
33. Machine Vision based Fruit Classification and Grading-A Review
34. Deep learning for smart agriculture: Concepts, tools, applications, and opportunities
35. Bhargava, A., and Bansal, A. (2018). Fruits and vegetables quality evaluation using computer vision: A review. J. King Saud Unive. Comput. Inf. Sci., Available online: https://doi.org/10.1016/j.jksuci.2018.06.002.10.1016/j.jksuci.2018.06.002 ↗
36. Sa, I., Ge, Z., Dayoub, F., Upcroft, B., Perez, T., and McCool, C. (2016). Deepfruits: A fruit detection system using deep neural networks. Sensors, 16.10.3390/s16081222 ↗
37. A comprehensive review of fruit and vegetable classification techniques10.1016/j.imavis.2018.09.016 ↗
38. Optical non-destructive techniques for small berry fruits: A review
39. Simonyan, K., and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv.
40. A deep convolutional neural network with new training methods for bearing fault diagnosis under noisy environment and different working load10.1016/j.ymssp.2017.06.022 ↗
41. Cascio, D., Taormina, V., and Raso, G. (2019). Deep Convolutional Neural Network for HEp-2 Fluorescence Intensity Classification. Appl. Sci., 9.10.3390/app9030408 ↗
42. LeCun, Y., Boser, B.E., Denker, J.S., Henderson, D., Howard, R.E., Hubbard, W.E., and Jackel, L.D. (1990). Handwritten digit recognition with a back-propagation network. Advances in Neural Information Processing Systems, Morgan Kaufmann.
43. LeCun, Y., Kavukcuoglu, K., and Farabet, C. (June, January 30). Convolutional networks and applications in vision. Proceedings of the IEEE 2010 IEEE International Symposium on Circuits and Systems, Paris, France.10.1109/ISCAS.2010.5537907 ↗
44. Dumoulin, V., and Visin, F. (2016). A guide to convolution arithmetic for deep learning. arXiv.
45. Convolutional neural networks: An overview and application in radiology10.1007/s13244-018-0639-9 ↗
46. Nair, V., and Hinton, G.E. (2010, January 21–24). Rectified linear units improve restricted boltzmann machines. Proceedings of the 27th International Conference On Machine Learning (ICML-10), Haifa, Israel.
47. Recent advances in convolutional neural networks10.1016/j.patcog.2017.10.013 ↗
48. Scherer, D., Müller, A., and Behnke, S. (2010). Evaluation of pooling operations in convolutional architectures for object recognition. International Conference on Artificial Neural Networks, Springer.10.1007/978-3-642-15825-4_10 ↗
49. Lee, C.Y., Gallagher, P.W., and Tu, Z. (2016). Generalizing pooling functions in convolutional neural networks: Mixed, gated, and tree. Artif. Intell. Stat., 464–472.
50. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015, January 7–12). Going deeper with convolutions. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298594 ↗
51. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
52. Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., and Darrell, T. (2014). Caffe: Convolutional Architecture for Fast Feature Embedding. arXiv.10.1145/2647868.2654889 ↗
53. Katarzyna, R., and Paweł, M. (2019). A Vision-Based Method Utilizing Deep Convolutional Neural Networks for Fruit Variety Classification in Uncertainty Conditions of Retail Sales. Appl. Sci., 9.10.3390/app9193971 ↗
54. Sakib, S., Ashrafi, Z., and Siddique, M.A.B. (2019). Implementation of Fruits Recognition Classifier using Convolutional Neural Network Algorithm for Observation of Accuracies for Various Hidden Layers. arXiv.
55. Fruit recognition from images using deep learning10.2478/ausi-2018-0002 ↗
56. High performance vegetable classification from images based on alexnet deep learning model
57. Hussain, I., He, Q., and Chen, Z. (2018). Automatic fruit recognition based on dcnn for commercial source trace system. Int. J. Comput. Sci. Appl. IJCSA, 8.10.5121/ijcsa.2018.8301 ↗
58. Lu, S., Lu, Z., Aok, S., and Graham, L. (2018, January 19–21). Fruit classification based on six layer convolutional neural network. Proceedings of the 2018 IEEE 23rd International Conference on Digital Signal Processing (DSP), Shanghai, China.10.1109/ICDSP.2018.8631562 ↗
59. Patino-Saucedo, A., Rostro-Gonzalez, H., and Conradt, J. (2018). Tropical Fruits Classification Using an AlexNet-Type Convolutional Neural Network and Image Augmentation. International Conference on Neural Information Processing, Springer.10.1007/978-3-030-04212-7_32 ↗
60. Wang, S.H., and Chen, Y. (2018). Fruit category classification via an eight-layer convolutional neural network with parametric rectified linear unit and dropout technique. Multim. Tools Appl., 1–17.
61. Zeng, G. (2017, January 3–5). Fruit and vegetables classification system using image saliency and convolutional neural network. Proceedings of the 2017 IEEE 3rd Information Technology and Mechatronics Engineering Conference (ITOEC), Chongquing, China.10.1109/ITOEC.2017.8122370 ↗
62. Hou, S., Feng, Y., and Wang, Z. (2017, January 22–29). Vegfru: A domain-specific dataset for fine-grained visual categorization. Proceedings of the 2017 IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.66 ↗
63. Zhang, W., Zhao, D., Gong, W., Li, Z., Lu, Q., and Yang, S. (2015). Food image recognition with convolutional neural networks. 2015 IEEE 12th Intl Conf on Ubiquitous Intelligence and Computing and 2015 IEEE 12th Intl Conf on Autonomic and Trusted Computing and 2015 IEEE 15th Intl Conf on Scalable Computing and Communications and Its Associated Workshops (UIC-ATC-ScalCom), IEEE.10.1109/UIC-ATC-ScalCom-CBDCom-IoP.2015.139 ↗
64. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., and Li, F.-F. (2009, January 20–25). Imagenet: A large-scale hierarchical image database. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.10.1109/CVPR.2009.5206848 ↗
65. Classification of fruits using computer vision and a multiclass support vector machine10.3390/s120912489 ↗
66. Wang, S., Lu, Z., Yang, J., Zhang, Y., Liu, J., Wei, L., Chen, S., Phillips, P., and Dong, Z. (2016). Fractional Fourier entropy increases the recognition rate of fruit type detection. BMC Plant Biol., 16.
67. Lu, Z., and Li, Y. (2017). A fruit sensing and classification system by fractional fourier entropy and improved hybrid genetic algorithm. Proceedings of the 5th International Conference on Industrial Application Engineering (IIAE), Institute of Industrial Applications Engineers.10.12792/iciae2017.053 ↗
68. An effective model based on Haar wavelet entropy and genetic algorithm for fruit identification
69. Introducing new shape features for classification of cucumber fruit based on image processing technique and artificial neural networks10.1111/jfpe.12558 ↗
70. Oltean, M. (2018). Fruits 360 dataset. Mendeley Data.
71. Rocha, A., Hauagge, D.C., Wainer, J., and Goldenstein, S. (2008). Automatic produce classification from images using color, texture and appearance cues. 2008 XXI Brazilian Symposium on Computer Graphics and Image Processing, IEEE.10.1109/SIBGRAPI.2008.9 ↗
72. Matsuda, Y., Hoashi, H., and Yanai, K. (2012, January 9–13). Recognition of Multiple-Food Images by Detecting Candidate Regions. Proceedings of the 2012 IEEE International Conference on Multimedia and Expo (ICME), Melbourne, Australia.10.1109/ICME.2012.157 ↗
73. Detection of apple defect using laser-induced light backscattering imaging and convolutional neural network10.1016/j.compeleceng.2019.106454 ↗
74. Classification of sour lemons based on apparent defects using stochastic pooling mechanism in deep convolutional neural networks10.1016/j.scienta.2019.109133 ↗
75. Automated phenotyping of epicuticular waxes of grapevine berries using light separation and convolutional neural networks10.1016/j.compag.2018.11.012 ↗
76. Munasingha, L.V., Gunasinghe, H.N., and Dhanapala, W.W.G.D.S. (2019, January 29–30). Identification of Papaya Fruit Diseases using Deep Learning Approach. Proceedings of the 4th International Conference on Advances in Computing and Technology (ICACT2019), Kelaniya, Sri Lanka.
77. Deep Features Based Approach for Fruit Disease Detection and Classification
78. Tran, T.T., Choi, J.W., Le, T.T.H., and Kim, J.W. (2019). A Comparative Study of Deep CNN in Forecasting and Classifying the Macronutrient Deficiencies on Development of Tomato Plant. Appl. Sci., 9.10.3390/app9081601 ↗
79. Evaluation of Deep Convolutional Neural Network Architectures for Strawberry Quality Inspection10.14419/ijet.v7i4.40.24080 ↗
80. Wang, Z., Hu, M., and Zhai, G. (2018). Application of deep learning architectures for accurate and rapid detection of internal mechanical damage of blueberry using hyperspectral transmittance data. Sensors, 18.10.3390/s18041126 ↗
81. Deep indicator for fine-grained classification of banana’s ripening stages10.1186/s13640-018-0284-8 ↗
82. Cen, H., He, Y., and Lu, R. (2016). Hyperspectral imaging-based surface and internal defects detection of cucumber via stacked sparse auto-encoder and convolutional neural network. 2016 ASABE Annual International Meeting, American Society of Agricultural and Biological Engineers.
83. Intelligent alerting for fruit-melon lesion image based on momentum deep learning10.1007/s11042-015-2940-7 ↗
84. Robotic kiwifruit harvesting using machine vision, convolutional neural networks, and robotic arms10.1016/j.biosystemseng.2019.03.007 ↗
85. Grape detection, segmentation, and tracking using deep neural networks and three-dimensional association10.1016/j.compag.2020.105247 ↗
86. Fruit detection for strawberry harvesting robot in non-structural environment based on Mask-RCNN10.1016/j.compag.2019.06.001 ↗
87. Deep Orange: Mask R-CNN based Orange Detection and Segmentation; 6th IFAC Conference on Sensing, Control and Automation Technologies for Agriculture AGRICONTROL 201910.1016/j.ifacol.2019.12.499 ↗
88. Improved kiwifruit detection using pre-trained VGG16 with RGB and NIR information fusion10.1109/ACCESS.2019.2962513 ↗
89. Instance Segmentation and Localization of Strawberries in Farm Conditions for Automatic Fruit Harvesting; 6th IFAC Conference on Sensing, Control and Automation Technologies for Agriculture AGRICONTROL 201910.1016/j.ifacol.2019.12.537 ↗
90. Date fruit classification for robotic harvesting in a natural environment using deep learning10.1109/ACCESS.2019.2936536 ↗
91. Towards Active Robotic Vision in Agriculture: A Deep Learning Approach to Visual Servoing in Occluded and Unstructured Protected Cropping Environments10.1016/j.ifacol.2019.12.508 ↗
92. Lin, G., Tang, Y., Zou, X., Xiong, J., and Li, J. (2019). Guava detection and pose estimation using a low-cost RGB-D sensor in the field. Sensors, 19.10.3390/s19020428 ↗
93. Detecting greenhouse strawberries (mature and immature), using deep convolutional neural network10.1016/j.eaef.2018.03.001 ↗
94. Rahnemoonfar, M., and Sheppard, C. (2017). Deep Count: Fruit Counting Based on Deep Simulated Learning. Sensors, 17.10.3390/s17040905 ↗
95. Image segmentation for fruit detection and yield estimation in apple orchards10.1002/rob.21699 ↗
96. Stein, M., Bargoti, S., and Underwood, J. (2016). Image based mango fruit detection, localisation and yield estimation using multiple view geometry. Sensors, 16.10.3390/s16111915 ↗
97. Detection of passion fruits and maturity classification using Red-Green-Blue Depth images10.1016/j.biosystemseng.2018.09.004 ↗
98. Convolutional neural network based on an extreme learning machine for image classification10.1016/j.neucom.2018.12.080 ↗
99. Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G.S., Davis, A., Dean, J., and Devin, M. (2016). Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv.
100. Al-Rfou, R., Alain, G., Almahairi, A., Angermueller, C., Bahdanau, D., Ballas, N., Bastien, F., Bayer, J., Belikov, A., and Belopolsky, A. (2016). Theano: A Python framework for fast computation of mathematical expressions. arXiv.