Green Sweet Pepper Fruit and Peduncle Detection Using Mask R-CNN in Greenhouses

In this paper, a mask region-based convolutional neural network (Mask R-CNN) is used to improve the performance of machine vision in the challenging task of detecting peduncles and fruits of green sweet peppers (Capsicum annuum L.) in greenhouses. One of the most complicated stages of the sweet pepper harvesting process is to achieve a precise cut of the peduncle or stem because this type of specialty crop cannot be grabbed and pulled by the fruit since the integrity and value of the product are compromised. Therefore, accurate peduncle detection becomes vital for the autonomous harvesting of sweet peppers. ResNet-101 combined with the feature pyramid network (FPN) architecture (ResNet-101 + FPN) is adopted as the backbone network for feature extraction and object representation enhancement at multiple scales. Mask images of fruits and peduncles are generated, focused on green sweet pepper, which is the most complex color variety due to its resemblance to the background. In addition to bounding boxes, Mask R-CNN provides binary masks as a result of instance segmentation, which would help improve the localization process in 3D space, the next phase of the autonomous harvesting process of sweet peppers, since it isolates the pixels belonging to the object and demarcates its boundaries. The prediction results of 1148 fruits on 100 test images showed a precision rate of 84.53%. The prediction results of 265 peduncles showed a precision rate of 71.78%. The mean average precision rate with an intersection over union at 50 percent (mAP@IoU=50) for model-wide instance segmentation was 72.64%. The average detection time for sweet pepper fruit and peduncle using high-resolution images was 1.18 s. The experimental results show that the proposed implementation manages to segment the peduncle and fruit of the green sweet pepper in real-time in an unmodified production environment under occlusion, overlap, and light variation conditions with effectiveness not previously reported for simultaneous 2D detection models of peduncles and fruits of green sweet pepper.