RT-BMTR: A Bilateral Hybrid Backbone Network for Crop and Weed Detection in Complex Agricultural Scenarios
Source abstract
Abstract
For modern agricultural management, the accuracy of plant identification is crucial. However, the task becomes challenging because crops and weeds at early growth stages often exhibit similar color, leaf morphology, and texture in two-dimensional images captured under field conditions, despite their clear biological differences in terms of botanical species, root systems, and phenological characteristics. Furthermore, computing hardware in the field also has strict limits. Therefore, we developed the RT-BMTR network to handle these physical constraints. Within this architecture, image data is processed through a bilateral hybrid backbone named Bi-HMB. The DSFM captures small local details, and MambaVision understands the broader background information. Then, these features are fused by RepNCSPELAN4. We adopted this structure to reduce redundant calculations. Next, the model determines its bounding boxes using the Inner-ShapeIoU loss function. This geometric constraint improves the detection of small targets. When evaluated on the CropAndWeed dataset, our model achieved an average precision (AP) at IoU threshold 0.5 (AP50) of 68.1%, AP75 of 54.8%, and a mean AP averaged over IoU thresholds from 0.5 to 0.95 (AP50–95) of 50.9%. Detection precision recorded 26.5% for small objects, 44.7% for moderate ones, and with 59.3% for large objects. Rates for the first two categories saw enhancements of 16.2% and 4.6%. Overall, our modified model outperforms the original RT-DETR baseline. We also shrank the overall parameter count by 30.1%, alongside a 4.2% decrease in computational demand.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Smart Agriculture and AI · Advanced Neural Network Applications · Remote Sensing in Agriculture
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Electrical and Electronic Engineering, Hubei University of Technology, Wuhan 430068, China
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen 518055, China
Funding
Innovation and Entrepreneurship Training Program for College Students of Hubei Province
References (33)
Show source-provided references
1. Autonomous robotic weed control systems: A review10.1016/j.compag.2007.05.008 ↗
2. Weed management in 2050: Perspectives on the future of weed science10.1017/wsc.2017.78 ↗
3. Yang, Z., Zhang, W., Hu, C., Feng, Z., and Li, J. (2026). RDD-DETR Algorithm for Full-Scale Detection of Rice Diseases. Agriculture, 16.10.3390/agriculture16070799 ↗
4. Deep learning in agriculture: A survey10.1016/j.compag.2018.02.016 ↗
5. DeepWeeds: A multiclass weed species image dataset for deep learning10.1038/s41598-018-38343-3 ↗
6. Faster R-CNN: Towards real-time object detection with region proposal networks10.1109/TPAMI.2016.2577031 ↗
7. Zhang, X., Cui, J., Liu, H., Han, Y., Ai, H., Dong, C., Zhang, J., and Chu, Y. (2023). Weed Identification in Soybean Seedling Stage Based on Optimized Faster R-CNN Algorithm. Agriculture, 13.10.3390/agriculture13010175 ↗
8. Weed detection in soybean crops using ConvNets10.1016/j.compag.2017.10.027 ↗
9. Weed location and recognition based on UAV imaging and deep learning
10. Wang, C.-Y., Bochkovskiy, A., and Liao, H.-Y.M. (2021). Scaled-YOLOv4: Scaling cross stage partial network. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA, 20–25 June 2021, IEEE.10.1109/CVPR46437.2021.01283 ↗
11. A deep learning approach incorporating YOLO v5 and attention mechanisms for field real-time detection of the invasive weed Solanum rostratum Dunal seedlings10.1016/j.compag.2022.107194 ↗
12. Weed identification using deep learning and image processing in vegetable plantation10.1109/ACCESS.2021.3050296 ↗
13. Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and Jégou, H. (2021, January 18–24). Training data-efficient image transformers & distillation through attention. Proceedings of the International Conference on Machine Learning (ICML), Vienna, Austria.
14. Wang, W., Xie, E., Li, X., Fan, D.-P., Song, K., and Liang, D. (2021). Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada, 10–17 October 2021, IEEE.10.1109/ICCV48922.2021.00061 ↗
15. Zhu, X., Su, W., Lu, L., Li, B., Wang, X., and Dai, J. (2021, January 3–7). Deformable DETR: Deformable transformers for end-to-end object detection. Proceedings of the International Conference on Learning Representations (ICLR), Virtual Only Conference.
16. Reedha, R., Dericquebourg, E., Canals, R., and Hafiane, A. (2022). Transformer neural network for weed and crop classification of high resolution UAV images. Remote Sens., 14.10.3390/rs14030592 ↗
17. Zhao, Y., Lv, W., Xu, S., Wei, J., Wang, G., Dang, Q., Liu, Y., and Chen, J. (2024). DETRs beat YOLOs on real-time object detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 16–22 June 2024, IEEE.10.1109/CVPR52733.2024.01605 ↗
18. Steininger, D., Trondl, A., Croonen, G., Simon, J., and Widhalm, V. (2023). The CropAndWeed Dataset: A Multi-Modal Learning Approach for Efficient Crop and Weed Manipulation. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Waikoloa, HI, USA, 2–7 January 2023, IEEE.10.1109/WACV56688.2023.00372 ↗
19. Gu, A., and Dao, T. (2024, January 7–9). Mamba: Linear-time sequence modeling with selective state spaces. Proceedings of the First Conference on Language Modeling (COLM), Philadelphia, PA, USA.
20. Hatamizadeh, A., and Kautz, J. (2025). MambaVision: A hybrid Mamba-transformer vision backbone. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA, 15–20 June 2025, IEEE.10.1109/CVPR52734.2025.02352 ↗
21. Wang, C.Y., Liao, H.Y.M., Wu, Y.H., Chen, P.-Y., Hsieh, J.-W., and Yeh, I.-H. (2020). CSPNet: A new backbone that can enhance learning capability of CNN. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Seattle, WA, USA, 14–19 June 2020, IEEE.10.1109/CVPRW50498.2020.00203 ↗
22. Ding, X., Zhang, X., Ma, N., Han, J., Ding, G., and Sun, J. (2021). RepVGG: Making VGG-style ConvNets great again. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA, 20–25 June 2021, IEEE.10.1109/CVPR46437.2021.01352 ↗
23. Rezatofighi, H., Tsoi, N., Gwak, J., Sadeghian, A., Reid, I., and Savarese, S. (2019). Generalized intersection over union: A metric and a loss for bounding box regression. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 15–20 June 2019, IEEE.10.1109/CVPR.2019.00075 ↗
24. Zhang, H., Xu, C., and Zhang, S. (2023). Inner-IoU: More effective intersection over union metric with auxiliary bounding box. arXiv, Available online: https://arxiv.org/abs/2311.02877.
25. Zhang, H., and Zhang, S. (2023). Shape-IoU: More accurate metric considering bounding box shape and scale. arXiv, Available online: https://arxiv.org/abs/2312.17663.
26. Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C.L. (2014, January 6–12). Microsoft COCO: Common objects in context. Proceedings of the European Conference on Computer Vision (ECCV), Zurich, Switzerland.10.1007/978-3-319-10602-1_48 ↗
27. Jocher, G., Chaurasia, A., Qiu, J., Burhan, M., Stoken, A., Borovec, J., Kwon, Y., Michael, K., Fang, J., and Wong, C. (2026, May 08). Ultralytics YOLOv8. Available online: https://github.com/ultralytics/ultralytics.
28. Wang, C.Y., Yeh, I.H., and Liao, H.Y.M. (2024). YOLOv9: Learning what you want to learn using programmable gradient information. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 16–22 June 2024, IEEE.10.1007/978-3-031-72751-1_1 ↗
29. Wang, A., Chen, H., Liu, L., Chen, K., Lin, Z., Han, J., and Ding, G. (2024, January 10–15). YOLOv10: Real-time end-to-end object detection. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Vancouver, BC, Canada.
30. Ultralytics (2026, May 08). YOLO11. Available online: https://docs.ultralytics.com/models/yolo11/.
31. Peng, Y., Li, H., Wu, P., Zhang, Y., Sun, X., and Wu, F. (2025, January 24–28). D-FINE: Redefine Regression Task of DETRs as Fine-grained Distribution Refinement. Proceedings of the International Conference on Learning Representations (ICLR), Singapore.
32. Zhang, H., Li, F., Liu, S., Zhang, L., Su, H., Zhu, J., Ni, L.M., and Shum, H.-Y. (2023, January 1–5). DINO: DETR with improved de-noising anchor boxes for end-to-end object detection. Proceedings of the International Conference on Learning Representations (ICLR), Kigali, Rwanda.
33. Hou, X., Liu, M., Zhang, S., Wei, P., and Chen, B. (2024). Salience DETR: Enhancing detection transformer with hierarchical salience filtering refinement. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 17–21 June 2024, IEEE.10.1109/CVPR52733.2024.01664 ↗