Visual Object Detection with DETR to Support Video-Diagnosis Using Conference Tools
Source abstract
Abstract
Real-time multilingual phrase detection from/during online video presentations—to support instant remote diagnostics—requires near real-time visual (textual) object detection and preprocessing for further analysis. Connecting remote specialists and sharing specific ideas is most effective using the native language. The main objective of this paper is to analyze and propose—through DEtection TRansformer (DETR) models, architectures, hyperparameters—recommendation, and specific procedures with simplified methods to achieve reasonable accuracy to support real-time textual object detection for further analysis. The development of real-time video conference translation based on artificial intelligence supported solutions has a relevant impact in the health sector, especially on clinical practice via better video consultation (VC) or remote diagnosis. The importance of this development was augmented by the COVID-19 pandemic. The challenge of this topic is connected to the variety of languages and dialects that the involved specialists speak and that usually needs human translator proxies which can be substituted by AI-enabled technological pipelines. The sensitivity of visual textual element localization is directly connected to complexity, quality, and the variety of collected training data sets. In this research, we investigated the DETR model with several variations. The research highlights the differences of the most prominent real-time object detectors: YOLO4, DETR, and Detectron2, and brings AI-based novelty to collaborative solutions combined with OCR. The performance of the procedures was evaluated through two research phases: a 248/512 (Phase1/Phase2) record train data set, with a 55/110 set of validated data instances for 7/10 application categories and 3/3 object categories, using the same object categories for annotation. The achieved score breaks the expected values in terms of visual text detection scope, giving high detection accuracy of textual data, the mean average precision ranging from 0.4 to 0.65.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Multimodal Machine Learning Applications · Advanced Image and Video Retrieval Techniques · Handwritten Text Recognition Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Electrical Engineering and Information Technology, George Emil Palade University of Medicine, Pharmacy, Science, and Technology of Targu Mures, Str. Nicolae Iorga, Nr. 1, 540088 Targu Mures, Romania
Department of Physiotherapy, University of Malaga, 29071 Malaga, Spain
Biomedical Research Institute of Malaga (IBIMA), 29590 Malaga, Spain
Computational Intelligence Research Group, Sapientia Hungarian University of Transylvania, 540485 Targu Mures, Romania
Physiological Controls Research Center, Óbuda University, 1034 Budapest, Hungary
Faculty of Health Science, School of Clinical Science, Queensland University Technology, Brisbane 4000, Australia
Legal and Forensic Medicine Area, Department of Human Anatomy, Legal Medicine and History of Science, Faculty of Medicine, University of Malaga, 29071 Malaga, Spain
Funding
Sapientia Foundation—Institute for Scientific Research
References (43)
Show source-provided references
1. Ozili, P.K., and Arun, T. (2020). Spillover of COVID-19: Impact on the Global Economy. SSRN.10.2139/ssrn.3562570 ↗
2. Conducting remote medical asylum evaluations in the United States during COVID-19: Clinicians’ perspectives on acceptability, challenges and opportunities10.1016/j.jflm.2021.102255 ↗
3. Object detection with deep learning: A review10.1109/TNNLS.2018.2876865 ↗
4. Salient object detection based on global to local visual search guidance10.1016/j.image.2021.116618 ↗
5. Diagnostic accuracy in remote expert consultation using standard video-conference technology10.1007/BF00240711 ↗
6. Historical review of OCR research and development10.1109/5.156468 ↗
7. Smith, R., Antonova, D., and Lee, D.-S. (2009, January 25). Adapting the Tesseract open source OCR engine for multilingual OCR. Proceedings of the International Workshop on Multilingual OCR, Barcelona, Spain.10.1145/1577802.1577804 ↗
8. Gu, J.T., Neubig, G., Cho, K.H., and Li, V.O.K. (2017, January 3–7). Learning to translate in real-time with neural machine translation. Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, Valencia, Spain.10.18653/v1/E17-1099 ↗
9. A CNN-BiLSTM based hybrid model for Indian language identification10.1016/j.apacoust.2021.108274 ↗
10. Ding, L.A., Wu, D., and Tao, D.C. (2021, January 7–11). Improving neural machine translation by bidirectional training. Proceedings of the Conference on Empirical Methods in Natural Language Processing, Punta Cana, Dominican Republic.10.18653/v1/2021.emnlp-main.263 ↗
11. Qi, J.W., and Peng, Y.X. (2018, January 13–19). Cross-modal bidirectional translation via reinforcement learning. Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence (IJCAI-18), Stockholm, Sweden.10.24963/ijcai.2018/365 ↗
12. Towards modeling user behavior in interactions mediated through an automated bidirectional speech translation system10.1016/j.csl.2009.04.008 ↗
13. Ataman, D., Firat, O., Di Gangi, M.A., Federico, F., and Birch, A. (2019, January 4). On the importance of word boundaries in character-level neural machine translation. Proceedings of the 3rd Workshop on Neural Generation and Translation, Hong Kong.10.18653/v1/D19-5619 ↗
14. Integrating machine translation into translator training: Towards ‘Human Translator Competence’?10.29228/transLogos.11 ↗
15. Bizzoni, Y., Juzek, T.S., España-Bonet, C., Chowdhury, K.D., van Genabith, J., and Teich, E. (2020, January 9–10). How human is machine translationese? Comparing human and machine translations of text and speech. Proceedings of the 17th International Conference on Spoken Language Translation, Online.10.18653/v1/2020.iwslt-1.34 ↗
16. Briones, J. (2022, April 28). Object Detection with Transformers. Available online: https://medium.com/swlh/object-detection-with-transformers-437217a3d62e.
17. Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., and Zagoruyko, S. (2020, January 23–28). End-to-end object detection with transformers. Proceedings of the European Conference on Computer Vision, Glasgow, UK.10.1007/978-3-030-58452-8_13 ↗
18. Wang, J.F., Song, L., Li, Z.M., Sun, H.B., Sun, J., and Zheng, N.N. (2021, January 19–25). End-to-end object detection with fully convolutional network. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.01559 ↗
19. Wieniawska, H.W. (2022, April 28). What Exactly is Happening Inside the Transformer. Available online: https://medium.com/swlh/what-exactly-is-happening-inside-the-transformer-b7f713d7aded.
20. Understanding of object detection based on CNN family and YOLO10.1088/1742-6596/1004/1/012029 ↗
21. Redmon, J., and Farhadi, A. (2018). YOLOv3: An incremental improvement. arXiv.
22. (2022, April 28). TensorFlow 2 YOLOv4. Available online: https://bit.ly/3kgtaff.
23. (2022, April 28). DETR: End-to-End Object Detection with Transformers. Available online: https://github.com/facebookresearch/detr.
24. Lin, T.Y., Goyal, P., Girshick, R., He, K.M., and Dollár, P. (2017, January 21–26). Focal loss for dense object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/ICCV.2017.324 ↗
25. He, Y.H., Zhu, C.C., Wang, J.R., Savvides, M., and Zhang, X.Y. (2019, January 16–17). Bounding box regression with uncertainty for accurate object detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00300 ↗
26. DetNet: Design backbone for object detection10.1007/978-3-030-01240-3_21 ↗
27. Shao, S., Li, Z.M., Zhang, T.Y., Peng, C., Yu, G., Zhang, X.Y., Li, J., and Sun, J. (November, January 27). Objects365: A large-scale, high-quality dataset for object detection. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Korea.
28. Dynamic R-CNN: Towards high quality object detection via dynamic training10.1007/978-3-030-58555-6_16 ↗
29. Sensitivity analysis of different components of transfer function for detection and classification of type, location and extent of transformer faults10.1016/j.measurement.2021.110292 ↗
30. (2022, April 28). GitHub—George Kalitsios: Face Detection Using DETR End-to-End Object Detection with Transformers. Available online: https://github.com/george-kalitsios/Face-Detection-using-DETR-End-to-End-Object-Detection-with-Transformers.
31. Ma, T.L., Mao, M.Y., Zheng, H.H., Gao, P., Wang, X.D., Han, S.M., Ding, E.R., Zhang, B.C., and Doermann, D. (2021). Oriented object detection with transformer. arXiv.
32. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017, January 4–9). Attention is all you need. Proceedings of the 31st Conference on Neural Information Processing Systems, Long Beach, CA, USA.
33. (2022, April 28). VoTT—Visual Object Tagging Tool 2020. Available online: https://github.com/microsoft/VoTT.
34. (2022, April 28). LabelImg. Available online: https://github.com/tzutalin/labelImg.
35. Girshick, R. (2015, January 7–13). Fast r-cnn. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.10.1109/ICCV.2015.169 ↗
36. Nath, U., and Kushagra, S. (2020). Better Together: Resnet-50 accuracy with 13× fewer parameters and at 3× speed. arXiv.
37. Gao, P., Zheng, M., Wang, X., Dai, J., and Li, H. (2021, January 11–17). Fast convergence of DETR with spatially modulated co-attention. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.10.1109/ICCV48922.2021.00360 ↗
38. Tan, R.J. (2022, April 28). Breaking Down Mean Average Precision (mAP)—Another Metric for Your Data Science Toolkit. Available online: https://towardsdatascience.com/breaking-down-mean-average-precision-map-ae462f623a52#1a59.
39. Bochkovskiy, A., Wang, C.Y., and Liao, H.Y.M. (2020). YOLOv4: Optimal speed and accuracy of object detection. arXiv.
40. Honda, H. (2022, April 28). Digging into Detectron 2—Part 1. Available online: https://medium.com/@hirotoschwert/digging-into-detectron-2-47b2e794fabd.
41. (2022, April 28). FAIR’s Research Platform for Object Detection Research, Implementing Popular Algorithms Like MASK R-CNN and RetinaNet. Available online: https://github.com/facebookresearch/Detectron.
42. Zheng, M.H., Gao, P., Zhang, R.R., Li, K.C., Wang, X.G., Li, H.S., and Dong, H. (2021, January 22–25). End-to-end object detection with adaptive clustering transformer. Proceedings of the 32nd British Machine Vision Conference, Online.
43. Dai, Z.G., Cai, B.L., Lin, Y.G., and Chen, J.Y. (2021, January 20–25). UP-DETR: Unsupervised pre-training for object detection with transformers. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.00165 ↗