Video Processing from a Virtual Unmanned Aerial Vehicle: Comparing Two Approaches to Using OpenCV in Unity
Source abstract
Abstract
Virtual reality (VR) simulators enable the evaluation of engineering systems and robotic solutions in safe and realistic environments. To do so, VR simulators must run algorithms in real time to accurately recreate the expected behaviour of real-life processes. This work was aimed at determining a suitable configuration for processing images taken from a virtual unmanned aerial vehicle developed in Unity using OpenCV. To this end, it was focused on comparing two approaches to integrate video processing in order to avoid potential pitfalls such as delays and bottlenecks. The first approach used a dynamic link library (DLL) programmed in C++, and the second an external module programmed in Python. The native DLL ran internally on the same Unity thread, as opposed to the Python module that ran in parallel to the main process and communicated with Unity through the Message Queue Telemetry Transport (MQTT) protocol. Pre-transmission processing, data transmission and video processing were evaluated for a pair of typical image-processing tasks like colour and face detection. The analysis confirmed that running the Python module in parallel does not overload the main Unity thread and achieves better performance than the C++ plugin in real-time simulation.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Video Surveillance and Tracking Methods · UAV Applications and Optimization · Advanced Vision and Imaging
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Instituto de Investigación en Informática de Albacete, 02071 Albacete, Spain
Departamento de Ingeniería Eléctrica, Universidad de Castilla-La Mancha, Electrónica, Automática y Comunicaciones, 02071 Albacete, Spain
Polytechnic Institute of Leiria, Computer Science and Communications Research Centre, School of Technology and Management, 2411-901 Leiria, Portugal
INOV INESC INOVAÇÃO, Institute of New Technologies—Leiria Office, 2411-901 Leiria, Portugal
Departamento de Sistemas Informáticos, Universidad de Castilla-La Mancha, 02071 Albacete, Spain
Biomedical Research Networking Centre in Mental Health (CIBERSAM), 28016 Madrid, Spain
Funding
MCIN/AEI/10.13039/ 501100011033 and ERDF A way to make Europe · Junta de Comunidades de Castilla-La Mancha · Portuguese Fundação para a Ciência e a Tecnologia—FCT · Portuguese National Funds · CIBERSAM of the Instituto de Salud Carlos III
References (48)
Show source-provided references
1. Visual detection and tracking with UAVs, following a mobile object10.1080/01691864.2019.1596834 ↗
2. A vision-based home security system using OpenCV on Raspberry Pi 310.1063/1.5133928 ↗
3. Gómez-Reyes, J.K., Benítez-Rangel, J.P., Morales-Hernández, L.A., Resendiz-Ochoa, E., and Camarillo-Gomez, K.A. (2022). Image Mosaicing Applied on UAVs Survey. Appl. Sci., 12.10.3390/app12052729 ↗
4. Bouassida, S., Neji, N., Nouvelière, L., and Neji, J. (2021). Evaluating the Impact of Drone Signaling in Crosswalk Scenario. Appl. Sci., 11.10.3390/app11010157 ↗
5. Islam, S., Huang, Q., Afghah, F., Fule, P., and Razi, A. (2019, January 3–6). Fire Frontline Monitoring by Enabling UAV-Based Virtual Reality with Adaptive Imaging Rate. Proceedings of the 2019 53rd Asilomar Conference on Signals, Systems, and Computers, Grove, CA, USA.10.1109/IEEECONF44664.2019.9049048 ↗
6. Ribeiro, R., Ramos, J., Safadinho, D., Reis, A., Rabadão, C., Barroso, J., and Pereira, A. (2021). Web AR Solution for UAV Pilot Training and Usability Testing. Sensors, 21.10.3390/s21041456 ↗
7. Selecký, M., Faigl, J., and Rollo, M. (2018). Communication Architecture in Mixed-Reality Simulations of Unmanned Systems. Sensors, 18.10.3390/s18030853 ↗
8. Lan, G., Sun, J., Li, C., Ou, Z., Luo, Z., Liang, J., and Hao, Q. (2016, January 19–21). Development of UAV based virtual reality systems. Proceedings of the 2016 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems, Baden-Baden, Germany.10.1109/MFI.2016.7849534 ↗
9. Virtual Reality Simulation of a Quadrotor to Monitor Dependent People at Home10.1109/TETC.2020.3000352 ↗
10. Artificial intelligence within the interplay between natural and artificial computation: Advances in data science, trends and applications10.1016/j.neucom.2020.05.078 ↗
11. Jie, L., Jian, C., and Lei, W. (2017, January 27–29). Design of multi-mode UAV human-computer interaction system. Proceedings of the 2017 IEEE International Conference on Unmanned Systems, Beijing, China.10.1109/ICUS.2017.8278368 ↗
12. Vukić, M., Grgić, B., Dinčir, D., Kostelac, L., and Marković, I. (2019, January 20–24). Unity based Urban Environment Simulation for Autonomous Vehicle Stereo Vision Evaluation. Proceedings of the 2019 42nd International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO), Opatija, Croatia.10.23919/MIPRO.2019.8756805 ↗
13. Belmonte, L.M., Morales, R., and Fernández-Caballero, A. (2019). Computer vision in autonomous unmanned aerial vehicles—A systematic mapping study. Appl. Sci., 9.10.3390/app9153196 ↗
14. Arévalo, F., Sunaringtyas, D., Tito, C., Piolo, C., and Schwung, A. (2020, January 26–28). Interactive Visual Procedure using an extended FMEA and Mixed-Reality. Proceedings of the 2020 IEEE International Conference on Industrial Technology (ICIT), Buenos Aires, Argentina.10.1109/ICIT45562.2020.9067296 ↗
15. Tredinnick, R., Boettcher, B., Smith, S., Solovy, S., and Ponto, K. (2017, January 18–22). Uni-CAVE: A Unity3D plugin for non-head mounted VR display systems. Proceedings of the 2017 IEEE Virtual Reality (VR), Los Angeles, CA, USA.10.1109/VR.2017.7892342 ↗
16. Industrial Collaborative Robot Digital Twin integration and Control Using Robot Operating System
17. Immersive and interactive cyber-physical system (I2CPS) and virtual reality interface for human involved robotic manufacturing10.1016/j.jmsy.2021.11.018 ↗
18. Caiza, G., Bonilla-Vasconez, P., Garcia, C.A., and Garcia, M.V. (2020, January 8–11). Augmented Reality for Robot Control in Low-cost Automation Context and IoT. Proceedings of the 2020 25th IEEE International Conference on Emerging Technologies and Factory Automation, Vienna, Austria.10.1109/ETFA46521.2020.9212056 ↗
19. Kuzmic, J., and Rudolph, G. (2021, January 23–25). Comparison between Filtered Canny Edge Detector and Convolutional Neural Network for Real Time Lane Detection in a Unity 3D Simulator. Proceedings of the 6th International Conference on Internet of Things, Big Data and Security, Online.10.5220/0010383701480155 ↗
20. Wang, Z., Han, K., and Tiwari, P. (August, January 15). Digital Twin Simulation of Connected and Automated Vehicles with the Unity Game Engine. Proceedings of the 2021 IEEE 1st International Conference on Digital Twins and Parallel Intelligence, Beijing, China.10.1109/DTPI52967.2021.9540074 ↗
21. A Novel Approach to Communicate with Video Game Character using Cascade Classifiers
22. Ileperuma, I., Gunathilake, H., Dilshan, K., Nishali, S., Gamage, A., and Priyadarshana, Y. (2020, January 10–11). An Enhanced Virtual Fitting Room using Deep Neural Networks. Proceedings of the 2020 2nd International Conference on Advancements in Computing, Malabe, Sri Lanka.10.1109/ICAC51239.2020.9357160 ↗
23. Adão, T., Pinho, T., Pádua, L., Magalhães, L.G.J., Sousa, J., and Peres, E. (2021). Prototyping IoT-Based Virtual Environments: An Approach toward the Sustainable Remote Management of Distributed Mulsemedia Setups. Appl. Sci., 11.10.3390/app11198854 ↗
24. Fleck, P., Schmalstieg, D., and Arth, C. (2020, January 9–13). Creating IoT-Ready XR-WebApps with Unity3D. Proceedings of the The 25th International Conference on 3D Web Technology, Virtual Event.10.1145/3424616.3424691 ↗
25. Borkman, S., Crespi, A., Dhakad, S., Ganguly, S., Hogins, J., Jhang, Y., Kamalzadeh, M., Li, B., Leal, S., and Parisi, P. (2021). Unity Perception: Generate Synthetic Data for Computer Vision. arXiv.
26. NOVA: Rendering Virtual Worlds with Humans for Computer Vision Tasks10.1111/cgf.14271 ↗
27. Saviolo, A., Bonotto, M., Evangelista, D., Imperoli, M., Menegatti, E., and Pretto, A. (2021). Learning to Segment Human Body Parts with Synthetically Trained Deep Convolutional Networks. arXiv.10.1007/978-3-030-95892-3_52 ↗
28. Numata, S., and Jozen, T. (2021, January 12–15). A 3D Marker Recognition Method for AR Game Development. Proceedings of the 2021 IEEE 10th Global Conference on Consumer Electronics, Kyoto, Japan.10.1109/GCCE53005.2021.9621963 ↗
29. Alvey, B., Anderson, D.T., Buck, A., Deardorff, M., Scott, G., and Keller, J.M. (2021, January 11–17). Simulated Photorealistic Deep Learning Framework and Workflows to Accelerate Computer Vision and Unmanned Aerial Vehicle Research. Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision Workshops, Montreal, BC, Canada.10.1109/ICCVW54120.2021.00435 ↗
30. Fedoseev, A., Serpiva, V., Karmanova, E., Cabrera, M.A., Shirokun, V., Vasilev, I., Savushkin, S., and Tsetserukou, D. (2021, January 12–16). DroneTrap: Drone Catching in Midair by Soft Robotic Hand with Color-Based Force Detection and Hand Gesture Recognition. Proceedings of the 2021 IEEE 4th International Conference on Soft Robotics, New Haven, CT, USA.10.1109/RoboSoft51838.2021.9479353 ↗
31. Singh, N., Chouhan, S.S., and Verma, K. (2021, January 22–23). Object Oriented Programming: Concepts, Limitations and Application Trends. Proceedings of the 2021 5th International Conference on Information Systems and Computer Networks, Mathura, India.10.1109/ISCON52037.2021.9702463 ↗
32. Comparative Analysis of Complexity of C++ and Python Programming Languages
33. Zehra, F., Javed, M., Khan, D., and Pasha, M. (2020). Comparative Analysis of C++ and Python in Terms of Memory and Time, Preprints.10.20944/preprints202012.0516.v1 ↗
34. Ullah, K., Ahmed, I., Ahmad, M., and Khan, I. (2019, January 13–15). Comparison of Person Tracking Algorithms Using Overhead View Implemented in OpenCV. Proceedings of the 2019 9th Annual Information Technology, Electromechanical Engineering and Microelectronics Conference, Jaipur, India.10.1109/IEMECONX.2019.8877025 ↗
35. Boyko, N., Basystiuk, O., and Shakhovska, N. (2018, January 21–25). Performance Evaluation and Comparison of Software for Face Recognition, Based on Dlib and Opencv Library. Proceedings of the 2018 IEEE Second International Conference on Data Stream Mining Processing, Lviv, Ukraine.10.1109/DSMP.2018.8478556 ↗
36. Comparison of tracking algorithms implemented in OpenCV10.1051/matecconf/20167604031 ↗
37. Raghava, N., Gupta, K., Kedia, I., and Goyal, A. (2020, January 28–30). An Experimental Comparison of Different Object Tracking Algorithms. Proceedings of the 2020 International Conference on Communication and Signal Processing, Chennai, India.10.1109/ICCSP48568.2020.9182101 ↗
38. Tang, K., Wang, Y., Liu, H., Sheng, Y., Wang, X., and Wei, Z. (2013, January 8–9). Design and Implementation of Push Notification System Based on the MQTT Protocol. Proceedings of the 2013 International Conference on Information Science and Computer Applications, Changsha, China.10.2991/isca-13.2013.20 ↗
39. Belmonte, L.M., García, A.S., Morales, R., de la Vara, J.L., López de la Rosa, F., and Fernández-Caballero, A. (2021). Feeling of safety and comfort towards a socially assistive unmanned aerial vehicle that monitors people in a virtual home. Sensors, 21.10.3390/s21030908 ↗
40. Facial expression recognition in ageing adults: From lab to ambient assisted living10.1007/s12652-017-0464-x ↗
41. Martínez, A., Belmonte, L.M., García, A.S., Fernández-Caballero, A., and Morales, R. (2021). Facial emotion recognition from an unmanned flying social robot for home care of dependent people. Electronics, 10.10.20944/preprints202103.0189.v1 ↗
42. Comparison between Open CV and MATLAB Performance in Real Time Applications
43. Guennouni, S., Ahaitouf, A., and Mansouri, A. (2014, January 20–22). Multiple object detection using OpenCV on an embedded platform. Proceedings of the 2014 Third IEEE International Colloquium in Information Science and Technology (CIST), Tetouan, Morocco.10.1109/CIST.2014.7016649 ↗
44. Borsatti, D., Cerroni, W., Tonini, F., and Raffaelli, C. (2020, January 19–23). From IoT to Cloud: Applications and Performance of the MQTT Protocol. Proceedings of the 2020 22nd International Conference on Transparent Optical Networks (ICTON), Bari, Italy.10.1109/ICTON51198.2020.9203167 ↗
45. Hasan, N.I., Hasan, M.T., Turja, N.H., Raiyan, R., Saha, S., and Hossain, M.F. (2019, January 21–23). A Cyber-Secured MQTT based Offline Automation System. Proceedings of the 2019 International Conference on Wireless Communications Signal Processing and Networking (WiSPNET), Chennai, India.10.1109/WiSPNET45539.2019.9032743 ↗
46. A review: Comparison of performance metrics of pretrained models for object detection using the TensorFlow framework10.1088/1757-899X/844/1/012024 ↗
47. Jackson, S., and Murphy, P. (1999). Managing Work-Role Performance: Challenges for 21st Century Organizations and Employees. The Changing Nature of Work Performance, Wiley.
48. Finding missing person using ML, AI