Evaluation and Selection of Hardware and AI Models for Edge Applications: A Method and A Case Study on UAVs
Source abstract
Abstract
This study proposes a method for selecting suitable edge hardware and Artificial Intelligence (AI) models to be deployed on these edge devices. Edge AI, which enables devices at the network periphery to perform intelligent tasks locally, is rapidly expanding across various domains. However, selecting appropriate edge hardware and AI models is a multi-faceted challenge due to the wide range of available options, diverse application requirements, and the unique constraints of edge environments, such as limited computational power, strict energy constraints, and the need for real-time processing. Ad hoc approaches often lead to non-optimal solutions and inefficiency problems. Considering these issues, we propose a method based on the ISO/IEC 25010:2011 quality standard, integrating Multi-Criteria Decision Analysis (MCDA) techniques to assess both the hardware and software aspects of Edge AI applications systematically. For the proposed method, we conducted an experiment consisting of two stages: In the first stage of the experiment, to show the applicability of the method across different use cases, we tested the method with four scenarios on UAVs, each presenting distinct edge requirements. In the second stage of the experiment, guided by the method’s recommendations for Scenario I, where the STM32H7 series microcontrollers were identified as the suitable hardware and the object detection model with Single Shot Multi-Box Detector (SSD) architecture and MobileNet backbone as the suitable AI model, we developed a TensorFlow Lite model from scratch to enhance the efficiency and versatility of the model for object detection tasks across various categories. This additional TensorFlow Lite model is aimed to show how the proposed method can guide the further development of optimized AI models tailored to the constraints and requirements of specific edge hardware.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Autonomous Vehicle Technology and Safety · Advanced Neural Network Applications · Robotic Path Planning Algorithms
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Engineering, Hacettepe University, 06800 Ankara, Türkiye
References (85)
Show source-provided references
1. Edge AI: Internet of Things and Cyber-Physical Systems10.1016/j.iotcps.2023.02.004 ↗
2. Edge Computing: Vision and Challenges10.1109/JIOT.2016.2579198 ↗
3. Edge Computing for Autonomous Driving: Opportunities and Challenges10.1109/JPROC.2019.2915983 ↗
4. A Smart, Efficient, and Reliable Parking Surveillance System with Edge Artificial Intelligence on IoT Devices10.1109/TITS.2020.2984197 ↗
5. Sharkov, G., Asif, W., and Rehman, I. (2022, January 26–29). Securing Smart Home Environment Using Edge Computing. Proceedings of the 2022 IEEE International Smart Cities Conference (ISC2), Pafos, Cyprus.10.1109/ISC255366.2022.9921912 ↗
6. Shankar, V. (2024, January 23–24). Edge AI: A Comprehensive Survey of Technologies, Applications, and Challenges. Proceedings of the 2024 1st International Conference on Advanced Computing and Emerging Technologies (ACET), Ghaziabad, India.10.1109/ACET61898.2024.10730112 ↗
7. Hadidi, R., Cao, J., Xie, Y., Asgari, B., Krishna, T., and Kim, H. (2019, January 3–5). Characterizing the Deployment of Deep Neural Networks on Commercial Edge Devices. Proceedings of the 2019 IEEE International Symposium on Workload Characterization (IISWC), Orlando, FL, USA.10.1109/IISWC47752.2019.9041955 ↗
8. Garcia-Perez, A., Miñón, R., Torre-Bastida, A.I., and Zulueta-Guerrero, E. (2023). Analysing Edge Computing Devices for the Deployment of Embedded AI. Sensors, 23.10.3390/s23239495 ↗
9. Selection of IoT Platform with Multi-Criteria Analysis: Defining Criteria and Experts to Interview10.13053/rcs-148-11-1 ↗
10. Silva, E.M., Agostinho, C., and Jardim-Goncalves, R. (2017, January 27–29). A multi-criteria decision model for the selection of a more suitable Internet-of-Things device. Proceedings of the 2017 International Conference on Engineering, Technology and Innovation (ICE/ITMC), Madeira, Portugal.10.1109/ICE.2017.8280026 ↗
11. Ilieva, G., and Yankova, T. (2022). IoT System Selection as a Fuzzy Multi-Criteria Problem. Sensors, 22.10.3390/s22114110 ↗
12. A DEA Approach towards to the Evaluation of IoT Applications in Intelligent Ports10.51400/2709-6998.1431 ↗
13. Park, S., and Lee, K. (2021). Improved Mitigation of Cyber Threats in IIoT for Smart Cities: A New-Era Approach and Scheme. Sensors, 21.10.3390/s21061976 ↗
14. A hybrid service selection optimization algorithm in internet of things10.1186/s13638-020-01883-2 ↗
15. How to Make a Decision: The Analytic Hierarchy Process
16. Extensions of the TOPSIS for Group Decision-making under Fuzzy Environment10.1016/S0165-0114(97)00377-1 ↗
17. A Fuzzy Compromise Solution for Multicriteria Problems10.1142/S0218488507004728 ↗
18. Uslu, B., Eren, T., Gur, S., and Ozcan, E. (2019). Evaluation of the Difficulties in the Internet of Things (IoT) with Multi-Criteria Decision-Making. Processes, 7.10.3390/pr7030164 ↗
19. A hybrid approach to automatic IaaS service selection10.1186/s13677-018-0113-8 ↗
20. Evaluating distributed IoT databases for edge/cloud platforms using the analytic hierarchy process10.1016/j.jpdc.2018.10.008 ↗
21. Silva, E.M., and Jardim-Goncalves, R. (2017). Multi-criteria analysis and decision methodology for the selection of internet-of-things hardware platforms. Doctoral Conference on Computing, Electrical and Industrial Systems, Springer.10.1007/978-3-319-56077-9_10 ↗
22. A framework for ranking of cloud computing services10.1016/j.future.2012.06.006 ↗
23. (2011). Systems and software engineering-Systems and Software Quality Requirements and Evaluation (SQuaRE)-System and software quality models (Standard No. ISO/IEC 25010:2011).
24. Sipola, T., Alatalo, J., Kokkonen, T., and Rantonen, M. (2022, January 27–29). Artificial Intelligence in the IoT Era: A Review of Edge AI Hardware and Software. Proceedings of the 31th Conference of Open Innovations Association FRUCT, Helsinki, Finland.10.23919/FRUCT54823.2022.9770931 ↗
25. Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. (2017). MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. arXiv.
26. Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.C. (2018, January 18–22). MobileNetV2: Inverted Residuals and Linear Bottlenecks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.10.1109/CVPR.2018.00474 ↗
27. Iandola, F.N., Han, S., Moskewicz, M.W., Ashraf, K., Dally, W.J., and Keutzer, K. (2016). SqueezeNet: AlexNet-level Accuracy with 50x Fewer Parameters and <0.5MB Model Size. arXiv.
28. Han, S., Pool, J., Tran, J., and Dally, W. (2015, January 7–12). Learning both Weights and Connections for Efficient Neural Networks. Proceedings of the 28th International Conference on Neural Information Processing Systems, Montreal, QC, Canada.
29. Tan, M., and Le, Q.V. (2019, January 10–15). EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. Proceedings of the International Conference on Machine Learning (ICML), Long Beach, CA, USA.
30. Tan, M., and Le, Q.V. (2020). EfficientNetV2: Smaller Models and Faster Training. arXiv.
31. Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. (July, January 26). You Only Look Once: Unified, Real-Time Object Detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.
32. Long Short-Term Memory10.1162/neco.1997.9.8.1735 ↗
33. LSTM: A Search Space Odyssey10.1109/TNNLS.2016.2582924 ↗
34. CloudAware: Empowering context-aware self-adaptation for mobile applications10.1002/ett.3210 ↗
35. Quality of service approaches in IoT: A systematic mapping10.1016/j.jss.2017.05.125 ↗
36. Quality attributes in edge computing for the Internet of Things: A systematic mapping study10.1016/j.iot.2020.100346 ↗
37. (2024, December 17). ISO/IEC 25010:2011. Available online: https://www.iso.org/standard/35733.html.
38. pyFDM: A Python Library for Uncertainty Decision Analysis Methods10.1016/j.softx.2022.101271 ↗
39. Technologies and Decision Support Systems to Aid Solid-waste Management: A Systematic Review10.1016/j.wasman.2016.10.045 ↗
40. CLUS-MCDA: A Novel Framework based on Cluster Analysis and Multiple Criteria Decision Theory in a Supplier Selection Problem10.1016/j.cie.2018.03.011 ↗
41. A New Integrated Grey MCDM Model: Case of Warehouse Location Selection
42. A Multi-criteria Decision-making Approach to the New Product Development Process in Industry10.31181/rme2001260122y ↗
43. Combining Spatial Analysis with MCDA for the Siting of Healthcare Facilities10.1016/j.landusepol.2018.02.044 ↗
44. Chen, S.J., and Hwang, C.L. (1992). Fuzzy Multiple Attribute Decision Making: Methods and Applications, Lecture Notes in Economics and Mathematical Systems; Springer.10.1007/978-3-642-46768-4 ↗
45. Zimmermann, H.J. (2001). Fuzzy Set Theory—And Its Applications, Springer Science & Business Media.10.1007/978-94-010-0646-0 ↗
46. A Hybrid Fuzzy AHP-MABAC Model: Application in the Serbian Army–The selection of the location for deep wading as a technique of crossing the river by tanks10.31181/dmame1801143b ↗
47. Sałabun, W., Karczmarczyk, A., Wątróbski, J., and Jankowski, J. (2018, January 18–21). Handling Data Uncertainty in Decision Making with COMET. Proceedings of the 2018 IEEE Symposium Series on Computational Intelligence (SSCI), Bangalore, India.10.1109/SSCI.2018.8628934 ↗
48. Shekhovtsov, A., Paradowski, B., Więckowski, J., Kizielewicz, B., and Sałabun, W. (2022, January 6–9). Extension of the SPOTIS Method for the Rank Reversal Free Decision-making under Fuzzy Environment. Proceedings of the 2022 IEEE 61st Conference on Decision and Control (CDC), Cancun, Mexico.10.1109/CDC51059.2022.9992833 ↗
49. A New Hybrid Fuzzy PSI-PIPRECIA-CoCoSo MCDM based Approach to Solving the Transportation Company Selection Problem10.3846/tede.2021.15058 ↗
50. Mandel, N., Milford, M., and Gonzalez, F. (2020). A Method for Evaluating and Selecting Suitable Hardware for Deployment of Embedded System on UAVs. Sensors, 20.10.3390/s20164420 ↗
51. Imran, H., Mujahid, U., Wazir, S., Latif, U., and Mehmood, K. (2020). Embedded Development Boards for Edge-AI: A Comprehensive Report. arXiv.
52. Merenda, M., Porcaro, C., and Iero, D. (2020). Edge Machine Learning for AI-enabled IoT devices: A review. Sensors, 20.10.3390/s20092533 ↗
53. Rajput, K.R., Kulkarni, C.D., Cho, B., Wang, W., and Kim, I.K. (2022, January 11–15). Edgefaasbench: Benchmarking Edge Devices using Serverless Computing. Proceedings of the 2022 IEEE International Conference on Edge Computing and Communications (EDGE), Barcelona, Spain.10.1109/EDGE55608.2022.00024 ↗
54. Feng, H., Mu, G., Zhong, S., Zhang, P., and Yuan, T. (2022). Benchmark Analysis of Yolo Performance on Edge Intelligence Devices. Cryptography, 6.10.3390/cryptography6020016 ↗
55. Cantero, D., Esnaola-Gonzalez, I., Miguel-Alonso, J., and Jauregi, E. (2022). Benchmarking Object Detection Deep Learning Models in Embedded Devices. Sensors, 22.10.3390/s22114205 ↗
56. Matching terms of quality models and meta-models: Toward a unified meta-model of OSS quality10.1007/s11219-022-09603-3 ↗
57. Quality Evaluation Meta-model for Open-source Software: Multi-method Validation Study10.1007/s11219-023-09658-w ↗
58. Boroujerdian, B., Genc, H., Krishnan, S., Cui, W., Faust, A., and Reddi, V. (2018, January 20–24). MAVBench: Micro Aerial Vehicle Benchmarking. Proceedings of the 2018 51st Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), Fukuoka, Japan.10.1109/MICRO.2018.00077 ↗
59. (2024, December 17). NVIDIA Series Products. Available online: https://developer.nvidia.com/.
60. (2024, December 17). Intel Atom 6000E Product. Available online: https://www.intel.com/content/www/us/en/content-details/635255/intel-atom-x6000e-series-processor-and-intel-pentium-and-celeron-n-and-j-series-processors-for-internet-of-things-iot-applications-datasheet-volume-2-book-1-of-3.html.
61. (2024, December 17). Google Coral Products. Available online: https://coral.ai/products/.
62. (2024, December 17). Raspberry Products. Available online: https://www.raspberrypi.com/.
63. (2024, December 17). STM32 Microcontroller Products. Available online: https://www.st.com/content/st_com/en.html.
64. (2024, December 17). Xilinx Boards. Available online: https://www.xilinx.com/products/boards-and-kits.html.
65. DYVERSE: DYnamic VERtical Scaling in multi-tenant Edge environments10.1016/j.future.2020.02.043 ↗
66. (2024, December 17). Beagle Board products. Available online: https://www.beagleboard.org/.
67. Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.Y., and Berg, A.C. (2016, January 11–14). SSD: Single Shot Multibox Detector. Proceedings of the European Conference on Computer Vision (ECCV), Amsterdam, The Netherlands.10.1007/978-3-319-46448-0_2 ↗
68. (2024, December 17). COCO Person Dataset. Available online: https://cocodataset.org/#home.
69. (2024, December 17). STMicroelectronics–Object Detection, STM32 Model Zoo. Available online: https://github.com/STMicroelectronics/stm32ai-modelzoo.
70. (2024, December 17). QEngineering, Jetson Nano and Raspberry Pi 4 Metrics. Available online: https://github.com/Qengineering.
71. Choe, C., Choe, M., and Jung, S. (2023). Run Your 3D Object Detector on NVIDIA Jetson Platforms: A Benchmark Analysis. Sensors, 23.10.3390/s23084005 ↗
72. (2024, December 17). ImageNet. Available online: https://www.image-net.org/.
73. (2024, December 17). PASCAL VOC Dataset. Available online: https://datasets.activeloop.ai/docs/ml/datasets/pascal-voc-2012-dataset/.
74. Rausand, M., and Hoyland, A. (2004). System Reliability Theory: Models, Statistical Methods, and Applications. Wiley-Interscience, John Wiley & Sons Inc.
75. Fei-Fei, L., Deng, J., Russakovksy, O., Berg, A., and Li, K. (2024, December 17). Common Evaluation Metrics Used in Computer Vision Tasks, Documented in ImageNet Benchmark Datasets. Available online: https://image-net.org/about.php.
76. Lin, T., Patterson, G., Ronchi, M., Maire, M., Belongie, S., Bourdev, L., Girschik, R., Hays, J., Perona, P., and Ramanan, D. (2024, December 17). Common Evaluation Metrics Used in Computer Vision Tasks, Documented in COCO. Available online: https://www.picsellia.com/post/coco-evaluation-metrics-explained.
77. Goodfellow, I., Shlens, J., and Szegedy, C. (2014). Explaining and Harnessing Adversarial Examples. arXiv.
78. Towards Deep Learning Models Resistant to Adversarial Attacks
79. (2024, December 17). Jetsons Benchmark. Available online: https://developer.nvidia.com/embedded/jetson-benchmarks.
80. (2024, December 17). MLPerf Inference: Edge. Available online: https://mlcommons.org/benchmarks/inference-edge/.
81. A Survey on Transfer Learning10.1109/TKDE.2009.191 ↗
82. Reliability Prediction of Electronic Equipment (2024, December 17). MIL-HDBK-217F. Available online: https://www.navsea.navy.mil/Portals/103/Documents/NSWC_Crane/SD-18/Test%20Methods/MILHDBK217.pdf.
83. International Electrotechnical Commission (2013). Environmental Testing-Part 2: Tests (IEC 60068-2 Series), IEC.
84. Laprie, J.C. (1991). Dependability: Basic Concepts and Terminology in English, French, German, Italian and Japanese, Dependable Computing and Fault-Tolerant Systems; Springer.10.1007/978-3-7091-9170-5 ↗
85. Bloomfield, R., and Rushby, J. (2024). Assurance of AI Systems from a Dependability Perspective. arXiv.