AI on the Road: NVIDIA Jetson Nano-Powered Computer Vision-Based System for Real-Time Pedestrian and Priority Sign Detection
Source abstract
Abstract
Advances in information and signal processing, driven by artificial intelligence techniques and recent breakthroughs in deep learning, have significantly impacted autonomous driving by enhancing safety and reducing the dependence on human intervention. Generally, prevailing ADASs (advanced driver assistance systems) incorporate costly components, making them financially unattainable for a substantial portion of the population. This paper proposes a solution: an embedded system designed for real-time pedestrian and priority sign detection, offering affordability and universal applicability across various vehicles. The suggested system, which comprises two cameras, an NVIDIA Jetson Nano B01 low-power edge device and an LCD (liquid crystal system) display, ensures seamless integration into a vehicle without occupying substantial space and provides a cost-effective alternative. The primary focus of this research is addressing accidents caused by the failure to yield priority to other drivers or pedestrians. Our study stands out from existing research by concurrently addressing traffic sign recognition and pedestrian detection, concentrating on identifying five crucial objects: pedestrians, pedestrian crossings (signs and road paintings separately), stop signs, and give way signs. Object detection was executed using a lightweight, custom-trained CNN (convolutional neural network) known as SSD (Single Shot Detector)-MobileNet, implemented on the Jetson Nano. To tailor the model for this specific application, the pre-trained neural network underwent training on our custom dataset consisting of images captured on the road under diverse lighting and traffic conditions. The outcomes of the proposed system offer promising results, positioning it as a viable candidate for real-time implementation; its contributions are noteworthy in advancing the safety and accessibility of autonomous driving technologies.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Neural Network Applications · Video Surveillance and Tracking Methods · Autonomous Vehicle Technology and Safety
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Mechatronics Department, Vehicles and Mechatronics Institute, Faculty of Engineering, University of Debrecen, Ótemető Str. 2-4, 4028 Debrecen, Hungary
Department of Vehicles Engineering, Vehicles and Mechatronics Institute, Faculty of Engineering, University of Debrecen, Ótemető Str. 2-4, 4028 Debrecen, Hungary
Funding
University of Debrecen, Faculty of Engineering
References (69)
Show source-provided references
1. Human Factors in the Causation of Road Traffic Crashes10.1023/A:1007649804201 ↗
2. World Health Organization (2023, March 25). Available online: https://www.who.int/news-room/fact-sheets/detail/road-traffic-injuries.
3. Technical Feasibility of Advanced Driver Assistance Systems (ADAS) for Road Traffic Safety10.1080/03081060500120282 ↗
4. Design of efficient embedded system for road sign recognition10.1007/s12652-017-0673-3 ↗
5. Combining Unmanned Aerial Vehicles with Artificial-Intelligence Technology for Traffic-Congestion Recognition: Electronic Eyes in the Skies to Spot Clogged Roads10.1109/MCE.2019.2892286 ↗
6. Infrared and visible images fusion by using sparse representation and guided filter10.1080/15472450.2019.1643725 ↗
7. Levinson, J., Askeland, J., Becker, J., Dolson, J., Held, D., Kammel, S., Kolter, J.Z., Langer, D., Pink, O., and Pratt, V. (2011, January 5–9). Towards fully autonomous driving: Systems and algorithms. Proceedings of the 2011 IEEE Intelligent Vehicles Symposium (IV), Baden, Germany.10.1109/IVS.2011.5940562 ↗
8. An overview of traffic sign detection and classification methods10.1007/s13735-017-0129-8 ↗
9. Traffic Surveillance: A Review of Vision Based Vehicle Detection, Recognition and Tracking
10. Robust obstacle detection for advanced driver assistance systems using distortions of inverse perspective mapping of a monocular camera10.1016/j.robot.2018.12.004 ↗
11. RefineNet: Refining Object Detectors for Autonomous Driving10.1109/TIV.2017.2695896 ↗
12. Computer vision and deep learning techniques for pedestrian detection and tracking: A survey10.1016/j.neucom.2018.01.092 ↗
13. Activity Recognition Using Temporal Optical Flow Convolutional Features and Multilayer LSTM10.1109/TIE.2018.2881943 ↗
14. (2024, February 01). Intel. Available online: https://les-svc.org/resources/LES-SVC%205-18-16%20Article%20-%20Intel-automotive-autonomous-driving-vision-paper.pdf.
15. (2023, March 15). Tesla. Available online: https://www.tesla.com/autopilot.
16. (2023, March 15). Mobileye. Available online: https://www.mobileye.com/true-redundancy/.
17. A Survey of Autonomous Driving: Common Practices and Emerging Technologies10.1109/ACCESS.2020.2983149 ↗
18. Toward Performing Image Classification and Object Detection with Convolutional Neural Networks in Autonomous Driving Systems: A Survey10.1109/ACCESS.2022.3147495 ↗
19. Imagenet classification with deep convolutional neural networks10.1145/3065386 ↗
20. Ren, S., He, K., Girshick, R., and Sun, J. (2015, January 7–12). Faster R-CNN: Towards real-time object detection with region proposal networks. Proceedings of the Advances in Neural Information Processing Systems 28 (NIPS 2015), Montreal, QC, Canada.
21. Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.-Y., and Berg, C.A. (2016). Computer Vision–ECCV 2016, Springer.
22. Redmon, J., and Farhadi, A. (2017, January 21–26). YOLO9000: Better, Faster, Stronger. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.690 ↗
23. Quassim, H., Verma, A., and Feinzimer, D. (2018, January 8–10). Compressed residual-VGG16 CNN model for big data places image recognition. Proceedings of the IEEE 8th Annual Computing and Communication Workshop and Conference (CCWC), Las Vegas, NV, USA.10.1109/CCWC.2018.8301729 ↗
24. Howard, A.G., Zhu, M., Chen, B., Kalenichencko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. (2017). MobileNets: Efficient Convolutional Neural Networks for Mobile Vision. arXiv.
25. Chiu, Y.-C., Tsai, C.-Y., Ruan, M.-D., Shen, G.-Y., and Lee, T.-T. (September, January 31). Mobilenet-SSDv2: An Improved Object Detection Model for Embedded Systems. Proceedings of the International Conference on System Science and Engineering (ICSSE), Kagawa, Japan.
26. Array programming with NumPy10.1038/s41586-020-2649-2 ↗
27. McKinney, W. (July, January 28). Data structures for statistical computing in python. Proceedings of the 9th Python in Science Conference, Austin, TX, USA.
28. McKinney, W. (2024, February 01). Pandas: Powerful Python Data Analysis Toolkit. Available online: https://pandas.pydata.org/pandas-docs/version/0.25.3/pandas.pdf.
29. Matplotlib: A 2D graphics environment10.1109/MCSE.2007.55 ↗
30. Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., and Isard, M. (2016). TensorFlow: A system for large-scale machine learning. arXiv.
31. Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., and Antiga, L. (2019, January 8). PyTorch: An Imperative Style, High-Performance. Proceedings of the 33rd International Conference on Neural Information Processing Systems, New York, NY, USA.
32. (2023, March 25). Papers With Code. Available online: https://paperswithcode.com/trends.
33. Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., and Darrell, T. (2014, January 7–8). Caffe: Convolutional Architecture for Fast Feature Embedding. Proceedings of the 22nd ACM international conference on Multimedia, Orlando, FL, USA.10.1145/2647868.2654889 ↗
34. Chollet, F. (2024, February 01). Keras: Deep Learning for humans. Available online: https://keras.io/keras_3/.
35. Camera-Based Driver Monitoring System for Abnormal Behavior Detection10.5455/jjee.204-1586348076 ↗
36. Oczan, K., Sharma, A., Knickerbocker, S., Merickel, J., Hawkins, N., and Rizzo, M. (2019). Advances in Computer Vision, Springer.
37. A two-stage real-time YOLOv2-based road marking detector with lightweight spatial transformation-invariant classification10.1016/j.imavis.2020.103978 ↗
38. Barba-Guaman, L., Naranjo, J.E., and Ortiz, A. (2020). Deep Learning Framework for Vehicle and Pedestrian Detection in Rural Roads on an Embedded GPU. Electronics, 9.10.3390/electronics9040589 ↗
39. Two-stage traffic sign detection and recognition based on SVM and convolutional neural networks10.1049/iet-ipr.2019.0634 ↗
40. Dalal, N., and Triggs, B. (2005, January 20–26). Histograms of Oriented Gradients for Human Detection. Proceedings of the Computer Vision and Pattern Recognition, San Diego, CA, USA.
41. An Automatic Traffic Sign Detection and Recognition System Based on Colour Segmentation, Shape Matching, and SVM10.1155/2015/250461 ↗
42. Muller, J., and Dietmayer, K. (2018, January 4–7). Detecting Traffic Lights by Single Shot Detection. Proceedings of the 21st International Conference on Intelligent Transportation Systems (ITSC), Maui, HI, USA.10.1109/ITSC.2018.8569683 ↗
43. Fregin, A., Muller, J., Krebel, U., and Dietmayer, K. (2018, January 21–25). The DriveU Traffic Light Dataset: Introduction and Comparison with Existing Datasets. Proceedings of the 2018 IEEE international conference on robotics and automation (ICRA), Brisbane, Australia.10.1109/ICRA.2018.8460737 ↗
44. An Implementation of Real-Time Traffic Signs and Road Objects Detection Based on Mobile GPU Platforms10.1109/ACCESS.2022.3198954 ↗
45. Ayachi, R., Afif, M., Said, Y., and Ben Abdelali, A. (2020, January 9–11). Traffic Sign Recognition Based on Scaled Convolutional Neural Network for Advanced Driver Assistance System. Proceedings of the IEEE 4th International Conference on Image Processing, Applications and Systems (IPAS), Genova, Italy.10.1109/IPAS50080.2020.9334944 ↗
46. Real-time traffic sign detection and classification towards real traffic scene10.1007/s11042-020-08722-y ↗
47. Chiu, Y.-C., Lin, H.-Y., and Tai, W.-L. (2019, January 3–6). Implementation and Evaluation of CNN Based Traffic Sign Detection with Different Resolutions. Proceedings of the International Symposium on Intelligent Signal Processing and Communication Systems (ISPACS), Taipei, Taiwan.10.1109/ISPACS48206.2019.8986319 ↗
48. Weber, M., Huber, M., and Zollner, J.M. (2018, January 4–7). HDTLR: A CNN based Hierarchical Detector for Traffic Lights. Proceedings of the 21st International Conference on Intelligent Transportation Systems (ITSC), Maui, HI, USA.10.1109/ITSC.2018.8569794 ↗
49. Deep CNN-Based Real-Time Traffic Light Detector for Self-Driving Vehicles10.1109/TMC.2019.2892451 ↗
50. Enhanced Detection and Recognition of Road Markings Based on Adaptive Region of Interest and Deep Learning10.1109/ACCESS.2019.2933598 ↗
51. (2023, March 25). NVIDIA. Available online: https://developer.nvidia.com/embedded/jetson-nano-developer-kit.
52. Object Detection Using IR Camera10.1007/978-3-642-31665-4_11 ↗
53. Park, J., Chen, J., Cho, Y.K., Kang, D.Y., and Son, B.J. (2019). CNN-Based Person Detection Using Infrared Images for Night-Time Intrusion Warning Systems. Sensors, 20.10.3390/s20010034 ↗
54. (2023, March 25). Raspberry PI. Available online: https://www.raspberrypi.com/products/camera-module-v2/.
55. (2023, March 25). Waveshare Electronics. Available online: https://www.waveshare.com/wiki/IMX219-160IR_Camera.
56. Kumar, P. (2023, March 25). e-con Systems. Available online: https://www.e-consystems.com/blog/camera/technology/mipi-camera-vs-usb-camera-a-detailed-comparison/.
57. (2023, March 25). MIPI Alliance. Available online: https://www.mipi.org/driving-the-wires-of-automotive.
58. Real-Time Assembly Operation Recognition with Fog Computing and Transfer Learning for Human-Centered Intelligent Manufacturing10.1016/j.promfg.2020.05.131 ↗
59. Hui, J. (2023, March 25). Medium. Available online: https://jonathan-hui.medium.com/object-detection-speed-and-accuracy-comparison-faster-r-cnn-r-fcn-ssd-and-yolo-5425656ae359.
60. (2023, March 25). Dusty-nv. GitHub. Available online: https://github.com/dusty-nv/jetson-inference.
61. Tan, R.J. (2023, March 25). Towards Data Science. Available online: https://towardsdatascience.com/breaking-down-mean-average-precision-map-ae462f623a52.
62. Subramanyam, V.S. (2023, March 25). Medium. Available online: https://medium.com/analytics-vidhya/iou-intersection-over-union-705a39e7acef.
63. (2023, March 25). Tensorflow 2 Detection Model Zoo. Available online: https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf2_detection_zoo.md.
64. (2023, March 25). Tensorflow. Available online: https://www.tensorflow.org/install/source.
65. (2023, March 25). Tensorflow Object Detection API Tutorial. Available online: https://tensorflow-object-detection-api-tutorial.readthedocs.io/en/latest/install.html#tf-install.
66. Bhande, A. (2023, March 25). Medium. Available online: https://medium.com/greyatom/what-is-underfitting-and-overfitting-in-machine-learning-and-how-to-deal-with-it-6803a989c76.
67. Mikołajczyk, A., and Grochowski, M. (2018, January 9–12). Data augmentation for improving deep learning in image classification problem. Proceedings of the International Interdisciplinary PhD Workshop (IIPhDW), Swinoujscie, Poland.10.1109/IIPHDW.2018.8388338 ↗
68. (2023, March 25). Github. Available online: https://github.com/NVIDIA/TensorRT.
69. Neta, Z., Wu, H., and Rodge, J. (2023, March 25). NVIDIA. Available online: https://developer.nvidia.com/blog/achieving-fp32-accuracy-for-int8-inference-using-quantization-aware-training-with-tensorrt/.