Multi-Modal System for Walking Safety for the Visually Impaired: Multi-Object Detection and Natural Language Generation
Source abstract
Abstract
This study introduces a system for visually impaired individuals in a walking environment. It combines object recognition using YOLOv5 and cautionary sentence generation with KoAlpaca. The system employs image data augmentation for diverse training data and GPT for natural language training. Furthermore, the implementation of the system on a single board was followed by a comprehensive comparative analysis with existing studies. Moreover, a pilot test involving visually impaired and healthy individuals was conducted to validate the system’s practical applicability and adaptability in real-world walking environments. Our pilot test results indicated an average usability score of 4.05. Participants expressed some dissatisfaction with the notification conveying time and online implementation, but they highly praised the system’s object detection range and accuracy. The experiments demonstrated that using QLoRA enables more efficient training of larger models, which is associated with improved model performance. Our study makes a significant contribution to the literature because the proposed system enables real-time monitoring of various environmental conditions and objects in pedestrian environments using AI.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Autonomous Vehicle Technology and Safety · Tactile and Sensory Interactions · Advanced Neural Network Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Artificial Intelligence, Daegu University, Gyeongsan 38453, Republic of Korea
Department of Computer and Information Engineering, Daegu University, Gyeongsan 38453, Republic of Korea
Funding
Ministry of Education of the Republic of Korea
References (57)
Show source-provided references
1. (2024, June 14). World Health Organization. Available online: https://www.who.int/news-room/fact-sheets/detail/blindness-and-visual-impairment.
2. A survey on assistive technology for visually impaired10.1016/j.iot.2020.100188 ↗
3. (2024, June 14). Research and Markets. Available online: https://www.researchandmarkets.com/report/visual-impairment-assistive-technology.
4. An analysis on O&M ability status and needs of people with visual impairments for the development of AI guide robot10.17337/JMBI.2023.25.2.99 ↗
5. El-taher, F.E.-z., Taha, A., Courtney, J., and Mckeever, S. (2021). A systematic review of urban navigation systems for visually impaired people. Sensors, 21.10.3390/s21093103 ↗
6. Panazan, C.-E., and Dulf, E.-H. (2024). Intelligent cane for assisting the visually impaired. Technologies, 12.10.3390/technologies12060075 ↗
7. Smart walking cane based on triboelectric nanogenerators for assisting the visually impaired10.1016/j.nanoen.2024.109485 ↗
8. Takano, T., Nakane, T., Akashi, T., and Zhang, C. (2021). Braille block detection via multi-objective optimization from an egocentric viewpoint. Sensors, 21.10.3390/s21082775 ↗
9. Walle, H., De Runz, C., Serres, B., and Venturini, G. (2022). A survey on recent advances in AI and vision-based methods for helping and guiding visually impaired people. Appl. Sci., 12.10.3390/app12052308 ↗
10. Said, Y., Atri, M., Albahar, M.A., Ben Atitallah, A., and Alsariera, Y.A. (2023). Obstacle detection system for navigation assistance of visually impaired people based on deep learning techniques. Sensors, 23.10.3390/s23115262 ↗
11. A wearable assistive system for the visually impaired using object detection, distance measurement and tactile presentation10.20517/ir.2023.24 ↗
12. Yang, K., Bergasa, L.M., Romera, E., Cheng, R., Chen, T., and Wang, K. (2018, January 26–30). Unifying terrain awareness through real-time semantic segmentation. Proceedings of the 2018 IEEE Intelligent Vehicles Symposium (IV), Changshu, China.10.1109/IVS.2018.8500506 ↗
13. Enhancing perception for the visually impaired with deep learning techniques and low-cost wearable sensors10.1016/j.patrec.2019.03.008 ↗
14. Eckert, M., Blex, M., and Friedrich, C.M. (2018, January 19–21). Object Detection featuring 3D audio localization for Microsoft HoloLens—A deep learning based sensor substitution approach for the blind. Proceedings of the 11th International Joint Conference on Biomedical Engineering Systems and Technologies, Funchal, Portugal.10.5220/0006655605550561 ↗
15. Ayes (2024, August 01). OKO App Leverages AI to Help Blind Pedestrians Recognize Traffic Signals. Available online: https://www.ayes.ai/post/oko-app-leverages-ai-to-help-blind-pedestrians-recognize-traffic-signals.
16. Be My Eyes (2024, August 01). Introducing Be My AI. Available online: https://www.bemyeyes.com/blog/introducing-be-my-ai.
17. OOrion (2024, August 01). OOrion Official Website. Available online: https://www.oorion.fr/en/.
18. Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. (2016, January 27–30). You only look once: Unified, real-time object detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.91 ↗
19. (2024, June 18). Github. Available online: https://github.com/Beomi/KoAlpaca.
20. Text to speech: A simple tutorial
21. O’Shea, K., and Nash, R. (2015). An introduction to convolutional neural networks. arXiv.
22. ImageNet classification with deep convolutional neural networks10.1145/3065386 ↗
23. Girshick, R., Donahue, J., Darrell, T., and Malik, J. (2014, January 23–28). Rich feature hierarchies for accurate object detection and semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Columbus, OH, USA.10.1109/CVPR.2014.81 ↗
24. Faster R-CNN: Towards real-time object detection with region proposal networks10.1109/TPAMI.2016.2577031 ↗
25. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
26. A comprehensive review of YOLO architectures in computer vision: From YOLOv1 to YOLOv8 and YOLO-NAS10.3390/make5040083 ↗
27. (2024, June 18). Github. Available online: https://github.com/ultralytics/yolov5.
28. (2024, June 18). Github. Available online: https://github.com/tatsu-lab/stanford_alpaca.
29. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., and Azhar, F. (2023). LLaMA: Open and efficient foundation language models. arXiv.
30. Ko, H., Yang, K., Ryu, M., Choi, T., Yang, S., Hyun, J., Park, S., and Park, K. (2023). A Technical report for Polyglot-Ko: Open-source large-scale Korean language models. arXiv.
31. Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. (2023). QLoRA: Efficient finetuning of quantized LLMs. arXiv.
32. (2024, June 18). Huggingface. Available online: https://huggingface.co/.
33. Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., and Gelly, S. (2019, January 9–15). Parameter-efficient transfer learning for NLP. Proceedings of the International Conference on Machine Learning, Long Beach, CA, USA.
34. (2024, June 18). Nvidia Developer. Available online: https://developer.nvidia.com/ko-kr/blog/develop-ai-powered-robots-smart-vision-systems-and-more-with-nvidia-jetson-orin-nano-developer-kit/.
35. Liu, S., Qi, L., Qin, H., Shi, J., and Jia, J. (2018, January 18–23). Path aggregation network for instance segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00913 ↗
36. (2024, June 18). Roboflow. Available online: https://roboflow.com/.
37. (2024, June 18). Roboflow. Available online: https://universe.roboflow.com/trafficlight-7r04p/traffic-light-szdih.
38. (2024, June 18). Roboflow. Available online: https://universe.roboflow.com/dkdkd/capstone-for-detection1.
39. (2024, June 18). Roboflow. Available online: https://universe.roboflow.com/sajotuna/traffic-sign-bykpq.
40. (2024, June 18). Roboflow. Available online: https://universe.roboflow.com/ps7/pedestrian-signs-and-lanes.
41. (2024, June 18). Roboflow. Available online: https://universe.roboflow.com/dkdkd/july_6.
42. (2024, June 18). Ultralytics. Available online: https://docs.ultralytics.com/yolov5/tutorials/tips_for_best_training_results/#model-selection.
43. Lin, T.Y., Dollár, P., Girshick, R., He, K., Hariharan, B., and Belongie, S. (2017, January 21–26). Feature pyramid networks for object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.106 ↗
44. Information bottleneck theory on convolutional neural networks10.1007/s11063-021-10445-6 ↗
45. Wang, C.Y., Liao, H.Y.M., Wu, Y.H., Chen, P.Y., Hsieh, J.W., and Yeh, I.H. (2020, January 14–19). CSPNet: A new backbone that can enhance learning capability of CNN. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Seattle, WA, USA.10.1109/CVPRW50498.2020.00203 ↗
46. (2024, June 18). Ultralytics. Available online: https://docs.ultralytics.com/yolov5/tutorials/hyperparameter_evolution/#before-you-start.
47. Confusion matrix: Machine learning
48. A review of object detection: Datasets, performance evaluation, architecture, applications and current trends10.1007/s11042-023-17949-4 ↗
49. Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N.A., Khashabi, D., and Hajishirzi, H. (2022). Self-instruct: Aligning language models with self-generated instructions. arXiv.10.18653/v1/2023.acl-long.754 ↗
50. Wei, X., Wei, H., Lin, H., Li, T., Zhang, P., Ren, X., Li, M., Wan, Y., Cao, Z., and Xie, B. (2023). Polylm: An open-source polyglot large language model. arXiv.
51. (2024, June 22). OpenAI. Available online: https://openai.com/blog/chatgpt.
52. (2024, June 22). Pandas. Available online: https://pandas.pydata.org.
53. Xiong, R., Yang, Y., He, D., Zheng, K., Zheng, S., Xing, C., and Liu, T. (2020, January 13–18). On layer normalization in the transformer architecture. Proceedings of the International Conference on Machine Learning, Virtual.
54. WhyLabs (2024, August 01). Evaluating Large Language Models (LLMs) 2024. Available online: https://whylabs.ai/learning-center/introduction-to-llms/evaluating-large-language-models-llms.
55. Terven, J., Cordova-Esparza, D.M., Ramirez-Pedraza, A., and Chavez-Urbiola, E.A. (2023). Loss functions and metrics in deep learning. A review. arXiv.
56. Slack (2024, August 01). Slack API Documentation. Available online: https://api.slack.com/.
57. (2024, June 22). Nvdia. Available online: https://www.nvidia.com/en-us/data-center/ampere-architecture/.