Application of Multimodal AI to Aid Scene Perception for the Visually Impaired
Source abstract
Abstract
In this paper, the use of generative multimodal models for image analysis is proposed, with the goal of determining the selection of parameters for 3D scene segmentation algorithms in systems designed to assist blind individuals in navigation. AI algorithms enable scene type detection, lighting condition assessment, and determination of whether a scene can be used to obtain parameters necessary for system initialization, such as the orientation of imaging sensors relative to the ground. Additionally, the effectiveness of extracting selected scene parameters using four multimodal models is evaluated, and the results are compared to annotations made by a human. The obtained results highlight the potential of utilizing such models to enhance the functionality of systems belonging to the Electronics Travel Aid group, particularly in terms of parameter selection for scene segmentation algorithms and scene presentation to visually impaired individuals.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Tactile and Sensory Interactions · Gaze Tracking and Assistive Technology · Currency Recognition and Detection
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Institute of Electronics, Lodz University of Technology, Al. Politechniki 8, 93-590 Lodz, Poland
References (26)
Show source-provided references
1. Sensory substitution: Closing the gap between basic research and widespread practical visual rehabilitation10.1016/j.neubiorev.2013.11.007 ↗
2. Pissaloux, E., and Velazquez, R. (2018). Mobility of Visually Impaired People, Springer.10.1007/978-3-319-54446-5 ↗
3. El-taher, F.E.-Z., Taha, A., Courtney, J., and Mckeever, S. (2021). A Systematic Review of Urban Navigation Systems for Visually Impaired People. Sensors, 21.10.3390/s21093103 ↗
4. Okolo, G.I., Althobaiti, T., and Ramzan, N. (2024). Assistive Systems for Visually Impaired Persons: Challenges and Opportunities for Navigation Assistance. Sensors, 24.10.3390/s24113572 ↗
5. Wearable Obstacle Avoidance Electronic Travel Aids for Blind and Visually Impaired Individuals: A Systematic Review10.1109/ACCESS.2023.3285396 ↗
6. Wearable assistive devices for visually impaired: A state of the art survey10.1016/j.patrec.2018.10.031 ↗
7. Krzhizhanovskaya, V.V., Závodszky, G., Lees, M.H., Dongarra, J.J., Sloot, P.M.A., Brissos, S., and Teixeira, J. (2020, January 3–5). Interactive Travel Aid for the Visually Impaired: From Depth Maps to Sonic Patterns and Verbal Messages. Proceedings of the Computational Science—ICCS 2020, Amsterdam, The Netherlands.
8. Strumiłło, P., Klepaczko, A., Strzelecki, M., and Bociaga, D. (2024). Haptic Display of Depth Images in an Electronic Travel Aid for the Blind: Technical Indoor Trials. Proceedings of the Latest Developments and Challenges in Biomedical Engineering, Springer.10.1007/978-3-031-38430-1 ↗
9. How Can Haptic Feedback Assist People with Blind and Low Vision (BLV): A Systematic Literature Review10.1145/3711931 ↗
10. Hu, Z., and Uchimura, K. (2005, January 6–8). U-V-disparity: An efficient algorithm for stereovision based scene analysis. Proceedings of the IEEE Proceedings. Intelligent Vehicles Symposium, Las Vegas, NV, USA.10.1109/IVS.2005.1505076 ↗
11. Herghelegiu, P., Burlacu, A., and Caraiman, S. (2016, January 13–15). Robust ground plane detection and tracking in stereo sequences using camera orientation. Proceedings of the 2016 20th International Conference on System Theory, Control and Computing (ICSTCC), Sinaia, Romania.10.1109/ICSTCC.2016.7790717 ↗
12. Battiato, S., Gallo, G., Schettini, R., and Stanco, F. (2017, January 11–15). Real Time Indoor 3D Pipeline for an Advanced Sensory Substitution Device. Proceedings of the Image Analysis and Processing—ICIAP 2017, Catania, Italy.10.1007/978-3-319-68560-1 ↗
13. Fischler, M.A., and Firschein, O. (1987). Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography. Readings in Computer Vision, Morgan Kaufmann.10.1016/B978-0-08-051581-6.50070-2 ↗
14. Research advances of indoor navigation for blind people: A brief review of technological instrumentation10.1109/MIM.2020.9126068 ↗
15. Assisting the Visually Impaired: Obstacle Detection and Warning System by Acoustic Feedback10.3390/s121217476 ↗
16. Caraiman, S., Zvoristeanu, O., Burlacu, A., and Herghelegiu, P. (2019). Stereo Vision Based Sensory Substitution for the Visually Impaired. Sensors, 19.10.3390/s19122771 ↗
17. Computer vision for assistive technologies10.1016/j.cviu.2016.09.001 ↗
18. Kuzdeuov, A., Mukayev, O., Nurgaliyev, S., Kunbolsyn, A., and Varol, H.A. (2024, January 19–22). ChatGPT for Visually Impaired and Blind. Proceedings of the 2024 International Conference on Artificial Intelligence in Information and Communication (ICAIIC), Osaka, Japan.10.1109/ICAIIC60209.2024.10463430 ↗
19. Be My Eyes (2025, May 21). Introducing Be My AI (Formerly Virtual Volunteer) for People Who Are Blind or Have Low Vision, Powered by OpenAI’s GPT-4. Available online: https://www.bemyeyes.com/news/introducing-be-my-ai-formerly-virtual-volunteer-for-people-who-are-blind-or-have-low-vision-powered-by-openais-gpt-4/.
20. Artificial intelligence for visually impaired10.1016/j.displa.2023.102391 ↗
21. Voutsakelis, G., Dimkaros, I., Tzimos, N., Kokkonis, G., and Kontogiannis, S. (2025). Development and Evaluation of a Tool for Blind Users Utilizing AI Object Detection and Haptic Feedback. Machines, 13.10.3390/machines13050398 ↗
22. Liu, H., Li, C., Li, Y., and Lee, Y.J. (2023, January 17–24). Improved Baselines with Visual Instruction Tuning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada.
23. Liu, H., Li, C., Wu, Q., and Lee, Y.J. (2023, January 10–16). Visual Instruction Tuning. Proceedings of the NeurIPS, New Orleans, LO, USA.
24. Liu, H., Li, C., Li, Y., Li, B., Zhang, Y., Shen, S., and Lee, Y.J. (2025, April 28). LLaVA-NeXT: Improved Reasoning, OCR, and World Knowledge. Available online: https://llava-vl.github.io/blog/2024-01-30-llava-next/.
25. Contributors, X. (2025, April 28). XTuner: A Toolkit for Efficiently Fine-Tuning LLM. Available online: https://github.com/InternLM/xtuner.
26. Yao, Y., Yu, T., Zhang, A., Wang, C., Cui, J., Zhu, H., Cai, T., Li, H., Zhao, W., and He, Z. (2024). MiniCPM-V: A GPT-4V Level MLLM on Your Phone. arXiv.