LoRA-Tuned Multimodal RAG System for Technical Manual QA: A Case Study on Hyundai Staria
Source abstract
Abstract
This study develops a domain-adaptive multimodal RAG (Retrieval-Augmented Generation) system to improve the accuracy and efficiency of technical question answering based on large-scale structured manuals. Using Hyundai Staria maintenance documents as a case study, we extracted text and images from PDF manuals and constructed QA, RAG, and Multi-Turn datasets to reflect realistic troubleshooting scenarios. To overcome limitations of baseline RAG models, we proposed an enhanced architecture that incorporates sentence-level similarity annotations and parameter-efficient fine-tuning via LoRA (Low-Rank Adaptation) using the bLLossom-8B language model and BAAI-bge-m3 embedding model. Experimental results show that the proposed system achieved improvements of 3.0%p in BERTScore, 3.0%p in cosine similarity, and 18.0%p in ROUGE-L compared to existing RAG systems, with notable gains in image-guided response accuracy. A qualitative evaluation by 20 domain experts yielded an average satisfaction score of 4.4 out of 5. This study presents a practical and extensible AI framework for multimodal document understanding, with broad applicability across automotive, industrial, and defense-related technical documentation.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Natural Language Processing Techniques · Topic Modeling · Handwritten Text Recognition Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Defense Artificial Intelligence Applications, Seoul National University of Science and Technology, 232 Gongneung-ro, Nowon-gu, Seoul 01811, Republic of Korea
Funding
Defense Acquisition Program Administration (DAPA) · Agency for Defense Development (ADD)
References (23)
Show source-provided references
1. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
2. Gao, Y., Liu, X., Wang, Z., Li, P., and Tang, J. (2023). A Survey on Retrieval-Augmented Generation. arXiv.
3. Hierarchical Document-Based Retrieval System for Automotive Manuals
4. Automatic Indexing and Structure Analysis of Technical Manuals: A Case Study on Toyota Service Documents
5. Hyundai Motor Group (2025, July 24). Beyond Just Driving: R&D for Infotainment System UX. Hyundai Motor Group Tech, Available online: https://www.hyundaimotorgroup.com/story/CONT0000000000003395.
6. Zhao, R., Lin, T., and Chen, Q. (2022, January 17–20). BERT-based Automatic Document Classification for Vehicle Service Manuals at Tesla. Proceedings of the 2022 IEEE International Conference on Big Data, Osaka, Japan.
7. Chen, L., Duan, N., Yin, P., Wang, H., and Zhou, M. (2022, January 22–27). MuRAG: Multimodal Retrieval-Augmented Generation for Visual Question Answering. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, Dublin, Ireland.
8. Surla, A., and Varshney, T. (2025, July 24). An Easy Introduction to Multimodal Retrieval-Augmented Generation. NVIDIA Technical Blog 2024. Available online: https://developer.nvidia.com/blog/an-easy-introduction-to-multimodal-retrieval-augmented-generation/.
9. Jin, H., Wang, Y., and Wang, X. (2023). MMed-RAG: Retrieval-Augmented Generation for Medical QA. arXiv.
10. Xu, L., Patel, S., and Wang, Z. (2023, January 19–23). LegalBench-RAG: Domain-Specific Retrieval-Augmented Generation for Legal Reasoning. Proceedings of the 2023 Conference on Legal AI, Braga, Portugal.
11. Mayer, R. (2009). Multimedia Learning, Cambridge University Press. [2nd ed.].
12. Nadel, E. (2021). How Augmented Reality Became a Serious Tool for Manufacturing. Autom. World, Available online: https://www.automationworld.com/process/iiot/article/21259479/how-augmented-reality-became-a-serious-tool-for-manufacturing.
13. Boeing (2020). 3D Interactive Maintenance Manuals for Commercial Aircraft. Technical Manual Publication, Boeing.
14. Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C.L., and Parikh, D. (2015, January 7–13). VQA: Visual Question Answering. Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile.10.1109/ICCV.2015.279 ↗
15. Mathew, M., Iyyer, M., and Zettlemoyer, L. (2021, January 20–25). DocVQA: Document Visual Question Answering. Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.
16. U.S. Department of Defense (2019). Interactive Electronic Technical Manual (IETM) System Overview. Def. Maint. J.
17. NATO (2020). Standardization Agreement (STANAG) 4677: Interactive Electronic Technical Manuals, NATO Standardization Office.
18. Agency for Defense Development (ADD) (2022). Weapon System Maintenance Support Platform, Agency for Defense Development. Defense Science Report.
19. Lockheed Martin (2021). F-35 AR-Based Maintenance Support Tools: Operational Efficiency Report, Lockheed Martin. Technical Briefing.
20. Li, J., Kim, D., Liu, J., Yao, Y., Tan, C.-S., Yang, Y., and Yin, X. (2023). BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. arXiv.
21. Girdhar, R., Singh, M., Caron, M., Misra, I., Joulin, A., Synnaeve, G., and Mairal, J. (2023). ImageBind: One Embedding Space to Bind Them All. arXiv.10.1109/CVPR52729.2023.01457 ↗
22. Hu, E., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, L., and Chen, W. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arXiv.
23. Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. arXiv.