Enhanced Retrieval-Augmented Generation Using Low-Rank Adaptation
Source abstract
Abstract
Recent advancements in retrieval-augmented generation (RAG) have substantially enhanced the efficiency of information retrieval. However, traditional RAG-based systems still encounter challenges, such as high latency in output decision making, the inaccurate retrieval of road traffic-related laws and regulations, and considerable processing overhead in large-scale searches. This study presents an innovative application of RAG technology for processing road traffic-related laws and regulations, particularly in the context of unmanned systems like autonomous driving. Our approach integrates embedding generation using a LoRA-enhanced BERT-based uncased model and an optimized retrieval strategy that combines maximal marginal similarity score thresholding with contextual compression retrieval. The proposed system enhances and achieves improved retrieval accuracy while reducing processing overhead. Leveraging road traffic-related regulatory datasets, the LoRA-enhanced model demonstrated remarkable performance gains over traditional RAG methods. Specifically, our model reduced the number of trainable parameters by 13.6% and lowered computational costs by 18.7%. Performance evaluations using BLEU, CIDEr, and SPICE scores revealed a 4.36% increase in BLEU-4, a 6.83% improvement in CIDEr, and a 5.46% improved in SPICE, confirming greater structural accuracy in regulatory text generation. Additionally, our method achieved an 8.5% improvement in retrieval accuracy across key metrics, outperforming baseline RAG systems. These contributions pave the way for more efficient and reliable traffic regulation processing, enabling better decision making in autonomous systems.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Multimodal Machine Learning Applications · Domain Adaptation and Few-Shot Learning · Advanced Image and Video Retrieval Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Computer Science and Engineering (CSE), Dongguk University-Seoul, Seoul 04620, Republic of Korea
Department of Computer Science and Artificial Intelligence, Dongguk University-Seoul, Seoul 04620, Republic of Korea
Funding
Institute of Information & Communications Technology Planning & Evaluation (IITP) under the Artificial Intelligence Convergence Innovation Human Resources Development
References (31)
Show source-provided references
1. Retrieval-augmented generation for knowledge-intensive nlp tasks
2. Predicting the next turn at road junction from big traffic data10.1007/s11227-017-2013-y ↗
3. Zhang, Y., and Sung, Y. (2023). Hybrid Traffic Accident Classification Models. Mathematics, 11.10.3390/math11041050 ↗
4. Carbonell, J., and Goldstein, J. (1998, January 24–28). The use of MMR, diversity-based reranking for reordering documents and producing summaries. Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Melbourne, Australia.10.1145/290941.291025 ↗
5. Lora: Low-rank adaptation of large language models
6. Gao, L., Dai, Z., and Callan, J. (April, January 28). Rethink training of BERT rerankers in multi-stage retrieval pipeline. Proceedings of the Advances in Information Retrieval: 43rd European Conference on IR Research, ECIR 2021, Virtual Event. Proceedings, Part II 43.10.1007/978-3-030-72240-1_26 ↗
7. A review on the attention mechanism of deep learning10.1016/j.neucom.2021.03.091 ↗
8. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (2017). Attention is all you need. Adv. Neural Inf. Process. Syst., 30.
9. Papineni, K., Roukos, S., Ward, T., and Zhu, W.J. (2002, January 6–12). Bleu: A method for automatic evaluation of machine translation. Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Philadelphia, PA, USA.10.3115/1073083.1073135 ↗
10. Vedantam, R., Lawrence Zitnick, C., and Parikh, D. (2015, January 7–12). Cider: Consensus-based image description evaluation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7299087 ↗
11. Anderson, P., Fernando, B., Johnson, M., and Gould, S. (2016, January 11–14). Spice: Semantic propositional image caption evaluation. Proceedings of the Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands. Proceedings, Part V 14.10.1007/978-3-319-46454-1_24 ↗
12. Ebrahimi Soorchaei, B., Razzaghpour, M., Valiente, R., Raftari, A., and Fallah, Y.P. (2022). High-definition map representation techniques for automated vehicles. Electronics, 11.10.20944/preprints202209.0067.v1 ↗
13. A comparison of semantic similarity methods for maximum human interpretability
14. Gao, C., Chen, K., Rao, J., Sun, B., Liu, R., Peng, D., Zhang, Y., Guo, X., Yang, J., and Subrahmanian, V. (2024). Higher layers need more lora experts. arXiv.
15. Wiratunga, N., Abeyratne, R., Jayawardena, L., Martin, K., Massie, S., Nkisi-Orji, I., Weerasinghe, R., Liret, A., and Fleisch, B. (2024, January 1–4). CBR-RAG: Case-based reasoning for retrieval augmented generation in LLMs for legal question answering. Proceedings of the International Conference on Case-Based Reasoning, Yucatan, Mexico.10.1007/978-3-031-63646-2_29 ↗
16. Lost in the middle: How language models use long contexts10.1162/tacl_a_00638 ↗
17. Wang, Z., Wang, Z., Le, L., Zheng, H.S., Mishra, S., Perot, V., Zhang, Y., Mattapalli, A., Taly, A., and Shang, J. (2024). Speculative rag: Enhancing retrieval augmented generation through drafting. arXiv.
18. Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. (2020, January 13–18). A simple framework for contrastive learning of visual representations. Proceedings of the International Conference on Machine Learning (PMLR), Virtual.
19. Li, H., Ai, Q., Chen, J., Dong, Q., Wu, Y., Liu, Y., Chen, C., and Tian, Q. (2023, January 23–27). SAILER: Structure-aware pre-trained language model for legal case retrieval. Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, Taipei, Taiwan.10.1145/3539618.3591761 ↗
20. Pickett, M., Hartman, J., Bhowmick, A.K., Alam, R.U., and Vempaty, A. (2024). Better RAG using Relevant Information Gain. arXiv.
21. Nogueira, R., and Cho, K. (2019). Passage Re-ranking with BERT. arXiv.
22. Pradeep, R., Nogueira, R., and Lin, J. (2021). The expando-mono-duo design pattern for text ranking with pretrained sequence-to-sequence models. arXiv.10.18653/v1/2020.findings-emnlp.63 ↗
23. Choi, D., Im, J., and Sung, Y. (2024). LoRA Fusion: Enhancing Image Generation. Mathematics, 12.10.3390/math12223474 ↗
24. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2019, January 3–5). Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, MN, USA.
25. Simulation-based testing and evaluation tools for transportation cyber–physical systems10.1109/TVT.2015.2407614 ↗
26. Ligeza, A. (2006). Logical Foundations for Rule-Based Systems, Springer.10.1007/3-540-32446-1 ↗
27. Generative AI service implementation using LLM application architecture: Based on RAG model and LangChain framework
28. Zhao, W.X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., and Dong, Z. (2023). A survey of large language models. arXiv.
29. Li, S., Stenzel, L., Eickhoff, C., and Bahrainian, S.A. (2025). Enhancing Retrieval-Augmented Generation: A Study of Best Practices. arXiv.
30. Pfeiffer, J., Kamath, A., Rücklé, A., Cho, K., and Gurevych, I. (2020). Adapterfusion: Non-destructive task composition for transfer learning. arXiv.10.18653/v1/2021.eacl-main.39 ↗
31. Lester, B., Al-Rfou, R., and Constant, N. (2021). The power of scale for parameter-efficient prompt tuning. arXiv.10.18653/v1/2021.emnlp-main.243 ↗