Evaluating Retrieval-Augmented Generation Models for Financial Report Question and Answering
Source abstract
Abstract
This study explores the application of retrieval-augmented generation (RAG) to improve the accuracy and reliability of large language models (LLMs) in the context of financial report analysis. The focus is on enabling private investors to make informed decisions by enhancing the question-and-answering capabilities regarding the half-yearly or quarterly financial reports of banks. The study adopts a Design Science Research (DSR) methodology to develop and evaluate an RAG system tailored for this use case. The study conducts a series of experiments to explore models in which different RAG components are used. The aim is to enhance context relevance, answer faithfulness, and answer relevance. The results indicate that model one (OpenAI ADA and OpenAI GPT-4) achieved the highest performance, showing robust accuracy and relevance in response. Model three (MiniLM Embedder and OpenAI GPT-4) scored significantly lower, indicating the importance of high-quality components. The evaluation also revealed that well-structured reports result in better RAG performance than less coherent reports. Qualitative questions received higher scores than the quantitative ones, demonstrating the RAG’s proficiency in handling descriptive data. In conclusion, a tailored RAG can aid investors in providing accurate and contextually relevant information from financial reports, thereby enhancing decision making.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Advanced Text Analysis Techniques · Information Retrieval and Search Behavior
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Business, University of Applied Sciences and Arts Northwestern Switzerland, 4600 Olten, Switzerland
Institute for Information Systems, University of Applied Sciences and Arts Northwestern Switzerland, 4600 Olten, Switzerland
References (46)
Show source-provided references
1. Zhao, W.X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., and Dong, Z. (2023). A Survey of Large Language Models. arXiv.
2. Language Models are Few-Shot Learners
3. The Working Limitations of Large Language Models
4. Guu, K., Lee, K., Tung, Z., Pasupat, P., and Chang, M. (2020, January 13–18). REALM: Retrieval-Augmented Language Model Pre-Training. Proceedings of the International Conference on Machine Learning, PMLR, Vienna, Austria.
5. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
6. Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Guo, Q., and Wang, M. (2024). Retrieval-Augmented Generation for Large Language Models: A Survey. arXiv.
7. IBM Research (2021, February 09). What Is Retrieval-Augmented Generation?. Available online: https://research.ibm.com/blog/retrieval-augmented-generation-RAG.
8. Chen, J., Lin, H., Han, X., and Sun, L. (2024, January 20–27). Benchmarking large language models in retrieval-augmented generation. Proceedings of the AAAI Conference on Artificial Intelligence, Vancouver, BC, Canada.
9. Shuster, K., Poff, S., Chen, M., Kiela, D., and Weston, J. (2021). Retrieval Augmentation Reduces Hallucination in Conversation. arXiv.10.18653/v1/2021.findings-emnlp.320 ↗
10. Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A.M., Hauth, A., and Millican, K. (2023). Gemini: A family of highly capable multimodal models. arXiv.
11. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., and Bhosale, S. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv.
12. He, H., Zhang, H., and Roth, D. (2022). Rethinking with Retrieval: Faithful Large Language Model Inference. arXiv.
13. Shen, X., Chen, Z., Backes, M., and Zhang, Y. (2023). In ChatGPT We Trust? Measuring and Characterizing the Reliability of ChatGPT. arXiv.
14. Zhang, Y., Li, Y., Cui, L., Cai, D., Liu, L., Fu, T., Huang, X., Zhao, E., Zhang, Y., and Chen, Y. (2023). Siren’s Song in the AI Ocean: A Survey on Hallucination in Large Language Models. arXiv.
15. Borgeaud, S., Mensch, A., Hoffmann, J., Cai, T., Rutherford, E., Millican, K., Van Den Driessche, G.B., Lespiau, J.-B., Damoc, B., and Clark, A. (2022, January 17–23). Improving language models by retrieving from trillions of tokens. Proceedings of the International Conference on Machine Learning, PMLR, Baltimore, MD, USA.
16. Lyu, Y., Li, Z., Niu, S., Xiong, F., Tang, B., Wang, W., Wu, H., Liu, H., Xu, T., and Chen, E. (2024). CRUD-RAG: A Comprehensive Chinese Benchmark for Retrieval-Augmented Generation of Large Language Models. arXiv.10.1145/3701228 ↗
17. Es, S., James, J., Espinosa-Anke, L., and Schockaert, S. (2023). RAGAS: Automated Evaluation of Retrieval Augmented Generation. arXiv.
18. Saad-Falcon, J., Khattab, O., Potts, C., and Zaharia, M. (2023). ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems. arXiv.10.18653/v1/2024.naacl-long.20 ↗
19. Ragas (2024, July 10). Core Concepts. Available online: https://docs.ragas.io/en/latest/concepts/index.html.
20. TruLens (2024, July 10). RAG Triad. Available online: https://truera.com/ai-quality-education/generative-ai-rags/what-is-the-rag-triad/.
21. Besbes, A. (2023, November 23). A 3-Step Approach to Evaluate a Retrieval Augmented Generation (RAG). Towards Data Science. Available online: https://towardsdatascience.com/a-3-step-approach-to-evaluate-a-retrieval-augmented-generation-rag-5acf2aba86de.
22. Besbes, A. (2023, December 21). Quickly Evaluate Your RAG without Manually Labeling Test Data. Towards Data Science. Available online: https://towardsdatascience.com/quickly-evaluate-your-rag-without-manually-labeling-test-data-43ade0ae187a.
23. Frenchi, C. (2023, December 01). Evaluating RAG: Using LLMs to Automate Benchmarking of Retrieval Augmented Generation Systems. Willow Tree Apps. Available online: https://www.willowtreeapps.com/craft/evaluating-rag-using-llms-to-automate-benchmarking-of-retrieval-augmented-generation-systems.
24. Leal, M., and Frenchi, C. (2023, September 21). Evaluating Truthfulness: Benchmarking LLM Accuracy. Willow Tree Apps. Available online: https://www.willowtreeapps.com/craft/evaluating-truthfulness-a-deeper-dive-into-benchmarking-llm-accuracy.
25. Nguyen, R. (2023, October 01). LlamaIndex: How to Evaluate Your RAG (Retrieval Augmented Generation) Applications. Better Programming. Available online: https://betterprogramming.pub/llamaindex-how-to-evaluate-your-rag-retrieval-augmented-generation-applications-2c83490f489.
26. Sarmah, B., Zhu, T., Mehta, D., and Pasquali, S. (2023). Towards reducing hallucination in extracting information from financial reports using Large Language Models. arXiv.10.1145/3639856.3639895 ↗
27. Yepes, A.J., You, Y., Milczek, J., Laverde, S., and Li, R. (2024). Financial Report Chunking for Effective Retrieval Augmented Generation. arXiv.
28. Zhang, B., Yang, H., Zhou, T., Ali Babar, M., and Liu, X.-Y. (2023, January 27–29). Enhancing Financial Sentiment Analysis via Retrieval Augmented Large Language Models. Proceedings of the Fourth ACM International Conference on AI in Finance, Brooklyn, NY, USA.10.1145/3604237.3626866 ↗
29. Hevner, A., and Chatterjee, S. (2010). Design Science Research in Information Systems. Design Research in Information Systems: Theory and Practice, Springer.10.1007/978-1-4419-5653-8 ↗
30. HuggingFace (2024, January 18). MiniLMEmbedder. Available online: https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2.
31. LangChain (2024, July 10). Bedrock Embeddings. Available online: https://python.langchain.com/v0.1/docs/integrations/text_embedding/bedrock/.
32. Meta (2017, March 29). Faiss: A Library for Efficient Similarity Search. Available online: https://engineering.fb.com/2017/03/29/data-infrastructure/faiss-a-library-for-efficient-similarity-search/.
33. Chroma (2024, July 10). Chroma Docs. Available online: https://docs.trychroma.com/getting-started.
34. OpenAI (2024, July 10). OpenAI Platform. Available online: https://platform.openai.com/docs/overview.
35. Meta (2024, July 10). Meta Llama 3. Available online: https://llama.meta.com/llama3/.
36. Google DeepMind (2024, May 20). Gemini Pro 1.5. Available online: https://deepmind.google/technologies/gemini/pro/.
37. LangChain (2024, July 10). Introduction to LangChain. Available online: https://python.langchain.com/v0.1/docs/get_started/introduction/.
38. LlamaIndex (2024, July 10). LlamaIndex Docs. Available online: https://docs.llamaindex.ai/en/stable/.
39. Weaviate (2024, July 10). Verba Docs. GitHub. Available online: https://github.com/weaviate/Verba.
40. Weaviate (2024, July 10). Verba—Demo Tool. Available online: https://verba.weaviate.io/.
41. OpenAI (2024, May 13). Hello GPT-4o. Available online: https://openai.com/index/hello-gpt-4o/.
42. Google (2024, April 09). Gemini 1.5 Pro Now Available in 180+ Countries. Available online: https://developers.googleblog.com/en/gemini-15-pro-now-available-in-180-countries-with-native-audio-understanding-system-instructions-json-mode-and-more/.
43. Google (2024, May 14). Gemini 1.5 Pro Updates. Available online: https://blog.google/technology/developers/gemini-gemma-developer-updates-may-2024/.
44. Large Model Systems Organization (2024, June 06). LLM Leaderboard. Available online: https://chat.lmsys.org/?leaderboard.
45. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. (2021, January 3–7). Measuring Massive Multitask Language Understanding. Proceedings of the 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria.
46. Papers with Code (2024, July 10). Multi-task Language Understanding on MMLU. Available online: https://paperswithcode.com/sota/multi-task-language-understanding-on-mmlu.