Fine-Tuning Retrieval-Augmented Generation with an Auto-Regressive Language Model for Sentiment Analysis in Financial Reviews
Source abstract
Abstract
Sentiment analysis is a well-known task that has been used to analyse customer feedback reviews and media headlines to detect the sentimental personality or polarisation of a given text. With the growth of social media and other online platforms, like Twitter (now branded as X), Facebook, blogs, and others, it has been used in the investment community to monitor customer feedback, reviews, and news headlines about financial institutions’ products and services to ensure business success and prioritise aspects of customer relationship management. Supervised learning algorithms have been popularly employed for this task, but the performance of these models has been compromised due to the brevity of the content and the presence of idiomatic expressions, sound imitations, and abbreviations. Additionally, the pre-training of a larger language model (PTLM) struggles to capture bidirectional contextual knowledge learnt through word dependency because the sentence-level representation fails to take broad features into account. We develop a novel structure called language feature extraction and adaptation for reviews (LFEAR), an advanced natural language model that amalgamates retrieval-augmented generation (RAG) with a conversation format for an auto-regressive fine-tuning model (ARFT). This helps to overcome the limitations of lexicon-based tools and the reliance on pre-defined sentiment lexicons, which may not fully capture the range of sentiments in natural language and address questions on various topics and tasks. LFEAR is fine-tuned on Hellopeter reviews that incorporate industry-specific contextual information retrieval to show resilience and flexibility for various tasks, including analysing sentiments in reviews of restaurants, movies, politics, and financial products. The proposed model achieved an average precision score of 98.45%, answer correctness of 93.85%, and context precision of 97.69% based on Retrieval-Augmented Generation Assessment (RAGAS) metrics. The LFEAR model is effective in conducting sentiment analysis across various domains due to its adaptability and scalable inference mechanism. It considers unique language characteristics and patterns in specific domains to ensure accurate sentiment annotation. This is particularly beneficial for individuals in the financial sector, such as investors and institutions, including those listed on the Johannesburg Stock Exchange (JSE), which is the primary stock exchange in South Africa and plays a significant role in the country’s financial market. Future initiatives will focus on incorporating a wider range of data sources and improving the system’s ability to express nuanced sentiments effectively, enhancing its usefulness in diverse real-world scenarios.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Sentiment Analysis and Opinion Mining · Topic Modeling · Advanced Text Analysis Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Science, University of Pretoria, Lynnwood Road, Pretoria 0002, South Africa
Funding
Data Science for Social Impact (DSFI) Group at the University of Pretoria
References (100+)
Show source-provided references
1. The usage of social media as a marketing tool in two Southern African countries10.1080/0376835X.2016.1204228 ↗
2. Digital 2021: Zimbabwe
3. He, W., Aboderin, I., and Adjaye-Gbewonyo, D. (2020). Africa Aging, US Government Printing Office.
4. Social media, mobile phones and migration in Africa: A review of the evidence10.1177/14649934211043615 ↗
5. Shields, A. (2024, October 01). Banking and Social Media – Channel, Information and Advice. Available online: https://rfi.global/banking-and-social-media-channel-information-and-advice/.
6. Hao, S., Hoi, S.C.H., Miao, C., and Zhao, P. (2015, January 6–9). Active Crowdsourcing for Annotation. Proceedings of the 2015 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT), Singapore.10.1109/WI-IAT.2015.34 ↗
7. Nowak, S., and Rüger, S. (2010, January 29–31). How reliable are annotations via crowdsourcing: A study about inter-annotator agreement for multi-label image annotation. Proceedings of the International Conference on Multimedia Information Retrieval, (MIR’10), New York, NY, USA.10.1145/1743384.1743478 ↗
8. Belal, M., She, J., and Wong, S. (2023). Leveraging ChatGPT As Text Annotation Tool For Sentiment Analysis. arXiv.
9. Vetulani, Z., and Mariani, J. (2014). Crowdsourcing for Language Resource Development: Criticisms About Amazon Mechanical Turk Overpowering Use. Human Language Technology Challenges for Computer Science and Linguistics, Springer.10.1007/978-3-319-08958-4 ↗
10. A comprehensive study on lexicon based approaches for sentiment analysis10.51983/ajcst-2019.8.S2.2037 ↗
11. ChatGPT as a Text Annotation Tool to Evaluate Sentiment Analysis on South African Financial Institutions10.1109/ACCESS.2024.3464374 ↗
12. Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., and Dean, J. (2013). Distributed representations of words and phrases and their compositionality. arXiv.
13. Pennington, J., Socher, R., and Manning, C.D. (2014, January 25–29). Glove: Global vectors for word representation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar.10.3115/v1/D14-1162 ↗
14. Shen, Y., and Liu, J. (2021, January 12–14). Comparison of Text Sentiment Analysis based on Bert and Word2vec. Proceedings of the 2021 IEEE 3rd International Conference on Frontiers Technology of Information and Computer (ICFTIC), Virtual Conference.10.1109/ICFTIC54370.2021.9647258 ↗
15. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv.
16. Let’s play mono-poly: BERT can reveal words’ polysemy level and partitionability into senses10.1162/tacl_a_00400 ↗
17. Zhang, W., Deng, Y., Liu, B., Pan, S.J., and Bing, L. (2023). Sentiment analysis in the era of large language models: A reality check. arXiv.10.18653/v1/2024.findings-naacl.246 ↗
18. Inserte, P.R., Nakhlé, M., Qader, R., Caillaut, G., and Liu, J. (2024). Large Language Model Adaptation for Financial Sentiment Analysis. arXiv.
19. All-in-One: Emotion, Sentiment and Intensity Prediction Using a Multi-Task Ensemble Framework10.1109/TAFFC.2019.2926724 ↗
20. Using a contextual entropy model to expand emotion words and their intensity for the sentiment classification of stock market news10.1016/j.knosys.2013.01.001 ↗
21. Big data applications in operations/supply-chain management: A literature review10.1016/j.cie.2016.09.023 ↗
22. Zaslavsky, A., Perera, C., and Georgakopoulos, D. (2013). Sensing as a Service and Big Data. arXiv.
23. Valdeolmillos, D., Mezquita, Y., and Ludeiro, A.R. (2020, January 17–19). Sensing as a service: An architecture proposal for big data environments in smart cities. Proceedings of the Ambient Intelligence—Software and Applications, 10th International Symposium on Ambient Intelligence, Ávila, Spain.10.1007/978-3-030-24097-4_12 ↗
24. Sentiment analysis of big data: Methods, applications, and open challenges10.1109/ACCESS.2018.2851311 ↗
25. Yang, Y., Uy, M.C.S., and Huang, A. (2020). Finbert: A pretrained language model for financial communications. arXiv.
26. Explainable stock prices prediction from financial news articles using sentiment analysis10.7717/peerj-cs.340 ↗
27. Lstm based sentiment analysis of financial news
28. Evaluation of Sentiment Analysis in Finance: From Lexicons to Transformers10.1109/ACCESS.2020.3009626 ↗
29. Financial Market Sentiment Analysis and Investment Strategy Formulation Based on Social Network Data
30. FinSentGPT: A universal financial sentiment engine?10.1016/j.irfa.2024.103291 ↗
31. A survey of sentiment analysis in social media10.1007/s10115-018-1236-4 ↗
32. Sentiment analysis of short informal texts10.1613/jair.4272 ↗
33. Louviere, J.J., and Woodworth, G.G. (1991). Best-Worst Scaling: A Model for the Largest Difference Judgments, Cambridge University Press. Technical Report, Working Paper.
34. Bradley, M.M., Lang, P.J., and Affective norms for English words (ANEW): Instruction Manual and Affective Ratings (2024, October 01). Technical Report, Technical Report C-1, The Center for Research in Psychophysiology. Available online: https://pdodds.w3.uvm.edu/teaching/courses/2009-08UVM-300/docs/others/everything/bradley1999a.pdf.
35. Årup Nielsen, F. (2011). A new ANEW: Evaluation of a word list for sentiment analysis in microblogs. arXiv.
36. Brain death: Diagnostic clues on imaging10.4103/0974-2700.102422 ↗
37. Hutto, C., and Gilbert, E. (2014, January 1–4). Vader: A parsimonious rule-based model for sentiment analysis of social media text. Proceedings of the International AAAI Conference on Web and Social Media, Ann Arbor, MI, USA.10.1609/icwsm.v8i1.14550 ↗
38. Twitter sentiment analysis during covid-19 outbreak with vader
39. Newman, H., and Joyner, D. (2018, January 27–30). Sentiment analysis of student evaluations of teaching. Proceedings of the Artificial Intelligence in Education: 19th International Conference, AIED 2018, London, UK.10.1007/978-3-319-93846-2_45 ↗
40. Twitter sentiment analysis using natural language toolkit and VADER sentiment
41. Explaining sentiment analysis results on social media texts through visualization10.1007/s11042-023-14432-y ↗
42. Using VADER sentiment and SVM for predicting customer response sentiment10.1016/j.eswa.2020.113746 ↗
43. Hu, M., and Liu, B. (2004, January 22–25). Mining and summarizing customer reviews. Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Seattle, DC, USA.10.1145/1014052.1014073 ↗
44. Baccianella, S., Esuli, A., and Sebastiani, F. (2010, January 17–23). Sentiwordnet 3.0: An enhanced lexical resource for sentiment analysis and opinion mining. Proceedings of the LREC, Valletta, Malta.
45. Moshkin, V., Yarushkina, N., and Andreev, I. (2019, January 7–10). The Sentiment Analysis of Unstructured Social Network Data Using the Extended Ontology SentiWordNet. Proceedings of the 2019 12th International Conference on Developments in eSystems Engineering (DeSE), Kazan, Russia.10.1109/DeSE.2019.00110 ↗
46. Sentiment analysis of Amazon products using ensemble machine learning algorithm
47. Psychological research using linguistic inquiry and word count (liwc) and korean linguistic inquiry and word count (kliwc) language analysis methodologies
48. Sentiment analysis on Twitter based on ensemble of psychological and linguistic feature sets10.17694/bajece.419538 ↗
49. The kernel of truth in text-based personality assessment: A meta-analysis of the relations between the Big Five and the Linguistic Inquiry and Word Count (LIWC)10.1037/bul0000381 ↗
50. Chen, F., Li, S., Lin, L., and Huang, X. (2024). Identifying temporal changes in student engagement in social annotation during online collaborative reading. Education and Information Technologies, Springer.10.1007/s10639-024-12494-5 ↗
51. Exploring behavioural patterns and their relationships with social annotation outcomes10.1111/jcal.12958 ↗
52. Boyd, R.L., Ashokkumar, A., Seraj, S., and Pennebaker, J.W. (2022). The Development and Psychometric Properties of LIWC-22, University of Texas at Austin.
53. Rentoumi, V., Giannakopoulos, G., Karkaletsis, V., and Vouros, G.A. (2009, January 14–16). Sentiment analysis of figurative language using a word sense disambiguation approach. Proceedings of the International Conference RANLP-2009, Borovets, Bulgaria.
54. Agirre, E., Màrquez, L., and Wicentowski, R. SemEval-2007 Task 14: Affective Text. Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007).10.3115/1621474.1621476 ↗
55. Jayakrishnan, R., Gopal, G.N., and Santhikrishna, M. (2018, January 4–6). Multi-class emotion detection and annotation in Malayalam novels. Proceedings of the 2018 International Conference on Computer Communication and Informatics (ICCCI), Tamilnadu, India.10.1109/ICCCI.2018.8441492 ↗
56. The impact of Word Sense Disambiguation on stock price prediction10.1016/j.eswa.2021.115568 ↗
57. Augustyniak, L., Szymański, P., Kajdanowicz, T., and Tuligłowicz, W. (2016). Comprehensive Study on Lexicon-based Ensemble Classification Sentiment Analysis. Entropy, 18.10.3390/e18010004 ↗
58. Sentiment analysis using deep learning architectures: A review10.1007/s10462-019-09794-5 ↗
59. Modupe, A., Celik, T., Marivate, V., and Olugbara, O.O. (2022). Post-authorship attribution using regularized deep neural network. Appl. Sci., 12.10.3390/app12157518 ↗
60. Aldoseri, A., Al-Khalifa, K.N., and Hamouda, A.M. (2023). Re-thinking data strategy and integration for artificial intelligence: Concepts, opportunities, and challenges. Appl. Sci., 13.10.20944/preprints202305.1565.v2 ↗
61. Overcoming the pitfalls and perils of algorithms: A classification of machine learning biases and mitigation methods10.1016/j.jbusres.2022.01.076 ↗
62. Zhao, P., Zhang, H., Yu, Q., Wang, Z., Geng, Y., Fu, F., Yang, L., Zhang, W., Jiang, J., and Cui, B. (2024). Retrieval-Augmented Generation for AI-Generated Content: A Survey. arXiv.
63. Liu, H., Liu, H., Liu, H., Liu, H., Yin, Q., Yin, Q., Wang, W.Y., and Wang, W.Y. (2019). Towards Explainable NLP: A Generative Explanation Framework for Text Classification. arXiv.10.18653/v1/P19-1560 ↗
64. Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Guo, Q., and Wang, M. (2023). Retrieval-Augmented Generation for Large Language Models: A Survey. arXiv.
65. Fan, W., Ding, Y., Ning, L., Wang, S., Li, H., Yin, D., Chua, T.S., and Li, Q. (2024). A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models. arXiv.10.1145/3637528.3671470 ↗
66. Hu, Y., and Lu, Y. (2024). RAG and RAU: A Survey on Retrieval-Augmented Language Model in Natural Language Processing. arXiv.10.2139/ssrn.5015182 ↗
67. Lewis, P., Lewis, P., Perez, E., Perez, E., Perez, E., Piktus, A., Piktus, A., Piktus, A., Petroni, F., and Petroni, F. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. arXiv.
68. Zhang, B., Yang, H., Zhou, T., Ali Babar, M., and Liu, X.Y. (2023). Enhancing financial sentiment analysis via retrieval augmented large language models. arXiv.10.1145/3604237.3626866 ↗
69. Shivaprasad, T.K., Shivaprasad, T.K., Shetty, J., Shetty, J., and Shetty, J. (2017, January 10–11). Sentiment analysis of product reviews: A review. Proceedings of the International Conference Inventive Communication and Computational Technologies, Coimbatore, India.10.1109/ICICCT.2017.7975207 ↗
70. Zhao, Z., and Welsch, R.E. (2024). Aligning LLMs with Human Instructions and Stock Market Feedback in Financial Sentiment Analysis. arXiv.
71. Vulic, I., Su, P.H., Coope, S., Gerz, D., Budzianowski, P., Casanueva, I., Mrkvsic, N., and Wen, T.H. (2021). ConvFiT: Conversational Fine-Tuning of Pretrained Language Models. arXiv.10.18653/v1/2021.emnlp-main.88 ↗
72. Alghisi, S., Rizzoli, M., Roccabruna, G., Mousavi, S.M., and Riccardi, G. (2024). Should We Fine-Tune or RAG? Evaluating Different Techniques to Adapt LLMs for Dialogue. arXiv.
73. Fütterer, T., Fischer, C., Alekseeva, A., Chen, X., Tate, T., Warschauer, M., and Gerjets, P. (2023). ChatGPT in education: Global reactions to AI innovations. Sci. Rep., 13.10.1038/s41598-023-42227-6 ↗
74. A Hybrid Neural Network Model for Sentiment Analysis of Financial Texts Using Topic Extraction, Pre-Trained Model, and Enhanced Attention Mechanism Methods10.1109/ACCESS.2024.3429150 ↗
75. Ling, T., Chen, L., Liao, C., Huang, S., Yu, Z., and Liu, Y. (November, January 31). Improving Aspect Sentiment Classification via Retrieving from Training Data. Proceedings of the 2023 Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), Taipei, Taiwan.10.1109/APSIPAASC58517.2023.10317451 ↗
76. Predicting microblog sentiments via weakly supervised multimodal deep learning10.1109/TMM.2017.2757769 ↗
77. Alawwad, H., Alhothali, A.M., Naseem, U., Alkhathlan, A., and Jamal, A. (2024). Enhancing Textbook Question Answering Task with Large Language Models and Retrieval Augmented Generation. arXiv.10.2139/ssrn.4761601 ↗
78. Information retrieval from textual data: Harnessing large language models, retrieval augmented generation and prompt engineering10.69554/QAFE6376 ↗
79. Wu, S., Xiong, Y., Cui, Y., Wu, H., Chen, C., Yuan, Y., Huang, L., Liu, X., Kuo, T.W., and Guan, N. (2024). Retrieval-Augmented Generation for Natural Language Processing: A Survey. arXiv.
80. Miao, J., Thongprayoon, C., Suppadungsuk, S., Garcia Valencia, O.A., and Cheungpasitporn, W. (2024). Integrating retrieval-augmented generation with large language models in nephrology: Advancing practical applications. Medicina, 60.10.3390/medicina60030445 ↗
81. Long, Q., Wang, W., and Pan, S. (2023, January 6–10). Adapt in Contexts: Retrieval-Augmented Domain Adaptation via In-Context Learning. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore.10.18653/v1/2023.emnlp-main.402 ↗
82. Pathak, A., Shree, O., Agarwal, M., Sarkar, S.D., and Tiwary, A. (2023, January 18–20). Performance Analysis of LoRA Finetuning Llama-2. Proceedings of the 2023 7th International Conference on Electronics, Materials Engineering & Nano-Technology (IEMENTech), Kolkata, India.10.1109/IEMENTech60402.2023.10423400 ↗
83. Huang, Y., and Huang, J. (2024). A Survey on Retrieval-Augmented Text Generation for Large Language Models. arXiv.
84. Sudirjo, F., Diantoro, K., Al-Gasawneh, J.A., Azzaakiyyah, H.K., and Ausat, A.M.A. (2023). Application of ChatGPT in Improving Customer Sentiment Analysis for Businesses. J. Teknol. Dan Sist. Inf. Bisnis.10.47233/jteksis.v5i3.871 ↗
85. Es, S., James, J., Espinosa-Anke, L., and Schockaert, S. (2024, January 17–22). RAGAS: Automated Evaluation of Retrieval Augmented Generation. Proceedings of the Conference of the European Chapter of the Association for Computational Linguistics, St. Julians, Malta.
86. Yang, L., Yao, Y., Ton, J.F., Zhang, X., Cheng, R., Klochkov, Y., Taufiq, M.F., and Li, H. (2023). Trustworthy LLMs: A Survey and Guideline for Evaluating Large Language Models’ Alignment. arXiv.
87. Salemi, A., and Zamani, H. (2024, January 14–18). Evaluating retrieval quality in retrieval-augmented generation. Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, Washington, DC, USA.10.1145/3626772.3657957 ↗
88. Xia, Y., Zhou, J., Shi, Z., Chen, J., and Huang, H. (2024). Improving Retrieval Augmented Language Model with Self-Reasoning. arXiv.
89. A survey on evaluation of large language models10.1145/3641289 ↗
90. Balaguer, M.A.D.L., Benara, V., Cunha, R.L.D.F., Filho, R.D.M.E., Hendry, T., Holstein, D., Marsman, J., Mecklenburg, N., Malvar, S., and Nunes, L. (2024). RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculture. arXiv.
91. Soudani, H., Kanoulas, E., and Hasibi, F. (2024). Fine Tuning vs. Retrieval Augmented Generation for Less Popular Knowledge. arXiv.
92. Danilevsky, M., Danilevsky, M., Dhanorkar, S., Dhanorkar, S., Li, Y., Li, Y., Popa, L., Popa, L., Qian, K., and Qian, K. (2021). Explainability for Natural Language Processing. Knowledge Discovery and Data Mining, Springer.10.1145/3447548.3470808 ↗
93. Sentiment analysis in financial texts10.1016/j.dss.2016.10.006 ↗
94. Jiang, M.T.J., Jiang, M.T.J., Wu, S.H., Wu, S.H., Chen, Y., Chen, Y.K., Gu, Z.X., Gu, Z.X., Chiang, C.J., and Chiang, C.J. (2020, January 7–10). Fine-tuning techniques and data augmentation on transformer-based models for conversational texts and noisy user-generated content. Proceedings of the International Conference on Advances in Social Networks Analysis and Mining, The Hague, Netherlands.10.1109/ASONAM49781.2020.9381329 ↗
95. Agarwal, P., and Gupta, A. (2024, January 19–22). Strategic Business Insights through Enhanced Financial Sentiment Analysis: A Fine-Tuned Llama 2 Approach. Proceedings of the 2024 International Conference on Inventive Computation Technologies (ICICT), London, UK.10.1109/ICICT60155.2024.10544544 ↗
96. Speech emotion recognition and text sentiment analysis for financial distress prediction10.1007/s00521-023-08470-8 ↗
97. Improving sentiment analysis of financial news headlines using hybrid Word2Vec-TFIDF feature extraction technique10.1016/j.procs.2024.10.172 ↗
98. Gururangan, S., Marasović, A., Swayamdipta, S., Lo, K., Beltagy, I., Downey, D., and Smith, N.A. (2020). Don’t stop pretraining: Adapt language models to domains and tasks. arXiv.10.18653/v1/2020.acl-main.740 ↗
99. Ruder, S., Peters, M.E., Swayamdipta, S., and Wolf, T. (2019, January 2–7). Transfer learning in natural language processing. Proceedings of the 2019 conference of the North American Chapter of the Association for Computational Linguistics: Tutorials, Minneapolis, MN, USA.10.18653/v1/N19-5004 ↗
100. Neural Approaches to Conversational AI