Smart Robotic Strategies and Advice for Stock Trading Using Deep Transformer Reinforcement Learning
Source abstract
Abstract
The many success stories of reinforcement learning (RL) and deep learning (DL) techniques have raised interest in their use for detecting patterns and generating constant profits from financial markets. In this paper, we combine deep reinforcement learning (DRL) with a transformer network to develop a decision transformer architecture for online trading. We use data from the Saudi Stock Exchange (Tadawul), one of the largest liquid stock exchanges globally. Specifically, we use the indices of four firms: Saudi Telecom Company, Al-Rajihi Banking and Investment, Saudi Electricity Company, and Saudi Basic Industries Corporation. To ensure the robustness and risk management of the proposed model, we consider seven reward functions: the Sortino ratio, cumulative returns, annual volatility, omega, the Calmar ratio, max drawdown, and normal reward without any risk adjustments. Our proposed DRL-based model provided the highest average increase in the net worth of Saudi Telecom Company, Saudi Electricity Company, Saudi Basic Industries Corporation, and Al-Rajihi Banking and Investment at 21.54%, 18.54%, 17%, and 19.36%, respectively. The Sortino ratio, cumulative returns, and annual volatility were found to be the best-performing reward functions. This work makes significant contributions to trading regarding long-term investment and profit goals.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Stock Market Forecasting Methods · Financial Markets and Investment Strategies · Market Dynamics and Volatility
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Science, Faculty of Computing and Information Technology, King Abdulaziz University, Jeddah 21589, Saudi Arabia
High Performance Computing Center, King Abdulaziz University, Jeddah 21589, Saudi Arabia
References (65)
Show source-provided references
1. Kaya, O., Schildbach, J., AG, D.B., and Schneider, S. (2022, October 20). Robo-Advice—A True Innovation in Asset Management. Available online: https://www.dbresearch.com/PROD/RPS_EN-PROD/PROD0000000000449125/Robo-advice_%E2%80%93_a_true_innovation_in_asset_managemen.PDF.
2. Méndez-Suárez, M., García-Fernández, F., and Gallardo, F. (2019). Artificial Intelligence Modelling Framework for Financial Automated Advising in the Copper Market. J. Open Innov. Technol. Mark. Complex., 5.10.3390/joitmc5040081 ↗
3. So, M.K.P. (2021). Robo-Advising Risk Profiling through Content Analysis for Sustainable Development in the Hong Kong Financial Market. Sustainability, 13.10.3390/su13031306 ↗
4. Reinforcement learning: An Introduction Second edition
5. (2022, October 10). ArgaamPlus. Tadawul’s Market Cap Slips 2.3% to SAR 12.178 trln Last Week. Available online: https://www.argaam.com/en/article/articledetail/id/1559878.
6. (2022, October 10). Capital Market Overview. Available online: https://www.saudiexchange.sa/wps/portal/tadawul/knowledge-center/about/Capital-Market-Overview?locale=en.
7. Wang, Y., Wang, D., Zhang, S., Feng, Y., Li, S., and Zhou, Q. (2017). Deep Q-trading. Cslt. Riit. Tsinghua. Edu. Cn, 1–9.
8. A causal knowledge-based expert system for planning an Internet-based stock trading system10.1016/j.eswa.2012.01.191 ↗
9. Using artificial neural network models in stock market index prediction10.1016/j.eswa.2011.02.068 ↗
10. Selvin, S., Vinayakumar, R., Gopalakrishnan, E.A., Menon, V.K., and Soman, K.P. (2017, January 13–16). Stock price prediction using LSTM, RNN and CNN-sliding window model. Proceedings of the 2017 International Conference on Advances in Computing, Communications and Informatics (ICACCI), Udupi, India.10.1109/ICACCI.2017.8126078 ↗
11. Stock price prediction using DEEP learning algorithm and its comparison with machine learning algorithms10.1002/isaf.1459 ↗
12. Predicting Stock Closing Prices in Emerging Markets with Transformer Neural Networks: The Saudi Stock Exchange Case
13. Performance functions and reinforcement learning for trading systems and portfolios10.1002/(SICI)1099-131X(1998090)17:5/6<441::AID-FOR707>3.0.CO;2-# ↗
14. Xiong, Z., Liu, X.Y., Zhong, S., Yang, H., and Walid, A. (2018). Practical Deep Reinforcement Learning Approach for Stock Trading. arXiv.
15. Gudelek, M.U., Boluk, S.A., and Ozbayoglu, A.M. (December, January 27). A deep learning based stock trading model with 2-D CNN trend detection. Proceedings of the 2017 IEEE Symposium Series on Computational Intelligence (SSCI), Honolulu, HI, USA.10.1109/SSCI.2017.8285188 ↗
16. Deep Direct Reinforcement Learning for Financial Signal Representation and Trading10.1109/TNNLS.2016.2522401 ↗
17. A novel CNN-DDPG based AI-trader: Performance and roles in business operations10.1016/j.tre.2019.09.013 ↗
18. Application of deep reinforcement learning in stock trading strategies and stock forecasting10.1007/s00607-019-00773-w ↗
19. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017, January 4–9). Attention is all you need. Proceedings of the Advances in Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.
20. Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. (2013). Playing Atari with Deep Reinforcement Learning. arXiv.
21. Human-level control through deep reinforcement learning10.1038/nature14236 ↗
22. An Adaptive Financial Trading System Using Deep Reinforcement Learning with Candlestick Decomposing Features10.1109/ACCESS.2020.2982662 ↗
23. Zarkias, K.S., Passalis, N., Tsantekidis, A., and Tefas, A. (2019, January 12–17). Deep reinforcement learning for financial trading using price trailing. Proceedings of the ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brighton, UK.10.1109/ICASSP.2019.8683161 ↗
24. Learning financial asset-specific trading rules via deep reinforcement learning10.1016/j.eswa.2022.116523 ↗
25. Thomas, G. (2022, October 16). Reinforcement learning in financial markets—A survey, FAU Discussion Papers in Economics, No. 12/2018, Friedrich-Alexander-Universität Erlangen-Nürnberg, Institute for Economics, Nürnberg. Available online: https://ideas.repec.org/p/zbw/iwqwdp/122018.html.
26. Dang, Q.V. (2022, October 16). Reinforcement Learning in Stock Trading. International Conference on Computer Science, Applied Mathematics and Applications Hanoi, Vietnam. 2019. hal-02306522. Available online: https://link.springer.com/chapter/10.1007/978-3-030-38364-0_28.
27. Improving financial trading decisions using deep Q-learning: Predicting the number of shares, action strategies, and transfer learning10.1016/j.eswa.2018.09.036 ↗
28. Yang, H., Liu, X.Y., Zhong, S., and Walid, A. (2020, January 15–16). Deep reinforcement learning for automated stock trading. Proceedings of the First ACM International Conference on AI in Finance, New York, NY, USA.10.1145/3383455.3422540 ↗
29. Deep Robust Reinforcement Learning for Practical Algorithmic Trading10.1109/ACCESS.2019.2932789 ↗
30. Decision transformer: Reinforcement learning via sequence modeling
31. Offline reinforcement learning as one big sequence modeling problem
32. Sadighian, J. (2020). Extending Deep Reinforcement Learning Frameworks in Cryptocurrency Market Making. arXiv.
33. Gašperov, B., Begušić, S., Šimović, P.P., and Kostanjčar, Z. (2021). Reinforcement learning approaches to optimal market making. Mathematics, 9.10.3390/math9212689 ↗
34. The sharpe ratio10.1515/9781400829408-022 ↗
35. Jiang, Z., Xu, D., and Liang, J. (2017). A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problem. arXiv.
36. Action-specialized expert ensemble trading system with extended discrete action space using deep reinforcement learning10.1371/journal.pone.0236178 ↗
37. Millea, A. (2021). Deep reinforcement learning for trading—A critical survey. Data, 6.10.20944/preprints202111.0044.v1 ↗
38. Yashaswi, K. (2021). Deep Reinforcement Learning for Portfolio Optimization using Latent Feature State Space (LFSS) Module. arXiv.
39. Bu, S.J., and Cho, S.B. (2018). Learning Optimal Q-Function Using Deep Boltzmann Machine for Reliable Trading of Cryptocurrency, Springer International Publishing.10.1007/978-3-030-03493-1_49 ↗
40. Commission Fee is not Enough: A Hierarchical Reinforced Framework for Portfolio Management
41. Deep Reinforcement Learning for Trading10.3905/jfds.2020.1.030 ↗
42. (2022, October 16). Argaam (Ed.) Al Rajhi Bank’s Board Proposes 60% Capital Increase to SAR 40 bln via Bonus Issue, ArgaamPlus, 20 February 2022. Available online: https://www.argaam.com/en/article/articledetail/id/1536623.
43. Data preparation for data mining10.1080/713827180 ↗
44. Data Cleaning: Problems and Current Approaches
45. Missing value imputation on missing completely at random data using multilayer perceptrons10.1016/j.neunet.2010.09.008 ↗
46. The use of multiple imputation for the analysis of missing data10.1037/1082-989X.6.4.317 ↗
47. Discrepancy-Based Theory and Algorithms for Forecasting Non-Stationary Time Series10.1007/s10472-019-09683-1 ↗
48. Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning, MIT Press.
49. Distribution of the Estimators for Autoregressive Time Series with a Unit Root
50. Raymaekers, J., and Rousseeuw, P.J. (2021). Transforming variables to central normality. Mach. Learn.10.1007/s10994-021-05960-5 ↗
51. An Ensembling Architecture Incorporating Machine Learning Models and Genetic Algorithm Optimization for Forex Trading10.3390/fintech1020008 ↗
52. Smyl, S., and Kuber, K. (2022, October 16). Data Preprocessing and Augmentation for Multiple Short Time Series Forecasting with Recurrent Neural Networks 2016; Technical Report; In 36th International Symposium on Forecasting. Available online: https://www.researchgate.net/publication/309385800_Data_Preprocessing_and_Augmentation_for_Multiple_Short_Time_Series_Forecasting_with_Recurrent_Neural_Networks.
53. An ensemble of LSTM neural networks for high-frequency stock market classification10.1002/for.2585 ↗
54. Macd—Analysis of Weaknesses of the Most Powerful Technical Analysis Tool10.14807/ijmp.v7i2.415 ↗
55. Padial, D.L. (2022, October 06). Technical Analysis Library Using Pandas and Numpy. Available online: https://github.com/bukosabino/ta.
56. Towards automated machine learning: Evaluation and comparison of AutoML approaches and tools
57. Dabérius, K., Granat, E., and Karlsson, P. (2019). Deep Execution—Value and Policy Based Reinforcement Learning for Trading and Beating Market Benchmarks. SSRN Electron. J.
58. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting
59. Informer: Beyond efficient transformer for long sequence time-series forecasting
60. Yarats, D., Brandfonbrener, D., Liu, H., Laskin, M., Abbeel, P., Lazaric, A., and Pinto, L. (2022). Don’t Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learning. arXiv.
61. Liu, X.y., Xia, Z., Rui, J., Gao, J., and Yang, H. (2022). FinRL-Meta: Market Environments and Benchmarks for Data-Driven Financial Reinforcement Learning. arXiv.10.2139/ssrn.4253139 ↗
62. Cobbe, K., Klimov, O., Hesse, C., Kim, T., and Schulman, J. (2019, January 10–15). Quantifying generalization in reinforcement learning. Proceedings of the International Conference on Machine Learning PMLR, Long Beach, CA, USA.
63. Lee, K., Lee, K., Shin, J., and Lee, H. (2019). Network randomization: A simple technique for generalization in deep reinforcement learning. arXiv.
64. Huotari, T., Savolainen, J., and Collan, M. (2020). Deep reinforcement learning agent for S&P 500 stock selection. Axioms, 9.10.3390/axioms9040130 ↗
65. An application of deep reinforcement learning to algorithmic trading10.1016/j.eswa.2021.114632 ↗