journal article·2025-03-20Open access

In-Depth Analysis of Phishing Email Detection: Evaluating the Performance of Machine Learning and Deep Learning Models Across Multiple Datasets

Abeer Alhuzali, Ahad Alloqmani, Manar Aljabri, Fatemah Alharbi

Applied Sciences· Volume 15· Issue 6

Abstract

Phishing emails remain a primary vector for cyberattacks, necessitating advanced detection mechanisms. Existing studies often focus on limited datasets or a small number of models, lacking a comprehensive evaluation approach. This study develops a novel framework for implementing and testing phishing email detection models to address this gap. A total of fourteen machine learning (ML) and deep learning (DL) models are evaluated across ten datasets, including nine publicly available datasets and a merged dataset created for this study. The evaluation is conducted using multiple performance metrics to ensure a comprehensive comparison. Experimental results demonstrate that DL models consistently outperform their ML counterparts in both accuracy and robustness. Notably, transformer-based models BERT and RoBERTa achieve the highest detection accuracies of 98.99% and 99.08%, respectively, on the balanced merged dataset, outperforming traditional ML approaches by an average margin of 4.7%. These findings highlight the superiority of DL in phishing detection and emphasize the potential of AI-driven solutions in strengthening email security systems. This study provides a benchmark for future research and sets the stage for advancements in cybersecurity innovation.
Source: crossref · Quality: baseline

Research context

OpenAlex topics: Spam and Phishing Detection · Misinformation and Its Impacts · Sentiment Analysis and Opinion Mining

SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.

Affiliations

Department of Computer Science, Faculty of Computing and Information Technology, King Abdulaziz University, Jeddah 21589, Saudi Arabia

Computer Science Department, College of Computer Science and Engineering, Taibah University, Yanbu 46522, Saudi Arabia

Funding

Institutional Fund Projects · Ministry of Education and King Abdulaziz University, DSR, Jeddah, Saudi Arabia

References (74)

Show source-provided references
1. Federal Bureau of Investigation (FBI) (2025, March 06). Business Email Compromise, Available online: https://www.fbi.gov/how-we-can-help-you/scams-and-safety/common-frauds-and-scams/business-email-compromise.
2. ZDNet (2025, March 06). Hackers Are Targeting Billions of Gmail Users with a Realistic AI Scam: How You Can Stay Safe. Available online: https://www.zdnet.com/article/hackers-are-targeting-billions-of-gmail-users-with-a-realistic-ai-scam-how-you-can-stay-safe/.
3. AAG IT (2025, March 06). The Latest Phishing Statistics. Available online: https://aag-it.com/the-latest-phishing-statistics/.
4. Verizon (2025, March 06). Data Breach Investigations Report. Available online: https://www.verizon.com/business/resources/reports/dbir/.
5. VIPRE (2025, March 06). Email Threats: Latest Trends Q2 2024. Available online: https://vipre.com/resources/email-threats-latest-trends-q2-2024.
6. New filtering approaches for phishing email10.3233/JCS-2010-0371 ↗
7. Fette, I., Sadeh, N., and Tomasic, A. (2007, January 8–12). Learning to detect phishing emails. Proceedings of the 16th International Conference on World Wide Web, Banff, AB, Canada.10.1145/1242572.1242660 ↗
8. Eze, C.S., and Shamir, L. (2024). Analysis and prevention of AI-based phishing email attacks. Electronics, 13.10.3390/electronics13101839 ↗
9. Agarwal, K., and Kumar, T. (2018, January 14–15). Email spam detection using integrated approach of Naïve Bayes and particle swarm optimization. Proceedings of the 2018 Second International Conference on Intelligent Computing and Control Systems (ICICCS), Madurai, India.10.1109/ICCONS.2018.8662957 ↗
10. Analysis of phishing detection using logistic regression and random forest
11. Classification of phishing email using random forest machine learning technique10.1155/2014/425731 ↗
12. Novel interpretable and robust web-based AI platform for phishing email detection10.1016/j.compeleceng.2024.109625 ↗
13. Evading obscure communication from spam emails10.3934/mbe.2022091 ↗
14. Machine learning and deep learning for phishing email classification using one-hot encoding10.3844/jcssp.2021.610.623 ↗
15. Email phishing: Text classification using natural language processing10.11591/csit.v1i1.pp1-12 ↗
16. Noah, N., Tayachew, A., Ryan, S., and Das, S. (2022, January 23–26). Poster: PhisherCop-An Automated Tool Using ML Classifiers for Phishing Detection. Proceedings of the 43rd IEEE Symposium on Security and Privacy (IEEE S&P 2022), San Francisco, CA, USA.10.2139/ssrn.4096243 ↗
17. Kim, S., Park, J., Ahn, H., and Lee, Y. (2024). Detection of Korean Phishing Messages Using Biased Discriminant Analysis under Extreme Class Imbalance Problem. Information, 15.10.3390/info15050265 ↗
18. Alotaibi, R., Al-Turaiki, I., and Alakeel, F. (2020, January 19–21). Mitigating email phishing attacks using convolutional neural networks. Proceedings of the 2020 3rd International Conference on Computer Applications & Information Security (ICCAIS), Riyadh, Saudi Arabia.10.1109/ICCAIS48893.2020.9096821 ↗
19. Halgaš, L., Agrafiotis, I., and Nurse, J.R. (2019, January 21–24). Catching the Phish: Detecting phishing attacks using recurrent neural networks (RNNs). Proceedings of the Information Security Applications: 20th International Conference, WISA 2019, Jeju Island, Republic of Korea. Revised Selected Papers 20.
20. LSTM based phishing detection for big email data10.1109/TBDATA.2020.2978915 ↗
21. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv.
22. Uddin, M.A., and Sarker, I.H. (2024). An Explainable Transformer-based Model for Phishing Email Detection: A Large Language Model Approach. arXiv.10.2139/ssrn.4785953 ↗
23. Gogoi, B., and Ahmed, T. (2022, January 24–26). Phishing and Fraudulent Email Detection thrgh Transfer Learning using pretrained transformer models. Proceedings of the 2022 IEEE 19th India Council International Conference (INDICON), IEEE, Kochi, India.10.1109/INDICON56171.2022.10040097 ↗
24. Deep learning: A comprehensive overview on techniques, taxonomy, applications and research directions10.1007/s42979-021-00815-1 ↗
25. Islam, M.R., Ahmed, M.U., Barua, S., and Begum, S. (2022). A systematic review of explainable artificial intelligence in terms of different application domains and tasks. Appl. Sci., 12.10.3390/app12031353 ↗
26. A comprehensive dual-layer architecture for phishing and spam email detection10.1016/j.cose.2023.103378 ↗
27. The development of phishing during the COVID-19 pandemic: An analysis of over 1100 targeted domains10.1016/j.cose.2023.103158 ↗
28. Alkhalil, Z., Hewage, C., Nawaf, L., and Khan, I. (2021). Phishing attacks: A recent comprehensive study and a new anatomy. Front. Comput. Sci., 3.10.3389/fcomp.2021.563060 ↗
29. Thakur, K., Ali, M.L., Obaidat, M.A., and Kamruzzaman, A. (2023). A systematic review on deep-learning-based phishing email detection. Electronics, 12.10.3390/electronics12214545 ↗
30. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (1907). RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv.
31. Building an Intelligent Phishing Email Detection System Using Machine Learning and Feature Engineering10.59324/ejaset.2025.3(2).04 ↗
32. Meléndez, R., Ptaszynski, M., and Masui, F. (2024). Comparative Investigation of Traditional Machine-Learning Models and Transformer Models for Phishing Email Detection. Electronics, 13.10.20944/preprints202410.1467.v2 ↗
33. Atawneh, S., and Aljehani, H. (2023). Phishing email detection model using deep learning. Electronics, 12.10.3390/electronics12204261 ↗
34. Applying machine learning and natural language processing to detect phishing email10.1016/j.cose.2021.102414 ↗
35. Phishing email detection using improved RCNN model with multilevel vectors and attention mechanism10.1109/ACCESS.2019.2913705 ↗
36. A memory-based approach to anti-spam filtering for mailing lists10.1023/A:1022948414856 ↗
37. Klimt, B., and Yang, Y. (2004). The enron corpus: A new dataset for email classification research. European Conference on Machine Learning, Springer.10.1007/978-3-540-30115-8_22 ↗
38. (2024, September 20). Apache SpamAssassin. Available online: https://zenodo.org/records/8339691.
39. Craswell, N., De Vries, A.P., and Soboroff, I. (2005, January 15–18). Overview of the TREC 2005 Enterprise Track. Proceedings of the Fourteenth Text REtrieval Conference, TREC 2005, Gaithersburg, MD, USA.10.6028/NIST.SP.500-266.enterprise-overview ↗
40. Bratko, A., Filipic, B., and Zupan, B. (2006, January 14–17). Towards Practical PPM Spam Filtering: Experiments for the TREC 2006 Spam Track. Proceedings of the Fifteenth Text REtrieval Conference, TREC 2006, Gaithersburg, MD, USA.10.6028/NIST.SP.500-272.spam-jozef-stefan-inst.bratko ↗
41. Spam detection using random boost10.1016/j.patrec.2012.03.012 ↗
42. Nazario, J. (2024, September 20). The Online Phishing Corpus. Available online: http://monkey.org/~jose/wiki/doku.php.
43. Radev, D. (2024, December 12). CLAIR Collection of Fraud Email. Available online: https://aclweb.org/aclwiki.
44. A probabilistic approach to automatic keyword indexing. Part I. On the distribution of specialty words in a technical literature10.1002/asi.4630260402 ↗
45. Regression shrinkage and selection via the lasso10.1111/j.2517-6161.1996.tb02080.x ↗
46. Shahrivari, V., Darabi, M.M., and Izadi, M. (2020). Phishing detection using machine learning techniques. arXiv.
47. A stochastic approximation method10.1214/aoms/1177729586 ↗
48. Chen, T., and Guestrin, C. (2016, January 13–17). XGBoost: A Scalable Tree Boosting System. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA.10.1145/2939672.2939785 ↗
49. Comparative study of catboost, xgboost, and lightgbm for enhanced URL phishing detection: A performance assessment
50. A systematic literature review on phishing email detection using natural language processing techniques10.1109/ACCESS.2022.3183083 ↗
51. Random forests10.1023/A:1010933404324 ↗
52. Phishing email detection using natural language processing techniques: A literature survey10.1016/j.procs.2021.05.077 ↗
53. Extremely randomized trees10.1007/s10994-006-6226-1 ↗
54. Champa, A.I., Rabbi, M.F., and Zibran, M.F. (2024, January 13–14). Curated datasets and feature analysis for phishing email detection with machine learning. Proceedings of the 2024 IEEE 3rd International Conference on Computing and Machine Intelligence (ICMI), Mt Pleasant, MI, USA.10.1109/ICMI60790.2024.10585821 ↗
55. Bai, S., Kolter, J.Z., and Koltun, V. (2018). An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv.
56. Kalchbrenner, N., Grefenstette, E., and Blunsom, P. (2014). A convolutional neural network for modelling sentences. arXiv.10.3115/v1/P14-1062 ↗
57. Wang, P., Xu, J., Xu, B., Liu, C., Zhang, H., Wang, F., and Hao, H. (2015, January 26–31). Semantic clustering and convolutional neural network for short text categorization. Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), Beijing, China.10.3115/v1/P15-2058 ↗
58. Set-CNN: A text convolutional neural network based on semantic extension for short text classification10.1016/j.knosys.2022.109948 ↗
59. Moriya, S., and Shibata, C. (2018, January 23–27). Transfer learning method for very deep CNN for text classification and methods for its evaluation. Proceedings of the 2018 IEEE 42nd Annual Computer Software and Applications Conference (COMPSAC), IEEE, Tokyo, Japan.10.1109/COMPSAC.2018.10220 ↗
60. Natural language processing (almost) from scratch
61. Jacovi, A., Shalom, O.S., and Goldberg, Y. (2018). Understanding convolutional neural networks for text classification. arXiv.10.18653/v1/W18-5408 ↗
62. Review of deep learning: Concepts, CNN architectures, challenges, applications, future directions10.1186/s40537-021-00444-8 ↗
63. Hochreiter, S. (1997). Long Short-Term Memory, Neural Computation MIT-Press.10.1162/neco.1997.9.8.1735 ↗
64. Khataei Maragheh, H., Gharehchopogh, F.S., Majidzadeh, K., and Sangar, A.B. (2022). A new hybrid based on long short-term memory network with spotted hyena optimization algorithm for multi-label text classification. Mathematics, 10.10.3390/math10030488 ↗
65. Eang, C., and Lee, S. (2024). Improving the Accuracy and Effectiveness of Text Classification Based on the Integration of the Bert Model and a Recurrent Neural Network (RNN_Bert_Based). Appl. Sci., 14.10.3390/app14188388 ↗
66. Transfer Learning Models for E-mail Classification
67. Sanh, V., Debut, L., Chaumond, J., and Wolf, T. (2019). DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter. arXiv.
68. Fake news detection: Taxonomy and comparative study10.1016/j.inffus.2023.102140 ↗
69. An improved transformer-based model for detecting phishing, spam and ham emails: A large language model approach10.1002/spy2.402 ↗
70. Kipf, T.N., and Welling, M. (2016). Semi-supervised classification with graph convolutional networks. arXiv.
71. Graph convolutional networks for text classification
72. Prasad, R. (February, January India). Phishing Email Detection Using Machine Learning: A Critical Review. Proceedings of the 2024 IEEE International Conference on Computing, Power and Communication Technologies (IC2PCT), IEEE, Greater Noida.
73. Miltchev, R., Dimitar, R., and Evgeni, G. (2025, January 20). Phishing Validation Emails Dataset. Zenodo. Available online: https://zenodo.org/records/13474746.
74. (2025, March 05). Microsoft Defender. Available online: https://www.microsoft.com/en-us/security/business/microsoft-defender.