journal article·2026-02-24Open access

Enterprise E-Mail Classification Using Instruction-Following Large Language Models

Ahmet Çağrı Sarıyıldız, Şafak Durukan-Odabaşı

Applied Sciences· Volume 16· Issue 5

Abstract

Enterprise e-mail corpora contain heterogeneous and domain-specific content that poses challenges for conventional supervised Natural Language Processing (NLP) approaches due to class imbalance, evolving terminology, and limited labeled data. This study examines the use of instruction-following Large Language Models (LLMs) for enterprise e-mail classification under realistic operational conditions. The study evaluates instruction-based classification and semantic enrichment derived from distributional similarity as two complementary approaches for distinguishing technical from nontechnical messages. The approaches are assessed on a large-scale enterprise e-mail corpus and validated using a manually annotated subset. The results indicate that instruction-following LLMs provide stable contextual reasoning across diverse message structures, while semantic enrichment improves coverage of previously unseen technical expressions. Overall, the study presents an applied NLP framework for enterprise e-mail classification, with attention to interpretability, scalability, and robustness in real-world organizational settings.
Source: crossref · Quality: baseline

Research context

OpenAlex topics: Text and Document Classification Technologies · Spam and Phishing Detection · Authorship Attribution and Profiling

SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.

Affiliations

Department of Cybersecurity, Bahçeşehir University, Istanbul 34349, Türkiye

Department of Computer Engineering, Istanbul University-Cerrahpaşa, Istanbul 34320, Türkiye

References (29)

Show source-provided references
1. (2025, September 25). Enron e-Mail Dataset. Carnegie Mellon University. Available online: https://www.cs.cmu.edu/~enron/.
2. Klimt, B., and Yang, Y. (2004). The Enron Corpus: A New Dataset for e-mail Classification Research. Proceedings of the 15th European Conference on Machine Learning (ECML 2004), Pisa, Italy, 20–24 September 2004, Springer.
3. Trask, A., Michalak, P., and Liu, J. (2015). Sense2Vec: A fast and accurate method for word sense disambiguation in neural word embeddings. arXiv.
4. Scaling instruction-finetuned language models
5. Spam e-mail classification based on cybersecurity potential risk using natural language processing10.1016/j.knosys.2024.112939 ↗
6. Enhancing spam detection with GANs and BERT embeddings: A novel approach to imbalanced datasets10.1016/j.procs.2024.05.049 ↗
7. Email spam detection by deep learning models using novel feature selection technique and BERT10.1016/j.eij.2024.100473 ↗
8. Anomaly Detection: A Survey10.1145/1541880.1541882 ↗
9. A Survey of Data Mining and Machine Learning Methods for Cyber Security Intrusion Detection10.1109/COMST.2015.2494502 ↗
10. Character-level convolutional networks for text classification
11. Alshingiti, Z., Alaqel, R., Al-Muhtadi, J., Haq, Q.E.U., Saleem, K., and Faheem, M.H. (2023). A deep learning-based phishing detection system using CNN, LSTM, and LSTM-CNN. Electronics, 12.10.3390/electronics12010232 ↗
12. Atawneh, S., and Aljehani, H. (2023). Phishing e-mail detection model using deep learning. Electronics, 12.10.3390/electronics12204261 ↗
13. A survey on deep learning for named entity recognition10.1109/TKDE.2020.2981314 ↗
14. Sammet, J., and Krestel, R. (2023, January 12–15). Domain-specific keyword extraction using BERT. Proceedings of the 4th Conference on Language, Data and Knowledge, Vienna, Austria.
15. Automated phrase mining from massive text corpora10.1109/TKDE.2018.2812203 ↗
16. Conphrase: Enhancing context-aware phrase mining from text corpora
17. Altuncu, E., Nurse, J.R., Xu, Y., Guo, J., and Li, S. (2025). Improving performance of automatic keyword extraction (AKE) methods using PoS tagging and enhanced semantic-awareness. Information, 16.10.3390/info16070601 ↗
18. Impact analysis of keyword extraction using contextual word embedding10.7717/peerj-cs.967 ↗
19. Yu, B., Hu, Y., Mang, Q., Hu, W., and He, P. (2023, January 3–9). Automated testing and improvement of named entity recognition systems. Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, San Francisco, CA, USA.10.1145/3611643.3616295 ↗
20. Labonne, M., and Moran, S. (2023). Spam-T5: Benchmarking large language models for few-shot e-mail spam detection. arXiv.
21. Patel, H., Rehman, U., and Iqbal, F. (2024, January 8–11). Evaluating the efficacy of large language models in identifying phishing attempts. Proceedings of the 16th International Conference on Human System Interaction (HSI 2024), Paris, France.10.1109/HSI61632.2024.10613528 ↗
22. Altwaijry, N., Al-Turaiki, I., Alotaibi, R., and Alakeel, F. (2024). Advancing phishing e-mail detection: A comparative study of deep learning models. Sensors, 24.10.3390/s24072077 ↗
23. Tarapiah, S., Abbas, L., Mardawi, O., Atalla, S., Himeur, Y., and Mansoor, W. (2025). Evaluating the effectiveness of large language models (LLMs) versus machine learning (ML) in identifying and detecting phishing e-mail attempts. Algorithms, 18.10.3390/a18100599 ↗
24. An LLM driven framework for email spam detection using DistilBERT embeddings and neural classifiers10.1007/s42452-025-08147-y ↗
25. Kaur, S., Smiley, C., Ramani, K., Kochkina, E., Sibue, M., Mensah, S., Totis, P., Tilli, C., Aguda, T., and Borrajo, D. (2025). Advanced Messaging Platform (AMP): Pipeline for Automated Enterprise Email Processing. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track), Vienna, Austria, 27 July–1 August 2025, Association for Computational Linguistics.10.18653/v1/2025.acl-industry.57 ↗
26. Kumar, R., Ishan, K., Kumar, H., and Singla, A. (2025). LLM-Powered Knowledge Graphs for Enterprise Intelligence and Analytics. arXiv.10.1109/RAIT65068.2025.11088890 ↗
27. Mariotti, L., Guidetti, V., Mandreoli, F., Belli, A., and Lombardi, P. (2024). Combining large language models with enterprise knowledge graphs: A perspective on enhanced natural language understanding. Front. Artif. Intell., 7.10.3389/frai.2024.1460065 ↗
28. Current and Future Use of Large Language Models for Knowledge Work10.1145/3757403 ↗
29. Efficient string matching: An aid to bibliographic search10.1145/360825.360855 ↗