FPE–Transformer: A Feature Positional Encoding-Based Transformer Model for Attack Detection
Source abstract
Abstract
The increase in cybersecurity threats has made attack detection systems critically important. Traditional deep learning methods often require large amounts of data and struggle to understand relationships between features effectively. With their self-attention mechanism, Transformers excel in modeling complex relationships and long-term dependencies. They are also adaptable to various data types and sources, making them advantageous in large-scale attack detection scenarios. This paper introduces the FPE–Transformer framework, leveraging the strengths of the Transformer architecture. FPE–Transformer incorporates an innovative feature positional encoding mechanism that encodes the positional information of each feature separately, enabling a deeper understanding of feature relationships and more precise attack detection. Additionally, the model includes a ClassificationHead for enhanced accuracy and complex pattern recognition. The framework’s performance was validated using the NSL-KDD and CIC-IDS2017 datasets, demonstrating its superiority over traditional methods in detecting diverse attack types and improving overall performance. This study highlights FPE–Transformer’s innovative approach and ability to address key limitations of traditional deep learning methods, establishing it as a robust solution for modern attack detection challenges.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Malware Detection Techniques · Network Security and Intrusion Detection · Anomaly Detection Techniques and Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Engineering, Eskisehir Osmangazi University, Eskisehir 26040, Türkiye
References (65)
Show source-provided references
1. Deep learning based attack detection for cyber-physical system cybersecurity: A survey10.1109/JAS.2021.1004261 ↗
2. Liu, H., and Lang, B. (2019). Machine learning and deep learning methods for intrusion detection systems: A survey. Appl. Sci., 9.10.3390/app9204396 ↗
3. Long short-term memory neural network-based attack detection model for in-vehicle network security10.1109/LSENS.2020.2993522 ↗
4. Zhao, Y., Jia, X., An, D., and Yang, Q. (2020, January 24–26). LSTM-Based false data injection attack detection in smart grids. Proceedings of the 35th Youth Academic Annual Conference of Chinese Association of Automation (YAC), Virtual Conference, China.10.1109/YAC51587.2020.9337674 ↗
5. Muhuri, P.S., Chatterjee, P., Yuan, X., Roy, K., and Esterline, A. (2020). Using a long short-term memory recurrent neural network (LSTM-RNN) to classify network attacks. Information, 11.10.3390/info11050243 ↗
6. Hwang, R.H., Peng, M.C., Nguyen, V.L., and Chang, Y.L. (2019). An LSTM-based deep learning approach for classifying malicious traffic at the packet level. Appl. Sci., 9.10.3390/app9163414 ↗
7. Botnet Attack Detection by Using CNN-LSTM Model for Internet of Things Applications10.1155/2021/3806459 ↗
8. Cao, B., Li, C., Song, Y., Qin, Y., and Chen, C. (2022). Network intrusion detection model based on CNN and GRU. Appl. Sci., 12.10.3390/app12094184 ↗
9. Zainel, H., and Koçak, C. (2022). LAN intrusion detection using convolutional neural networks. Appl. Sci., 12.10.3390/app12136645 ↗
10. Network intrusion detection based on multi-domain data and ensemble-bidirectional LSTM10.1186/s13635-023-00139-y ↗
11. Liao, D., Huang, S., Tan, Y., and Bai, G. (2020, January 21–23). Network intrusion detection method based on gan model. Proceedings of the 2020 International Conference on Computer Communication and Network Security (CCNS), Guilin, China.10.1109/CCNS50731.2020.00041 ↗
12. End-to-end transformer-based models in textual-based NLP10.3390/ai4010004 ↗
13. An, H., Ma, R., Yan, Y., Chen, T., Zhao, Y., Li, P., and Lv, C. (2024). Finsformer: A Novel Approach to Detecting Financial Attacks Using Transformer and Cluster-Attention. Appl. Sci., 14.10.3390/app14010460 ↗
14. Chen, C., Song, Y., Yue, S., Xu, X., Zhou, L., Lv, Q., and Yang, L. (2022). Fcnn-se: An intrusion detection model based on a fusion CNN and stacked ensemble. Appl. Sci., 12.10.3390/app12178601 ↗
15. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017). Attention is all you need. Adv. Neural Inf. Process. Syst., 30.
16. Li, Z., Gao, Z., Tan, C., Ren, B., Yang, L.T., and Li, S.Z. (2024, January 16–22). General Point Model Pretraining with Autoencoding and Autoregressive. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR52733.2024.01980 ↗
17. Kenton, J.D.M.W.C., and Toutanova, L.K. (2019, January 2–7). Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the NAACL-HLT, Minneapolis, MN, USA.
18. Radford, A. (2018). Improving Language Understanding by Generative Pre-Training, OpenAI.
19. BioBERT: A pre-trained biomedical language representation model for biomedical text mining10.1093/bioinformatics/btz682 ↗
20. An ensemble model for classifying idioms and literal texts using BERT and RoBERTa10.1016/j.ipm.2021.102756 ↗
21. Lan, Z.Z., Chen, M.D., Goodman, S., Gimpel, K., Sharma, P., and Soricut, R. (2020, January 26–30). ALBERT: A lite BERT for self-supervised learning of language representations. Proceedings of the 8th International Conference on Learning Representations, Addis Ababa, Ethiopia.
22. Performance comparison of transformer-based models on twitter health mention classification10.1109/TCSS.2022.3143768 ↗
23. He, P., Liu, X., Gao, J., and Chen, W. (2020). DeBERTa: Decoding-enhanced BERT with disentangled attention. arXiv.
24. Clark, K. (2020). Electra: Pre-training text encoders as discriminators rather than generators. arXiv.
25. Language models are unsupervised multitask learners
26. Wang, B., Chen, W., Pei, H., Xie, C., Kang, M., Zhang, C., and Li, B. (2023, January 12–14). DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models. Proceedings of the NeurIPS, New Orleans, LA, USA.
27. Exploring the limits of transfer learning with a unified text-to-text transformer
28. Khashabi, D., Min, S., Khot, T., Sabharwal, A., Tafjord, O., Clark, P., and Hajishirzi, H. (2020). Electra: Unifiedqa: Crossing format boundaries with a single qa system. arXiv.10.18653/v1/2020.findings-emnlp.171 ↗
29. Wang, Y., Wang, W., Joty, S., and Hoi, S.C. (2021). Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. arXiv.10.18653/v1/2021.emnlp-main.685 ↗
30. Lewis, M.C. (2019). Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv.10.18653/v1/2020.acl-main.703 ↗
31. Liu, Y. (2020). Multilingual denoising pre-training for neural machine translation. arXiv.10.1162/tacl_a_00343 ↗
32. Ahmad, W.U., Chakraborty, S., Ray, B., and Chang, K.W. (2021). Unified pre-training for program understanding and generation. arXiv.10.18653/v1/2021.naacl-main.211 ↗
33. Chouikhi, H., Jarray, F., and Alsuhaibani, M. (2023). ICAART, Science and Technology Publications, Ltd.
34. Kheddar, H. (2024). Transformers and Large Language Models for Efficient Intrusion Detection Systems: A Comprehensive Survey. arXiv.
35. Intrusion detection method using bi-directional GPT for in-vehicle controller area networks10.1109/ACCESS.2021.3110524 ↗
36. Hsiao, W.C., and Wang, C.H. (2023, January 23–25). Detection of SQL Injection and Cross-Site Scripting Based on Multi-Model CNN Combined with Bidirectional GRU and Multi-Head Self-Attention. Proceedings of the 2023 5th International Conference on Computer Communication and the Internet (ICCCI), Fujisawa, Japan.10.1109/ICCCI59363.2023.10210155 ↗
37. R1dit: Privacy-preserving malware traffic classification with attention-based neural networks10.1109/TNSM.2022.3211254 ↗
38. Transformer-based attention network for in-vehicle intrusion detection10.1109/ACCESS.2023.3282110 ↗
39. Dang, Q.V. (2023, January 22–24). Using Transformer Technique for Intrusion Detection. Proceedings of the International Conference on Future Data and Security Engineering, Singapore.10.1007/978-981-99-8296-7_13 ↗
40. Lan, Y., Truong Huu, T., Wu, J., and Teo, S.G. (December, January 28). Cascaded multi-class network intrusion detection with decision tree and self-attentive model. Proceedings of the 2022 IEEE International Conference on Data Mining Workshops (ICDMW), Orlando, FL, USA.10.1109/ICDMW58026.2022.00081 ↗
41. Ahmed, S.W., Kientz, F., and Kashef, R. (2023, January 18–20). A modified transformer neural network (MTNN) for robust intrusion detection in IoT networks. Proceedings of the 2023 International Telecommunications Conference (ITC-Egypt), Alexandria, Egypt.10.1109/ITC-Egypt58155.2023.10206134 ↗
42. A Densely Stacked Attention Method for Cyberattack Detection
43. A CNN-transformer hybrid approach for an intrusion detection system in advanced metering infrastructure10.1007/s11042-022-14121-2 ↗
44. He, G., Lu, Q., Yin, G., and Xiong, H. (2022, January 24–26). Network intrusion detection based on hybrid neural network. Proceedings of the International Conference on Wireless Algorithms, Systems, and Applications, Dalian, China.10.1007/978-3-031-19214-2_54 ↗
45. Network intrusion detection based on the temporal convolutional model10.1016/j.cose.2023.103465 ↗
46. Wang, H., and Li, W. (2021). DDosTC: A transformer-based network attack detection hybrid mechanism in SDN. Sensors, 21.10.3390/s21155047 ↗
47. Baul, A., Sarker, G.C., Sadhu, P.K., Yanambaka, V.P., and Abdelgawad, A. (2023). XTM: A novel transformer and LSTM-based model for detection and localization of formally verified fdi attack in smart grid. Electronics, 12.10.3390/electronics12040797 ↗
48. MF-Net: Multi-frequency intrusion detection network for Internet traffic data10.1016/j.patcog.2023.109999 ↗
49. IQA Vision Transformed: A Survey of Transformer Architectures in Perceptual Image Quality Assessment10.1109/ACCESS.2024.3506273 ↗
50. MFVT: An anomaly traffic detection method merging feature fusion network and vision transformer architecture10.1186/s13638-022-02103-9 ↗
51. Network intrusion detection via flow-to-image conversion and vision transformer classification10.1109/ACCESS.2022.3200034 ↗
52. Agrafiotis, G., Makri, E., Flionis, I., Lalas, A., Votis, K., and Tzovaras, D. (2022, January 23–26). Image-based neural network models for malware traffic classification using pcap to picture conversion. Proceedings of the 17th International Conference on Availability, Reliability and Security, Vienna, Austria.10.1145/3538969.3544473 ↗
53. Pre-trained language model-enhanced conditional generative adversarial networks for intrusion detection10.1007/s12083-023-01595-6 ↗
54. Wang, Z., Zhou, J., Wang, Z., and Hei, X. (2022, January 16–18). Research on Network Traffic Anomaly Detection for Class Imbalance. Proceedings of the China Intelligent Robotics Annual Conference, Xi’an, China.10.1007/978-981-99-0301-6_11 ↗
55. RTIDS: A robust transformer-based approach for intrusion detection system10.1109/ACCESS.2022.3182333 ↗
56. An attack detection framework based on BERT and deep learning10.1109/ACCESS.2022.3185748 ↗
57. A security model based on lightgbm and transformer to protect healthcare systems from cyberattacks10.1109/ACCESS.2022.3172432 ↗
58. Liu, Y., and Wu, L. (2023). Intrusion detection model based on improved transformer. Appl. Sci., 13.10.3390/app13106251 ↗
59. IDS-INT: Intrusion detection system using transformer-based transfer learning for imbalanced network traffic10.1016/j.dcan.2023.03.008 ↗
60. Sharafaldin, I., Lashkari, A.H., and Ghorbani, A.A. (2018, January 22–24). Toward generating a new intrusion detection dataset and intrusion traffic characterization. Proceedings of the 4th International Conference on Information Systems Security and Privacy (ICISSP 2018), Funchal, Portugal.10.5220/0006639801080116 ↗
61. Archibe, U.K. (2024, December 28). NSL Data. Available online: https://web.archive.org/web/20150205070216/http://nsl.cs.unb.ca/NSL-KDD/.
62. Han, H., Wang, W.Y., and Mao, B.H. (2005, January 23–26). Borderline-SMOTE: A new over-sampling method in imbalanced data sets learning. Proceedings of the International Conference on Intelligent Computing, Hefei, China.10.1007/11538059_91 ↗
63. Kubat, M., and Matwin, S. (1997, January 8–12). Addressing the curse of imbalanced training sets: One-sided selection. Proceedings of the of the 14th International Conference on Machine Learning, Nashville, TN, USA.
64. Batal, I., Fradkin, D., Harrison, J., Moerchen, F., and Hauskrecht, M. (2012, January 12–16). Mining recent temporal patterns for event detection in multivariate time series data. Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Beijing, China.10.1145/2339530.2339578 ↗
65. Time series forecasting with neural networks: A comparative study using the air line data10.1111/1467-9876.00109 ↗