journal article·2026-06-11Open access

Transformer–DDQN-Based Explainable and Active Intrusion Detection Architecture for Network Traffic Analysis

Ayşe Okutan Kara, Aytuğ Boyacı

Applied Sciences· Volume 16· Issue 12

Abstract

This study proposes a novel intrusion detection and response architecture that formulates network traffic analysis as a sequential decision-making problem rather than a static classification task. The architecture integrates a Transformer Encoder for temporal feature extraction with a Dueling Double Deep Q-Network (DDQN) to enable autonomous and risk-aware security decisions. Network flows are modeled within a Markov Decision Process, where the agent learns an optimal policy over a hierarchical action space consisting of IGNORE, LOG, ESCALATE, and BLOCK actions. To evaluate generalization capability, a transfer learning-based cross-domain adaptation strategy was employed. The CICIDS2018 and CICIoT2023 datasets were re-partitioned using a stratified 70/15/15 train/validation/test split. The proposed model achieved high detection performance on these datasets with F1-scores of 99.48% and 99.13%, respectively. After transfer learning to the AWID3 dataset, the model preserved strong generalization capability with F1-scores of 96.76% and 96.61%, demonstrating its robustness across wired, IoT, and wireless network environments. A risk-aware reward function is designed to balance detection accuracy and operational cost, while Integrated Gradients-based explainability is incorporated to analyze decision behavior. Experimental results further show that the proposed Transformer–DDQN framework achieves more stable learning, lower optimization loss, and more consistent action policies compared to alternative reinforcement learning-based approaches. The model operates with high computational efficiency while maintaining real-time processing capability in high-throughput network environments.
Source: crossref · Quality: baseline

Research context

OpenAlex topics: Network Security and Intrusion Detection · Internet Traffic Analysis and Secure E-voting · Network Packet Processing and Optimization

SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.

Affiliations

Department of Computer Engineering, National Defence University, Istanbul 34149, Türkiye

References (48)

Show source-provided references
1. A Survey on the Applications of Deep Learning in Network Intrusion Detection Systems to Enhance Network Security10.1109/ACCESS.2025.3624952 ↗
2. Tarafdar, R., Singh, H., Pahuja, V., Garg, G., Sivaraman, R., and Jegajothi, B. (2025). Deep Reinforcement Learning for Intelligent Cybersecurity in Smart City IoT Infrastructures. Proceedings of the 2025 6th International Conference on Inventive Research in Computing Applications (ICIRCA), IEEE.10.1109/ICIRCA65293.2025.11089735 ↗
3. Huang, W. (2025). Research on Computer Network Monitoring System Based on Artificial Intelligence. Proceedings of the 2025 International Symposium on Intelligent Robotics and Systems (ISoIRS), IEEE.10.1109/ISoIRS65690.2025.11167991 ↗
4. Smart Grid Security: Proactive Prediction of Advanced Persistent Threats
5. RTIDS: A Robust Transformer-Based Approach for Intrusion Detection System10.1109/ACCESS.2022.3182333 ↗
6. Explainability of Network Intrusion Detection Using Transformers: A Packet-Level Approach10.1109/ACCESS.2024.3525008 ↗
7. AboulEla, S., and Kashef, R. (2025). Enhancing Iot Intrusion Detection with Transformer-Based Network Traffic Classification. Proceedings of the 2025 IEEE International Systems Conference (SysCon), IEEE.10.1109/SysCon64521.2025.11014861 ↗
8. Design of a Multimodal Network Data Security Detection Model Based on Self-Supervised Learning10.1002/spy2.70062 ↗
9. IDS-INT: Intrusion Detection System Using Transformer-Based Transfer Learning for Imbalanced Network Traffic10.1016/j.dcan.2023.03.008 ↗
10. A Survey for Deep Reinforcement Learning Based Network Intrusion Detection10.1002/ail2.70026 ↗
11. Shaikh, J.A., Wang, C., Sima, M.W.U., Arshad, M., Owais, M., Hassan, D.S., Alkanhel, R., and Muthanna, M.S.A. (2025). A Deep Reinforcement Learning-Based Robust Intrusion Detection System for Securing IoMT Healthcare Networks. Front. Med., 12.10.3389/fmed.2025.1524286 ↗
12. Maliha, M., and Atiquzzaman, M. (2025). Q-ID: A Reinforcement Learning Framework for Adaptive Intrusion Detection. Proceedings of the 2025 20th Conference on Computer Science and Intelligence Systems (FedCSIS), IEEE.10.15439/2025F1820 ↗
13. Double Deep Q-Learning with Prioritized Experience Replay for Anomaly Detection in Smart Environments10.1109/ACCESS.2022.3179720 ↗
14. Shi, F., Zhou, Z., Guo, J., Li, R., Zhang, Z., Li, S., Liu, Q., and Bao, X. (2024). Lightrl-Ad: A Lightweight Online Reinforcement Learning Approach for Autonomous Defense against Network Attacks. Proceedings of the 2024 IEEE 23rd International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom), IEEE.10.1109/TrustCom63139.2024.00222 ↗
15. Zegarra Rodriguez, D., Daniel Okey, O., Maidin, S.S., Umoren Udo, E., and Kleinschmidt, J.H. (2023). Attentive Transformer Deep Learning Algorithm for Intrusion Detection on IoT Systems Using Automatic Xplainable Feature Selection. PLoS ONE, 18.10.1371/journal.pone.0286652 ↗
16. Federated Learning with Explainable AI for Malicious Traffic Detection in IoT Networks10.1109/ACCESS.2025.3613459 ↗
17. Explainable Artificial Intelligence for Cybersecurity: A Literature Survey10.1007/s12243-022-00926-7 ↗
18. Cao, B., Li, C., Song, Y., Qin, Y., and Chen, C. (2022). Network Intrusion Detection Model Based on CNN and GRU. Appl. Sci., 12.10.3390/app12094184 ↗
19. AI-Driven Frameworks for Enhancing Cybersecurity in Multi-Cloud Environments
20. Prasanga, D.G.T., Gutierrez, J.A., and Ray, S.K. (2025). The Role of Graph Neural Networks, Transformers, and Reinforcement Learning in Network Threat Detection: A Systematic Literature Review. Electronics, 14.10.3390/electronics14214163 ↗
21. A Federated Transformer-Enhanced Double Q-Network for Collaborative Intrusion Detection10.1038/s41598-025-31664-0 ↗
22. Hasan, M.F., Ahmed, S., and Hasan, M.J. (2025). Entropy Regularized Transformer Based-Anomaly Detection. [Ph.D. Thesis, IUB].
23. Wang, Z., Schaul, T., Hessel, M., Hasselt, H., Lanctot, M., and Freitas, N. (2016). Dueling Network Architectures for Deep Reinforcement Learning. Proceedings of the International Conference on Machine Learning, PMLR.
24. Reinforced Transformer Learning for VSI-DDoS Detection in Edge Clouds10.1109/ACCESS.2022.3204812 ↗
25. Bakhshad, S., Ponnusamy, V., Annur, R., Waqasyz, M., Alasmary, H., and Tux, S. (2022). Deep Reinforcement Learning Based Intrusion Detection System with Feature Selections Method and Optimal Hyper-Parameter in IoT Environment. Proceedings of the 2022 International Conference on Computer, Information and Telecommunication Systems (CITS), IEEE.10.1109/CITS55221.2022.9832976 ↗
26. QATNet: A Lightweight Quantum-Classical Tabular Network for Low-Latency Intrusion Detection10.1109/TQE.2026.3674396 ↗
27. Wei, J. (2024). Research on Computer Network Intrusion Detection Algorithm Based on Deep Learning. Proceedings of the 2024 IEEE 7th International Conference on Automation, Electronics and Electrical Engineering (AUTEEE), IEEE.
28. Janardhanan, H. (2025). A Reinforcement Learning Approach to Cybersecurity: Deep Q-Networks for Threat Modeling. Proceedings of the 2025 International Conference on Machine Learning and Autonomous Systems (ICMLAS), IEEE.10.1109/ICMLAS64557.2025.10968270 ↗
29. Wang, G., Liu, D., Zhang, C., and Hu, T. (2023). Deep Learning-Enabled Heterogeneous Transfer Learning for Improved Network Attack Detection in Internal Networks. Appl. Sci., 13.10.3390/app132112033 ↗
30. Enhanced Intrusion Detection Using Deep Q-Network with Priority Experience Replay and Multi-Agent Reinforcement Learning
31. A Novel Adaptive Transformer-Based Quantum Intrusion Detection System for Software Defined Networks10.1038/s41598-025-20356-4 ↗
32. Deep Transfer Learning Based on Hybrid Swin Transformers with LSTM for Intrusion Detection Systems in IoT Environment10.1109/OJCOMS.2025.3569301 ↗
33. Explainable AI-Driven Intrusion Detection System for DoS Attack Classification Using Deep Learning and Optimization Techniques10.1109/ACCESS.2026.3651187 ↗
34. Ghasemigol, M., and Takabi, D. (2025). Leveraging Transformer Models and eXplainable Reinforcement Learning Methods for Advanced Intrusion Detection and Response System. Proceedings of the 2025 IEEE 7th International Conference on Trust, Privacy and Security in Intelligent Systems, and Applications (TPS-ISA), IEEE.10.1109/TPS-ISA67132.2025.00041 ↗
35. Advanced Deep Reinforcement Learning for Agentic AI and Their Applications in Wireless Network10.1109/TCCN.2026.3683896 ↗
36. Transferability Evaluation in Wi-Fi Intrusion Detection Systems through Machine Learning and Deep Learning Approaches10.1109/ACCESS.2025.3528214 ↗
37. Yang, B., Zhang, G., and Wang, K. (2026). Transformer-Based Multi-Source Transfer Learning for Intrusion Detection Models with Privacy and Efficiency Balance. Entropy, 28.10.3390/e28020136 ↗
38. (2026, June 04). AWID3 Dataset; University of the Aegean, Information and Communication Systems Engineering Department, 2021. Available online: https://icsdweb.aegean.gr/awid/awid3.
39. Neto, E.C.P., Dadkhah, S., Ferreira, R., Zohourian, A., Lu, R., and Ghorbani, A.A. (2023). CICIoT2023: A Real-Time Dataset and Benchmark for Large-Scale Attacks in IoT Environment. Sensors, 23.10.20944/preprints202305.0443.v1 ↗
40. Enhancing Intrusion Detection Systems with Reinforcement Learning: A Comprehensive Survey of Rl-Based Approaches and Techniques10.1007/s42979-024-03001-1 ↗
41. del Real Torres, A., Andreiana, D.S., Ojeda Roldan, A., Hernandez Bustos, A., and Acevedo Galicia, L.E. (2022). A Review of Deep Reinforcement Learning Approaches for Smart Manufacturing in Industry 4.0 and 5.0 Framework. Appl. Sci., 12.10.3390/app122312377 ↗
42. Sundararajan, M., Taly, A., and Yan, Q. (2017). Axiomatic Attribution for Deep Networks. Proceedings of the International Conference on Machine Learning, PMLR.
43. El Gadal, W. (2025). AI-Driven Security in Software-Defined Networks: A Unified Framework for Intrusion Detection and Mitigation. [Ph.D. Thesis, University of Victoria].
44. Kheddar, H., Himeur, Y., and Awad, A.I. (2023). Deep Transfer Learning Applications in Intrusion Detection Systems: A Comprehensive Review. arXiv.10.1016/j.jnca.2023.103760 ↗
45. Anand, L.G. (2025). Reinforcement Learning-Based Intrusion Prevention in M2M Systems. EngrXiv.10.31224/4880 ↗
46. Shaker, B.N., Al-Musawi, B.Q., and Hassan, M.F. (2023, January 18–20). A Comparative Study of Ids-Based Deep Learning Models for IoT Network. Proceedings of the 2023 International Conference on Advances in Artificial Intelligence and Applications, Wuhan China.10.1145/3603273.3635058 ↗
47. Afraji, D.M.A.A., Lloret, J., and Peñalver, L. (2025). An Integrated Hybrid Deep Learning Framework for Intrusion Detection in IoT and IIoT Networks Using CNN-LSTM-GRU Architecture. Computation, 13.10.3390/computation13090222 ↗
48. A Hybrid Approach for Efficient Feature Selection in Anomaly Intrusion Detection for IoT Networks: AG Ayad et Al10.1007/s11227-024-06409-x ↗