Unsupervised Learning Framework for Cyber Threat Detection, Anomaly Identification, and Alert Prioritization

Conventional Security Operations Center (SOC) solutions struggle to process representative operational alert streams efficiently and adapt to evolving cyber threats, highlighting the need for automated, intelligent threat detection and prioritization. This study presents a custom AI-driven framework that leverages unsupervised learning techniques to support SOC analysts in cyber threat detection, anomaly identification, and alert prioritization. The framework applies several clustering methods: HDBSCAN, DBSCAN, KMeans, and Gaussian Mixture Models for alert segmentation, and integrates anomaly detection using LOF and Isolation Forest, complemented by semi-supervised detection via One-Class SVM. Using textual, categorical, and numerical features from Wazuh alerts across three datasets, the system performs clustering and anomaly detection in the original high-dimensional feature space, with UMAP applied solely for two-dimensional visualization. HDBSCAN consistently produced well-separated clusters with effective noise detection, while, Isolation Forest evaluated via 10-fold cross-validation exhibited stable anomaly flagging and clear score separation across both cyber alert event data and synthetic threat injection experiments. Furthermore, the framework formulates a composite priority ranking that integrates anomaly severity, cluster rarity, and SOC contextual weighting, yielding actionable alert rankings. An interactive, analyst-centric dashboard enables SOC teams to explore top alerts, clusters, associated MITRE techniques, priority rankings, and geolocation data, providing insights while preserving human oversight. Overall, the proposed system transforms complex alert streams into structured insights, enhancing SOC situational awareness, decision support, and operational efficiency.