Apprentissage automatique non supervisé pour la détection de trafics illégitimes

Increasingly sophisticated, premeditated and targeted cyber-attacks such as Advanced Persistent Threats (APTs) can be perpetrated over long periods of time before being disclosed or discovered. To do this, attackers implement strategies to camouflage their malicious activities as long as possible, such as the implementation of communication channels between infected machines and a command and control (C&C) server in order to exfiltrate sensitive data or remotely control zombie machines. One of the techniques used is to encapsulate the C&C traffic in authorized network protocols (such as the Domain Name System (DNS) protocol, the Secure Hypertext Transfer Protocol (HTTPS), etc.) to bypass security control mechanisms. The detection of these malicious flows by traditional detection methods, such as Security Information and Event Management (SIEM) systems, is limited. The main obstacle is the large number of parameters to consider in manually defining reliable indicators. To address this challenge, we propose in this thesis an approach based on unsupervised machine learning and more specifically anomaly detection algorithms that we apply to the detection of DNS tunnels. The choice of an unsupervised learning algorithm is guided by the excessively high cost of obtaining a comprehensive learning dataset that would be labeled by security experts and which is essential for supervised learning algorithms. Then, the attacks we target aim to stay below detection thresholds. Therefore, malicious events or network flows will be rare. An initial study we conducted allowed us to highlight the DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm. However, DBSCAN requires experimentally finding the values of two hyperparameters. To automate the detection of DNS tunnels, we propose an improved algorithm called AutoRoC-DBSCAN which can automatically determine the values of these hyperparameters. We compared its performance with 5 other unsupervised learning algorithms (K-means, GMM, Isolation Forest, One-class SVM and LOF) on two different datasets. We created the first dataset that allows for the verification of DNS tunnel detection. The second dataset is CIRA-CIC-DoHBrw-2020 which is provided by the Canadian Institute for Cybersecurity project. The experiments validate the detection of DNS over HTTPS tunnels where malicious flows are doubly encapsulated by DNS then by HTTPS. The results obtained in our tests reinforce the interest in our approach.

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.