Interactive Machine Learning in Cybersecurity: Using Human Expertise More Effectively

Cybersecurity is increasingly important in a world where malicious actors seek to profit from various forms of data exfiltration. In this dissertation I examine the potential for improving the detection of anomalies related to potential data exfiltration in a large financial service company through better use of human expertise. The overall approach that guided the research reported below is interactive Machine Learning (iML) where humans work together with Machine Learning to solve prediction and classification problems. After reviewing work on data exfiltration and ML, I carried out a series of three case studies with the overall goal of using iML to improve defence against data exfiltration. I first demonstrated that the best known synthetic data set did not provide credible results that would generalize to real world settings. I then did two further case studies on real world data (from a financial services company). In the first of those further studies I looked at how the organization of human labelers affects outcomes with active learning (AL), comparing model training performance between individuals, groups, and situations where pairs of labelers exchanged places halfway through the training process so that different people trained the model on the second half of the training process. I found that a single individual performed best and the role of expertise was further reinforced in my final case study where a group of skilled (vs. unskilled) labelers was shown to produce higher model performance, as well as producing confidence ratings in their labels that were better calibrated with resulting model accuracy. I also showed that an information gain maximizing approach was a viable method of AL in this application area, with better F1 scores and labeler confidence than a more traditional model uncertainty approach. One limitation of this research was that due to privacy concerns I was using redacted email data that did not contain the email body.

Récupéré en direct depuis OpenAlex et désinversé. Les résumés ne sont pas conservés dans cette base de données : les index inversés représentent 8,6 Go des 9,3 Go de texte de la base, et le serveur dispose de 13 Go libres.