Robustness Evaluation of Network Intrusion Detection Systems based on Sequential Machine Learning

The rise of sequential Machine Learning (ML) methods has paved the way for a new generation of Network Intrusion Detection Systems (NIDS) which base their classification on the temporal patterns exhibited by malicious traffic. Previous work presents successful algorithms in this field, but just a few attempts try to assess their robustness in real-world contexts. In this paper, we aim to fill this gap by presenting a novel evaluation methodology. In particular, we propose a new time-based adversarial attack in which we simulate a delay in the malicious communications that changes the arrangement of the samples in the test set. Moreover, we design an innovative evaluation technique simulating a worst-case training scenario in which the last portion of the training set does not include any malicious flow. Through them, we can evaluate how much sequential ML-based NIDS are sensible to modifications that an adaptive attacker might apply at temporal level, and we can verify their robustness to the unpredictable traffic produced by modern networks. Our experimental campaign validates our proposal against a recent NIDS trained on a public dataset for botnet detection. The results demonstrate its high resistance to temporal adversarial attacks, but also a drastic performance drop when even just 1% of benign flows are injected at the end of the training set. Our findings raise questions about the reliable deployment of sequential ML-NIDS in practice, and at the same time can guide researchers to develop more robust defensive tools in the future.