The Cross-evaluation of Machine Learning-based Network Intrusion Detection Systems

Network Intrusion Detection Systems are essential tools for protecting networks against attacks. Deep Learning approaches are increasingly employed in developing these systems due to their versatility and effectiveness. However, the common procedure for training and testing Deep Learning models typically leverages traffic data entirely collected from the operational network managed by a single organization, posing privacy and security concerns in sharing these data. As a result, the assessment of the performance of these models in real-world scenarios is significantly hindered. On the other hand, given the wide variety of existing attacks and the emergence of new attack types, it is crucial to evaluate the robustness of Intrusion Detection Systems when the network context varies. Indeed, it is highly desirable that the effectiveness of trained Deep Learning models is not severely impacted when ported into other networks.To this aim, in this work, we exploit various single-modal and multimodal Deep Learning approaches and leverage a cross-evaluation procedure to assess their capability to distinguish malicious from benign traffic in different network contexts. Furthermore, we investigate the impact of various informative fields extracted from traffic on the generalization capability of models. Our cross-evaluation leverages three recent public-available network attack datasets related to diverse scenarios. The results obtained suggest that the availability at training time of traffic generated by attacks conducted in the operational network is crucial for designing a robust Intrusion Detection System that keeps working with minimal Fl-score degradation, when the network context changes.