A Novel Versatile Framework for Enabling Early Detection of Evolving Network-based Cyberattacks

Abstract

Network-based cyber-attacks have been increasing in scale, frequency and sophistication, posing significant threats to nation states and organizations worldwide. Researchers have proposed various anomaly-based solutions to detect such attacks and address the shortcomings of traditional signature-based methods. However, these solutions either require complex preprocessing to extract network flow statistics or depend on hand-crafted features from domain expertise, thus adding computational overhead that limits the ability for early attack detection. To address these limitations, this thesis proposes a novel framework called FPAC (Flexible Parser Anonymizer Converter) which is designed to enable early detection of different types of attacks by processing only the first few packets of network flows. The study departs from established methods that rely on flow statistics and hand-crafted features by introducing innovative techniques for processing and learning from raw network traffic bytes. In the thesis, two attack detection scenarios i.e. Botnet and Low-rate Denial of Service (LDoS), and four different low overhead techniques i.e. Histogram of Oriented Gradients (HOG), entropy byte histogram, byte-based feature learning, and representation learning from bytes, were used to demonstrate the applicability of the FPAC framework for early attack detection. Experiments were performed to validate the FPAC approach using the CTU botnet and the UTSA 2021 LDoS datasets. For botnet attack detection, the byte-based feature learning techniques with Decision Trees (DT) and Extreme Gradient Boosting (XGB) performed optimally, achieving 99.9% accuracy with fast detection times ranging from 0.006 to 0.026 seconds. Image-based approaches using HOG and entropy byte histogram also achieved 99.4% and 100% accuracy, respectively, while incurring reduced overheads compared to related works. The 1D CNN model matched the best byte-based results with 99.9% accuracy, validating the role of deep learning within the FPAC framework. For LDoS attack detection, which is inherently more challenging due to its subtle nature, all four lightweight techniques employed in this thesis performed favourably compared to existing approaches. The byte-based method again delivered the best results, achieving 95.8% accuracy. Image-based techniques attained accuracies of 88.9% for HOG and 92.1% for entropy byte histogram with XGB, while the representation learning from bytes approach using 1D CNN achieved 95.6% accuracy. These results outperform computationally expensive methods reported in related works, showcasing that the FPAC framework achieves high detection performance with very low overheads while also generalizing effectively across different network attack types. Keywords: network-based attacks, early attack detection, machine learning, representation learning, botnet, LDoS, HOG, entropy byte histogram.