Network traffic classification for security analysis

We used unsupervised machine learning to identify anomalous patterns of network traffic that suggest intrusion. Such techniques allow one to classify network traffic into clusters that emerge from the training data and do not require that signatures already be known. Data is from the National Collegiate Cybersecurity Defense Competition (NCCDC). All but the TCP connections were filtered out, and the features extracted from the remaining data included characteristics of individual connections as well as patterns across time within a sliding window. The learning technique was k-means, with k = 5 giving the most natural and revealing partition of the data. The results bore out the following two hypotheses consistent with the literature: (1) most network traffic is normal, only a certain percentage being malicious; (2) the traffic from an attack is statistically different from normal traffic.