Netflow traffic generated using DOROTHEA (DOcker-based fRamework fOr gaTHering nEtflow trAffic) NetFlow is a network protocol developed by Cisco for the collection and monitoring of network traffic flow data generated. A flow is defined as a unidirectional sequence of packets with some common properties that pass through a network device. Netflow flows have been captured by sampling at the packet level. A sampling means that 1 out of every X packets is selected to be flow while the rest of the packets are not valued. In the construction of the datasets, different percentages of flows considered attacks and flows considered normal traffic have been used. These datasets have been used to test previously trained models.

NetFlow traffic generated using DOROTHEA (DOcker-based fRamework fOr gaTHering nEtflow trAffic) NetFlow is a network protocol developed by Cisco for the collection and monitoring of network traffic flow data generated. A flow is defined as a unidirectional sequence of packets with some common properties that pass through a network device.

NetFlow flows have been captured with sampling 1000 at the packet level. A sampling means that 1 out of every X packets is selected to be flow while the rest of the packets are not valued.

The version of NetFlow used to build the datasets is 5.

NetFlow traffic generated using DOROTHEA (DOcker-based fRamework fOr gaTHering nEtflow trAffic) NetFlow is a network protocol developed by Cisco for the collection and monitoring of network traffic flow data generated. A flow is defined as a unidirectional sequence of packets with some common properties that pass through a network device.

NetFlow flows have been captured with different sampling at the packet level. A sampling means that 1 out of every X packets is selected to be flow while the rest of the packets are not valued.

The version of NetFlow used to build the datasets is 5.

AIT Netflow Data Sets

This repository contains labeled synthetic netflows suitable for evaluation of intrusion detection systems, federated learning, and alert aggregation. The netflows are generated from the packet captures contained in the AIT-LDS-v2.0. A detailed description of that dataset is available in [1]. The packet captures were collected from eight testbeds that were built at the Austrian Institute of Technology (AIT) following the approach by [2]. Please cite these papers if the data is used for academic publications.

In brief, each of the datasets corresponds to a testbed representing a small enterprise network including mail server, file share, WordPress server, VPN, firewall, etc. Normal user behavior is simulated to generate background noise over a time span of 4-6 days. At some point, a sequence of attack steps is launched against the network. The following attacks are launched in the network:

• Scans (nmap, WPScan, dirb)
• Webshell upload (CVE-2020-24186)
• Password cracking (John the Ripper)
• Privilege escalation
• Remote command execution
• Data exfiltration (DNSteal)

This repository contains the following files:

• : CSV files of labeled TCP and UDP netflows for each testbed.
• README.md: Instructions on how to reproduce the generation and labeling of the netflows from the AIT-LDS-v2.0. Note that it is only necessary to run the python scripts if you want to extend or change the labeling procedure.
• 1_format_dataset_info.ipynb: Generates the tables necessary for labeling (see README.md).
• 2_label_logs.ipynb: Labels the netflows (see README.md).

Acknowledgements: Partially funded by the FFG projects INDICAETING (868306) and DECEPT (873980), and the EU projects GUARD (833456) and PANDORA (SI2.835928).

If you use the dataset, please cite the following publications:

[1] M. Landauer, F. Skopik, M. Frank, W. Hotwagner, M. Wurzenberger, and A. Rauber. "Maintainable Log Datasets for Evaluation of Intrusion Detection Systems". IEEE Transactions on Dependable and Secure Computing, vol. 20, no. 4, pp. 3466-3482. [PDF]

[2] M. Landauer, F. Skopik, M. Wurzenberger, W. Hotwagner and A. Rauber, "Have it Your Way: Generating Customized Log Datasets With a Model-Driven Simulation Testbed," in IEEE Transactions on Reliability, vol. 70, no. 1, pp. 402-415, March 2021, doi: 10.1109/TR.2020.3031317. [PDF]

Introduction This datasets have SQL injection attacks (SLQIA) as malicious Netflow data. The attacks carried out are SQL injection for Union Query and Blind SQL injection. To perform the attacks, the SQLMAP tool has been used. NetFlow traffic has generated using DOROTHEA (DOcker-based fRamework fOr gaTHering nEtflow trAffic). NetFlow is a network protocol developed by Cisco for the collection and monitoring of network traffic flow data generated. A flow is defined as a unidirectional sequence of packets with some common properties that pass through a network device. Datasets The firts dataset was colleted to train the detection models (D1) and other collected using different attacks than those used in training to test the models and ensure their generalization (D2). The datasets contain both benign and malicious traffic. All collected datasets are balanced. The version of NetFlow used to build the datasets is 5. Dataset Aim Samples Benign-malicious
traffic ratio D1 Training 400,003 50% D2 Test 57,239 50% Infrastructure and implementation Two sets of flow data were collected with DOROTHEA. DOROTHEA is a Docker-based framework for NetFlow data collection. It allows you to build interconnected virtual networks to generate and collect flow data using the NetFlow protocol. In DOROTHEA, network traffic packets are sent to a NetFlow generator that has a sensor ipt_netflow installed. The sensor consists of a module for the Linux kernel using Iptables, which processes the packets and converts them to NetFlow flows. DOROTHEA is configured to use Netflow V5 and export the flow after it is inactive for 15 seconds or after the flow is active for 1800 seconds (30 minutes) Benign traffic generation nodes simulate network traffic generated by real users, performing tasks such as searching in web browsers, sending emails, or establishing Secure Shell (SSH) connections. Such tasks run as Python scripts. Users may customize them or even incorporate their own. The network traffic is managed by a gateway that performs two main tasks. On the one hand, it routes packets to the Internet. On the other hand, it sends it to a NetFlow data generation node (this process is carried out similarly to packets received from the Internet). The malicious traffic collected (SQLI attacks) was performed using SQLMAP. SQLMAP is a penetration tool used to automate the process of detecting and exploiting SQL injection vulnerabilities. The attacks were executed on 16 nodes and launch SQLMAP with the parameters of the following table. Parameters Description '--banner','--current-user','--current-db','--hostname','--is-dba','--users','--passwords','--privileges','--roles','--dbs','--tables','--columns','--schema','--count','--dump','--comments', --schema' Enumerate users, password hashes, privileges, roles, databases, tables and columns --level=5 Increase the probability of a false positive identification --risk=3 Increase the probability of extracting data --random-agent Select the User-Agent randomly --batch Never ask for user input, use the default behavior --answers="follow=Y" Predefined answers to yes Every node executed SQLIA on 200 victim nodes. The victim nodes had deployed a web form vulnerable to Union-type injection attacks, which was connected to the MYSQL or SQLServer database engines (50% of the victim nodes deployed MySQL and the other 50% deployed SQLServer). The web service was accessible from ports 443 and 80, which are the ports typically used to deploy web services. The IP address space was 182.168.1.1/24 for the benign and malicious traffic-generating nodes. For victim nodes, the address space was 126.52.30.0/24.
The malicious traffic in the test sets was collected under different conditions. For D1, SQLIA was performed using Union attacks on the MySQL and SQLServer databases. However, for D2, BlindSQL SQLIAs were performed against the web form connected to a PostgreSQL database. The IP address spaces of the networks were also different from those of D1. In D2, the IP address space was 152.148.48.1/24 for benign and malicious traffic generating nodes and 140.30.20.1/24 for victim nodes. To run the MySQL server we ran MariaDB version 10.4.12.
Microsoft SQL Server 2017 Express and PostgreSQL version 13 were used.

NetFlow traffic generated using DOROTHEA (DOcker-based fRamework fOr gaTHering nEtflow trAffic) NetFlow is a network protocol developed by Cisco for the collection and monitoring of network traffic flow data generated. A flow is defined as a unidirectional sequence of packets with some common properties that pass through a network device. NetFlow flows have been captured with sampling 500 at the packet level. A sampling means that 1 out of every X packets is selected to be flow while the rest of the packets are not valued. The version of NetFlow used to build the datasets is 5.

NF-BoT-IoT-V2 is the extended NetFlow version of NF-BoT-IoT. Compared to the original NF-NIDS datasets, the feature set of NetFlow features has expanded from 8 to 43.

This is one dataset in the NFV2-collection by the university of Queensland aimed at standardizing network-security datasets to achieve interoperability and larger analyses.

All credit goes to the original authors: Dr. Mohanad Sarhan, Dr. Siamak Layeghy and Dr. Marius Portmann. Please cite their original journal article when using this dataset.

V1: Base dataset in CSV format as downloaded from here V2: Cleaning -> parquet files

In the parquet files all data types are already set correctly, there are 0 records with missing information and 0 duplicate records.

This dataset is comprised of NetFlow records, which capture the outbound network traffic of 8 commercial IoT devices and 5 non-IoT devices, collected during a period of 37 days in a lab at Ben-Gurion University of The Negev. The dataset was collected in order to develop a method for telecommunication providers to detect vulnerable IoT models behind home NATs. Each NetFlow record is labeled with the device model which produced it; for research reproducibilty, each NetFlow is also allocated to either the "training" or "test" set, in accordance with the partitioning described in:

Y. Meidan, V. Sachidananda, H. Peng, R. Sagron, Y. Elovici, and A. Shabtai, A novel approach for detecting vulnerable IoT devices connected behind a home NAT, Computers & Security, Volume 97, 2020, 101968, ISSN 0167-4048, https://doi.org/10.1016/j.cose.2020.101968. (http://www.sciencedirect.com/science/article/pii/S0167404820302418)

Please note:

• The dataset itself is free to use, however users are requested to cite the above-mentioned paper, which describes in detail the research objectives as well as the data collection, preparation and analysis.
• Following is a brief description of the features used in this dataset.

# NetFlow features, used in the related paper for analysis

'FIRST_SWITCHED': System uptime at which the first packet of this flow was switched
'IN_BYTES': Incoming counter for the number of bytes associated with an IP Flow
'IN_PKTS': Incoming counter for the number of packets associated with an IP Flow
'IPV4_DST_ADDR': IPv4 destination address
'L4_DST_PORT': TCP/UDP destination port number
'L4_SRC_PORT': TCP/UDP source port number
'LAST_SWITCHED': System uptime at which the last packet of this flow was switched
'PROTOCOL': IP protocol byte (6: TCP, 17: UDP)
'SRC_TOS': Type of Service byte setting when there is an incoming interface
'TCP_FLAGS': Cumulative of all the TCP flags seen for this flow

# Features added by the authors

'IP': Prefix of the destination IP address, representing the network (without the host)
'DURATION': Time (seconds) between first/last packet switching

# Label
'device_model':

# Partition
'partition': Training or test

# Additional NetFlow features (mostly zero-variance)
'SRC_AS': Source BGP autonomous system number
'DST_AS': Destination BGP autonomous system number
'INPUT_SNMP': Input interface index
'OUTPUT_SNMP': Output interface index
'IPV4_SRC_ADDR': IPv4 source address
'MAC': MAC address of the source

# Additional data
'category': IoT or non-IoT
'type': IoT, access_point, smartphone, laptop
'date': Datepart of FIRST_SWITCHED
'inter_arrival_time': Time (seconds) between successive flows of the same device (identified by its MAC address)

One day of Netflow version 5 collected in flow tools format at an academic department. Collection includes traffic between all switches within the department and the egress switch to the college, university, and Internet. Departmental IP addresses in the flows are anaonymized via constant substitution.

NF-UQ-NIDS-V2 is the extended NetFlow version of NF-UQ-NIDS. Compared to the original NF-NIDS datasets, the feature set of NetFlow features has expanded from 8 to 43.

This is one dataset in the NFV2-collection by the university of Queensland aimed at standardizing network-security datasets to achieve interoperability and larger analyses.

All credit goes to the original authors: Dr. Mohanad Sarhan, Dr. Siamak Layeghy and Dr. Marius Portmann. Please cite their original journal article when using this dataset.

V1: Base dataset in CSV format as downloaded from here V2: Cleaning -> parquet files

In the parquet files all data types are already set correctly, there are 0 records with missing information and 0 duplicate records.

AssureMOSS Kubernetes Run-time Monitoring Dataset This dataset contains NetFlow data that is collected from a Kubernetes cluster. The cluster is used to monitor the microservice applications that are running on the cluster. The goal is to use the (NetFlow) logs to learn a state machine model that models the normal network behaviour within the cluster. The state machine model is then used to monitor and detect potential anomalies that might occur during the runtime of the cluster. This dataset contains both benign data (produced by real-life users) and malicious data (produced by launching several attacks against the clusters). The label of each flow is included in the dataset.

Traffic collected from the Merit border router at Chicago. This real-world traffic is currently collected without packet sampling. Combined with the Netflow-1 and Netflow-3 datasets, it describes the majority of ingress and egress traffic of the Merit Network.

India Mutual Funds Net Flow: SEBI: Private Sector data was reported at 193,265.022 INR mn in Oct 2018. This records an increase from the previous number of -1,983,968.988 INR mn for Sep 2018. India Mutual Funds Net Flow: SEBI: Private Sector data is updated monthly, averaging 21,615.000 INR mn from Jan 2000 (Median) to Oct 2018, with 226 observations. The data reached an all-time high of 1,550,710.000 INR mn in Apr 2010 and a record low of -1,983,968.988 INR mn in Sep 2018. India Mutual Funds Net Flow: SEBI: Private Sector data remains active status in CEIC and is reported by Securities and Exchange Board of India. The data is categorized under Global Database’s India – Table IN.ZC002: Mutual Funds Statistics: Securities and Exchange Board of India: Net Flow.

This dataset has SQL injection attacks as malicious Netflow data. The attacks carried out are SQL injection for Union Query and Blind SQL injection. To perform the attacks, the SQLmap tool has been used.

NetFlow traffic has generated using DOROTHEA (DOcker-based fRamework fOr gaTHering nEtflow trAffic). NetFlow is a network protocol developed by Cisco for the collection and monitoring of network traffic flow data generated. A flow is defined as a unidirectional sequence of packets with some common properties that pass through a network device.

The version of NetFlow used to build the datasets is 5.

Train dataset only contains benign traffic, this dataset has been collected using Netflow and implementing a sampling rate of 1 packet out of each 1000 to generate the flows, simulating the conditions of the RedCayle’s routers.

The traffic has been generated using three Python scripts. The first one simulates email sending using SMTP protocol. The second script simulates SSH connections. The third script simulates a user browsing the internet using different search engines and different protocols like HTTP and HTTPS.

France Medium and Long Term Debt: Net Flow data was reported at 72,234.000 EUR mn in 2017. This records an increase from the previous number of 62,535.000 EUR mn for 2016. France Medium and Long Term Debt: Net Flow data is updated yearly, averaging 57,321.000 EUR mn from Dec 2001 (Median) to 2017, with 17 observations. The data reached an all-time high of 105,830.000 EUR mn in 2010 and a record low of 27,411.000 EUR mn in 2001. France Medium and Long Term Debt: Net Flow data remains active status in CEIC and is reported by Agence France Tresor. The data is categorized under Global Database’s France – Table FR.F031: Medium and Long Term Debt: Agence France Tresor.

Hornet 7 is a dataset of seven days of network traffic attacks captured in cloud servers used as honeypots to help understand how geography may impact the inflow of network attacks. The honeypots were placed in eight different geographical locations: Amsterdam, London, Frankfurt, San Francisco, New York, Singapore, Toronto, Bangalore. The data was captured in April 2021.

The eight cloud servers were created and configured simultaneously following identical instructions. The network capture was performed using the Argus network monitoring tool in each cloud server. The cloud servers had only one service running (SSH on a non-standard port) and was fully dedicated to be used as a honeypot. No honeypot software was used in this dataset.

The dataset consists of eight scenarios, one for each geographically located cloud server. Each scenario contains bidirectional NetFlow files in the following format: - hornet7-biargus.tar.gz: all scenarios with bidirectional NetFlow files in Argus binary format; - hornet7-netflow-v5.tar.gz: all scenarios with bidirectional NetFlow v5 files in CSV format; - hornet7-netflow-extended.tar.gz: all scenarios with bidirectional NetFlows files in CSV format containing all features provided by Argus. - hornet7-full.tar.gz: download all the data (biargus, netflow v5 and extended netflows)

Hornet 15 is a dataset of fifteen days of network traffic attacks captured in cloud servers used as honeypots to help understand how geography may impact the inflow of network attacks. The honeypots are located in eight different cities: Amsterdam, London, Frankfurt, San Francisco, New York, Singapore, Toronto, Bangalore. The data was captured in April and May 2021.

The eight cloud servers were created and configured simultaneously, following identical instructions. The network capture was performed using the Argus network monitoring tool in each cloud server. The cloud servers had only one service running (SSH on a non-standard port) and were fully dedicated as a honeypot. No honeypot software was used in this dataset.

The dataset consists of eight scenarios, one for each geographically located cloud server. Each scenario contains bidirectional NetFlow files in the following format: - hornet15-biargus.tar.gz: all scenarios with bidirectional NetFlow files in Argus binary format; - hornet15-netflow-v5.tar.gz: all scenarios with bidirectional NetFlow v5 files in CSV format; - hornet15-netflow-extended.tar.gz: all scenarios with bidirectional NetFlows files in CSV format containing all features provided by Argus. - hornet15-full.tar.gz: download all the data (biargus, NetFlow v5, and extended NetFlows)

Hong Kong GNI: SNA08: EPIF: Net Flow: USA data was reported at -3,239.000 HKD mn in Mar 2018. This records a decrease from the previous number of -2,352.000 HKD mn for Dec 2017. Hong Kong GNI: SNA08: EPIF: Net Flow: USA data is updated quarterly, averaging -2,636.000 HKD mn from Mar 1999 (Median) to Mar 2018, with 77 observations. The data reached an all-time high of 7,203.000 HKD mn in Mar 2006 and a record low of -21,530.000 HKD mn in Jun 2013. Hong Kong GNI: SNA08: EPIF: Net Flow: USA data remains active status in CEIC and is reported by Census and Statistics Department. The data is categorized under Global Database’s Hong Kong – Table HK.A101: GNI: External Primary Income Flows.

This dataset has SQL injection attacks as malicious Netflow data. The attacks carried out are SQL injection for Union Query and Blind SQL injection. To perform the attacks, the SQLmap tool has been used.

NetFlow traffic has generated using DOROTHEA (DOcker-based fRamework fOr gaTHering nEtflow trAffic). NetFlow is a network protocol developed by Cisco for the collection and monitoring of network traffic flow data generated. A flow is defined as a unidirectional sequence of packets with some common properties that pass through a network device.

The version of NetFlow used to build the datasets is 5.