Here are a few use cases for this project:

1. Traffic Flow Analysis: The dataset can be used in machine learning models to analyze traffic flow in cities. It can identify the type of vehicles on the city roads at different times of the day, helping in planning and traffic management.

2. Vehicle Class Based Toll Collection: Toll booths can use this model to automatically classify and charge vehicles based on their type, enabling a more efficient and automated system.

3. Parking Management System: Parking lot owners can use this model to easily classify vehicles as they enter for better space management. Knowing the vehicle type can help assign it to the most suitable parking spot.

4. Traffic Rule Enforcement: The dataset can be used to create a computer vision model to automatically detect any traffic violations like wrong lane driving by different vehicle types, and notify law enforcement agencies.

5. Smart Ambulance Tracking: The system can help in identifying and tracking ambulances and other emergency vehicles, enabling traffic management systems to provide priority routing during emergencies.

Daily utilization metrics for data.lacity.org and geohub.lacity.org. Updated monthly

Network traffic datasets created by Single Flow Time Series Analysis

Datasets were created for the paper: Network Traffic Classification based on Single Flow Time Series Analysis -- Josef Koumar, Karel Hynek, Tomáš Čejka -- which was published at The 19th International Conference on Network and Service Management (CNSM) 2023. Please cite usage of our datasets as:

J. Koumar, K. Hynek and T. Čejka, "Network Traffic Classification Based on Single Flow Time Series Analysis," 2023 19th International Conference on Network and Service Management (CNSM), Niagara Falls, ON, Canada, 2023, pp. 1-7, doi: 10.23919/CNSM59352.2023.10327876.

This Zenodo repository contains 23 datasets created from 15 well-known published datasets which are cited in the table below. Each dataset contains 69 features created by Time Series Analysis of Single Flow Time Series. The detailed description of features from datasets is in the file: feature_description.pdf

In the following table is a description of each dataset file:

File name Detection problem Citation of original raw dataset

botnet_binary.csv Binary detection of botnet S. García et al. An Empirical Comparison of Botnet Detection Methods. Computers & Security, 45:100–123, 2014.

botnet_multiclass.csv Multi-class classification of botnet S. García et al. An Empirical Comparison of Botnet Detection Methods. Computers & Security, 45:100–123, 2014.

cryptomining_design.csv Binary detection of cryptomining; the design part Richard Plný et al. Datasets of Cryptomining Communication. Zenodo, October 2022

cryptomining_evaluation.csv Binary detection of cryptomining; the evaluation part Richard Plný et al. Datasets of Cryptomining Communication. Zenodo, October 2022

dns_malware.csv Binary detection of malware DNS Samaneh Mahdavifar et al. Classifying Malicious Domains using DNS Traffic Analysis. In DASC/PiCom/CBDCom/CyberSciTech 2021, pages 60–67. IEEE, 2021.

doh_cic.csv Binary detection of DoH

Mohammadreza MontazeriShatoori et al. Detection of doh tunnels using time-series classification of encrypted traffic. In DASC/PiCom/CBDCom/CyberSciTech 2020, pages 63–70. IEEE, 2020

doh_real_world.csv Binary detection of DoH Kamil Jeřábek et al. Collection of datasets with DNS over HTTPS traffic. Data in Brief, 42:108310, 2022

dos.csv Binary detection of DoS Nickolaos Koroniotis et al. Towards the development of realistic botnet dataset in the Internet of Things for network forensic analytics: Bot-IoT dataset. Future Gener. Comput. Syst., 100:779–796, 2019.

edge_iiot_binary.csv Binary detection of IoT malware Mohamed Amine Ferrag et al. Edge-iiotset: A new comprehensive realistic cyber security dataset of iot and iiot applications: Centralized and federated learning, 2022.

edge_iiot_multiclass.csv Multi-class classification of IoT malware Mohamed Amine Ferrag et al. Edge-iiotset: A new comprehensive realistic cyber security dataset of iot and iiot applications: Centralized and federated learning, 2022.

https_brute_force.csv Binary detection of HTTPS Brute Force Jan Luxemburk et al. HTTPS Brute-force dataset with extended network flows, November 2020

ids_cic_binary.csv Binary detection of intrusion in IDS Iman Sharafaldin et al. Toward generating a new intrusion detection dataset and intrusion traffic characterization. ICISSp, 1:108–116, 2018.

ids_cic_multiclass.csv Multi-class classification of intrusion in IDS Iman Sharafaldin et al. Toward generating a new intrusion detection dataset and intrusion traffic characterization. ICISSp, 1:108–116, 2018.

ids_unsw_nb_15_binary.csv Binary detection of intrusion in IDS Nour Moustafa and Jill Slay. Unsw-nb15: a comprehensive data set for network intrusion detection systems (unsw-nb15 network data set). In 2015 military communications and information systems conference (MilCIS), pages 1–6. IEEE, 2015.

ids_unsw_nb_15_multiclass.csv Multi-class classification of intrusion in IDS Nour Moustafa and Jill Slay. Unsw-nb15: a comprehensive data set for network intrusion detection systems (unsw-nb15 network data set). In 2015 military communications and information systems conference (MilCIS), pages 1–6. IEEE, 2015.

iot_23.csv Binary detection of IoT malware Sebastian Garcia et al. IoT-23: A labeled dataset with malicious and benign IoT network traffic, January 2020. More details here https://www.stratosphereips.org /datasets-iot23

ton_iot_binary.csv Binary detection of IoT malware Nour Moustafa. A new distributed architecture for evaluating ai-based security systems at the edge: Network ton iot datasets. Sustainable Cities and Society, 72:102994, 2021

ton_iot_multiclass.csv Multi-class classification of IoT malware Nour Moustafa. A new distributed architecture for evaluating ai-based security systems at the edge: Network ton iot datasets. Sustainable Cities and Society, 72:102994, 2021

tor_binary.csv Binary detection of TOR Arash Habibi Lashkari et al. Characterization of Tor Traffic using Time based Features. In ICISSP 2017, pages 253–262. SciTePress, 2017.

tor_multiclass.csv Multi-class classification of TOR Arash Habibi Lashkari et al. Characterization of Tor Traffic using Time based Features. In ICISSP 2017, pages 253–262. SciTePress, 2017.

vpn_iscx_binary.csv Binary detection of VPN Gerard Draper-Gil et al. Characterization of Encrypted and VPN Traffic Using Time-related. In ICISSP, pages 407–414, 2016.

vpn_iscx_multiclass.csv Multi-class classification of VPN Gerard Draper-Gil et al. Characterization of Encrypted and VPN Traffic Using Time-related. In ICISSP, pages 407–414, 2016.

vpn_vnat_binary.csv Binary detection of VPN Steven Jorgensen et al. Extensible Machine Learning for Encrypted Network Traffic Application Labeling via Uncertainty Quantification. CoRR, abs/2205.05628, 2022

vpn_vnat_multiclass.csv Multi-class classification of VPN Steven Jorgensen et al. Extensible Machine Learning for Encrypted Network Traffic Application Labeling via Uncertainty Quantification. CoRR, abs/2205.05628, 2022

Abstract: The task for this dataset is to forecast the spatio-temporal traffic volume based on the historical traffic volume and other features in neighboring locations.

Source: Liang Zhao, liang.zhao '@' emory.edu, Emory University.

Data Set Information: The task for this dataset is to forecast the spatio-temporal traffic volume based on the historical traffic volume and other features in neighboring locations. Specifically, the traffic volume is measured every 15 minutes at 36 sensor locations along two major highways in Northern Virginia/Washington D.C. capital region. The 47 features include: 1) the historical sequence of traffic volume sensed during the 10 most recent sample points (10 features), 2) week day (7 features), 3) hour of day (24 features), 4) road direction (4 features), 5) number of lanes (1 feature), and 6) name of the road (1 feature). The goal is to predict the traffic volume 15 minutes into the future for all sensor locations. With a given road network, we know the spatial connectivity between sensor locations. For the detailed data information, please refer to the file README.docx.

Attribute Information: The 47 features include: (1) the historical sequence of traffic volume sensed during the 10 most recent sample points (10 features), (2) week day (7 features), (3) hour of day (24 features), (4) road direction (4 features), (5) number of lanes (1 feature), and (6) name of the road (1 feature).

Relevant Papers: Liang Zhao, Olga Gkountouna, and Dieter Pfoser. 2019. Spatial Auto-regressive Dependency Interpretable Learning Based on Spatial Topological Constraints. ACM Trans. Spatial Algorithms Syst. 5, 3, Article 19 (August 2019), 28 pages. DOI:[Web Link]

Citation Request: To use these datasets, please cite the papers:

Liang Zhao, Olga Gkountouna, and Dieter Pfoser. 2019. Spatial Auto-regressive Dependency Interpretable Learning Based on Spatial Topological Constraints. ACM Trans. Spatial Algorithms Syst. 5, 3, Article 19 (August 2019), 28 pages. DOI:[Web Link]

Public (anonymized) road traffic prediction datasets from Huawei Munich Research Center.

Datasets from a variety of traffic sensors (i.e. induction loops) for traffic prediction. The data is useful for forecasting traffic patterns and adjusting stop-light control parameters, i.e. cycle length, offset and split times.

The dataset contains recorded data from 6 crosses in the urban area for the last 56 days, in the form of flow timeseries, depicted the number of vehicles passing every 5 minutes for a whole day (i.e. 12 readings/h, 288 readings/day, 16128 readings / 56 days).

DESCRIPTION OF THE RESEARCH AND DATA: This work presents the Madrid Traffic Dataset (MTD), a comprehensive resource for the analysis and modeling of traffic patterns in Madrid. The dataset integrates data from traffic sensors, weather observations, calendar information, road infrastructure, and geolocation data to support advanced studies of urban mobility and predictive modeling.

In addition to the core data sources, the dataset includes temporal sequences and a traffic adjacency matrix, enabling the application of time-series analysis and graph-based modeling approaches.

-COMPLETE DATASET: The complete version of the MTD includes data from 554 traffic sensors distributed across the Madrid region, covering a total of 30 months (from June 2022 to November 2024).

-SUBSET DATASET: A more compact version derived from the complete dataset, focused on a subset of 300 traffic sensors with 17 months of data (from June 2022 to October 2023). This subset is designed for researchers requiring a lighter dataset.

DATA ORGANIZATION: The dataset is organized in a main directory containing a subfolder identified by the configuration data hash. This subfolder includes all key components: datasets, temporal sequences, adjacency matrices, and configuration files. The structure ensures that all resources are clearly arranged to facilitate easy access and reproducibility for researchers.

For more details, see [Submitted to IEEE Internet of the Things Journal].

Update NotesMar 16 2024, remove spaces in the file and folder names.Mar 31 2024, delete the underscore in the city names with a space (such as San Francisco) in the '02_TransCAD_results' folder to ensure correct data loading by TransCAD (software version: 9.0).Aug 31 2024, add the 'cityname_link_LinkFlows.csv' file in the '02_TransCAD_results' folder to match the link from input data and the link from TransCAD results (LinkFlows) with the same Link_ID.IntroductionThis is a unified and validated traffic dataset for 20 US cities. There are 3 folders for each city.01 Input datathe initial network data obtained from OpenStreetMap (OSM)the visualization of the OSM dataprocessed node / link / od data02 TransCAD results (software version: 9.0)cityname.dbd : geographical network database of the city supported by TransCAD (version 9.0)cityname_link.shp / cityname_node.shp : network data supported by GIS software, which can be imported into TransCAD manually. Then the corresponding '.dbd' file can be generated for TransCAD with a version lower than 9.0od.mtx : OD matrix supported by TransCADLinkFlows.bin / LinkFlows.csv : traffic assignment results by TransCADcityname_link_LinkFlows.csv: the input link attributes with the traffic assignment results by TransCADShortestPath.mtx / ue_travel_time.csv : the traval time (min) between OD pairs by TransCAD03 AequilibraE results (software version: 0.9.3)cityname.shp : shapefile network data of the city support by QGIS or other GIS softwareod_demand.aem : OD matrix supported by AequilibraEnetwork.csv : the network file used for traffic assignment in AequilibraEassignment_result.csv : traffic assignment results by AequilibraEPublicationXu, X., Zheng, Z., Hu, Z. et al. (2024). A unified dataset for the city-scale traffic assignment model in 20 U.S. cities. Sci Data 11, 325. https://doi.org/10.1038/s41597-024-03149-8Usage NotesIf you use this dataset in your research or any other work, please cite both the dataset and paper above.A brief introduction about how to use this dataset can be found in GitHub. More detailed illustration for compiling the traffic dataset on AequilibraE can be referred to GitHub code or Colab code.ContactIf you have any inquiries, please contact Xiaotong Xu (email: kid-a.xu@connect.polyu.hk).

This traffic dataset contains a balance size of encrypted malicious and legitimate traffic for encrypted malicious traffic detection and analysis. The dataset is a secondary csv feature data that is composed of six public traffic datasets.

Our dataset is curated based on two criteria: The first criterion is to combine widely considered public datasets which contain enough encrypted malicious or encrypted legitimate traffic in existing works, such as Malware Capture Facility Project datasets. The second criterion is to ensure the final dataset balance of encrypted malicious and legitimate network traffic.

Based on the criteria, 6 public datasets are selected. After data pre-processing, details of each selected public dataset and the size of different encrypted traffic are shown in the “Dataset Statistic Analysis Document”. The document summarized the malicious and legitimate traffic size we selected from each selected public dataset, the traffic size of each malicious traffic type, and the total traffic size of the composed dataset. From the table, we are able to observe that encrypted malicious and legitimate traffic equally contributes to approximately 50% of the final composed dataset.

The datasets now made available were prepared to aim at encrypted malicious traffic detection. Since the dataset is used for machine learning or deep learning model training, a sample of train and test sets are also provided. The train and test datasets are separated based on 1:4. Such datasets can be used for machine learning or deep learning model training and testing based on selected features or after processing further data pre-processing.

SDCC Traffic Congestion Saturation Flow Data April to Sept 2024. Traffic volumes, traffic saturation, and congestion data for sites across South Dublin County. Used by traffic management to control stage timings on junctions.

The main aim of this dataset is to enable detection of traffic congestion from surveillance cameras using one-stage object detectors. The dataset contains congested and uncongested traffic scenes with their respective labels. This dataset is collected from different surveillance cameras video footage. To prepare the dataset frames are extracted from video sources and resized to a dimension of 500 x 500 with .jpg image format. To Annotate, the image LabelImg software has used. The format of the label is .txt with the same name as the image. The dataset is mainly prepared for YOLO Models but it can be converted to other models format.

Q-Traffic is a large-scale traffic prediction dataset, which consists of three sub-datasets: query sub-dataset, traffic speed sub-dataset and road network sub-dataset.

Dataset

This dataset was created by YUCHU Shen

Released under Apache 2.0

Contents

Network Address Translation (NAT)

Feature layer containing authoritative traffic count points for Sioux Falls, South Dakota.The traffic counts listed are 24-hour, weekday, two-directional counts. Traffic counts are normally collected during the summer months, but may be taken any season, as weather permits. The traffic counts are factored by the day of the week as well as by the month of the year to become an Average Annual Daily Total (AADT). Traffic volumes (i.e. count data) can fluctuate depending on the month, week, day of collection; the weather, type of road surface, nearby construction, etc. All of the historical data should be averaged to reflect the "normal" traffic count. More specific count data (time, date, hourly volume) can be obtained from the Sioux Falls Engineering Division at 367-8601.

This dataset was created by a LoRaWAN sniffer and contains packets, which are thoroughly analyzed in the paper Exploring LoRaWAN Traffic: In-Depth Analysis of IoT Network Communications (not yet published). Data from the LoRaWAN sniffer was collected in four cities: Liege (Belgium), Graz (Austria), Vienna (Austria), and Brno (Czechia).

Gateway ID: b827ebafac000001

• Uplink reception (end-device => gateway)
• Only packets containing CRC, inverted IQ
• RX0: 867.1 MHz, 867.3 MHz, 867.5 MHz, 867.7 MHz, 867.9 MHz - BW 125 kHz and all SF
• RX1: 868.1 MHz, 868.3 MHz, 868.5 MHz - BW 125 kHz and all SF

Gateway ID: b827ebafac000002

• Downlink reception (gateway => end-device)
• Includes packets without CRC, non-inverted IQ
• RX0: 867.1 MHz, 867.3 MHz, 867.5 MHz, 867.7 MHz, 867.9 MHz - BW 125 kHz and all SF
• RX1: 868.1 MHz, 868.3 MHz, 868.5 MHz - BW 125 kHz and all SF

Gateway ID: b827ebafac000003

• Downlink reception (gateway => end-device) and Class-B beacon on 869.525 MHz
• Includes packets without CRC, non-inverted IQ
• RX0: 869.525 MHz - BW 125 kHz and all SF, BW 125 kHz and SF9 with implicit header, CR 4/5 and length 17 B

To open the pcap files, you need Wireshark with current support for LoRaTap and LoRaWAN protocols. This support will be available in the official 4.1.0 release. A working version for Windows is accessible in the automated build system.

The source data is available in the log.zip file, which contains the complete dataset obtained by the sniffer. A set of conversion tools for log processing is available on Github. The converted logs, available in Wireshark format, are stored in pcap.zip. For the LoRaWAN decoder, you can use the attached root and session keys. The processed outputs are stored in csv.zip, and graphical statistics are available in png.zip.

This data represents a unique, geographically identifiable selection from the full log, cleaned of any errors. The records from Brno include communication between the gateway and a node with known keys.

Test file :: 00_Test

• short test file for parser verification
• comparison of LoRaTap version 0 and version 1 formats

Brno, Czech Republic :: 01_Brno

• 49.22685N, 16.57536E, ASL 306m
• lines 150873 to 529796
• time 1.8.2022 15:04:28 to 17.8.2022 13:05:32
• preliminary experiment
• experimental device
  • Device EUI: 70b3d5cee0000042
  • Application key: d494d49a7b4053302bdcf96f1defa65a
  • Device address: 00d85395
  • Network session key: c417540b8b2afad8930c82fcf7ea54bb
  • Application session key: 421fea9bedd2cc497f63303edf5adf8e

Liege, Belgium :: 02_Liege :: evaluated in the paper

• 50.66445N, 5.59276E, ASL 151m
• lines 636205 to 886868
• time 25.8.2022 10:12:24 to 12.9.2022 06:20:48

Brno, Czech Republic :: 03_Brno_join

• 49.22685N, 16.57536E, ASL 306m
• lines 947787 to 979382
• time 30.9.2022 15:21:27 to 4.10.2022 10:46:31
• record contains OTAA activation (Join Request / Join Accept)
• experimental device:
  • Device EUI: 70b3d5cee0000042
  • Application key: d494d49a7b4053302bdcf96f1defa65a
  • Device address: 01e65ddc
  • Network session key: e2898779a03de59e2317b149abf00238
  • Application session key: 59ca1ac91922887093bc7b236bd1b07f

Graz, Austria :: 04_Graz :: evaluated in the paper

• 47.07049N, 15.44506E, ASL 364m
• lines 1015139 to 1178855
• time 26.10.2022 06:21:07 to 29.11.2022 10:03:00

Vienna, Austria :: 05_Wien :: evaluated in the paper

• 48.19666N, 16.37101E, ASL 204m
• lines 1179308 to 3657105
• time 1.12.2022 10:42:19 to 4.1.2023 14:00:05
• contains a total of 14 short restarts (under 90 seconds)

Brno, Czech Republic :: 07_Brno :: evaluated in the paper

• 49.22685N, 16.57536E, ASL 306m
• lines 4969648 to 6919392
• time 16.2.2023 8:53:43 to 30.3.2023 9:00:11

To provide a shared basis for comparing traffic light recognition (TLR) systems, the authors publish an extensive public LISA Traffic Light Dataset based on footage from US roads. The dataset contains annotated video sequences, captured under varying light and weather conditions using a stereo camera. The database consists of continuous test and training video sequences, totaling 43,007 frames and 113,888 annotated traffic lights. The sequences are captured by a stereo camera mounted on the roof of a vehicle driving under both night- and daytime with varying light and weather conditions.

Feature layer containing authoritative traffic count points for the traffic model for Sioux Falls, South Dakota. The data in the traffic counts model feature layer is collected for traffic count modeling and transportation planning. This data is collected on a five-to-seven-year basis, with data from 2001, 2008, 2013, 2018, and 2023. The traffic counts are 24-hour, weekday, two-directional counts. Traffic counts are normally collected during the summer months, but may be taken any season, as weather permits. The traffic counts are factored by the day of the week as well as by the month of the year to become an Average Annual Daily Total (AADT). Traffic volumes (i.e., count data) can fluctuate depending on the month, week, day of collection, the weather, type of road surface, nearby construction, etc. All the historical data should be averaged to reflect the "normal" traffic count. More specific count data (time, date, hourly volume) can be obtained from the Sioux Falls Engineering Division at 605-367-8601.

PlayStation 4.

This dataset contains 145063 time series representing the number of hits or web traffic for a set of Wikipedia pages from 2015-07-01 to 2022-06-30. This is an extended version of the dataset that was used in the Kaggle Wikipedia Web Traffic forecasting competition. For consistency, the same Wikipedia pages that were used in the competition have been used in this dataset as well. The colons (:) in article names have been replaced by dashes (-) to make the .tsf file readable using our data loaders.

The data were downloaded from the Wikimedia REST API. According to the conditions of the API, this dataset is licensed under CC-BY-SA 3.0 and GFDL licenses.

Here are a few use cases for this project:

1. Autonomous Vehicles Navigation: The "Carla traffic dataset" can be used to develop and improve algorithms for autonomous vehicles, enabling them to effectively identify other road users, traffic lights, and various traffic signs, improving the cars’ ability to navigate safely in different weather conditions including fog.

2. Traffic Management Systems: The dataset could be leveraged to create advanced traffic management systems, identifying car, bike, or pedestrian movement, detecting traffic light states, and understanding if road users respect speed limits (30, 60, 90 km/h signs). This could improve urban traffic flow and increase overall road safety.

3. Driver Assistance Systems: The dataset could be used to develop advanced driver assistance systems (ADAS) that could alert drivers of pedestrians, other vehicles, traffic signs, and the status of traffic lights, particularly in foggy or difficult conditions.

4. Safety Testing for Vehicle Manufacturers: Companies manufacturing cars, bikes, or motorbikes could use the data to carry out safety testing under different situations, including different weather conditions and traffic light changes.

5. Virtual Driving Simulation: Game developers or driving schools could use this model to develop realistic driving simulations. The players or trainee drivers would need to respond correctly and promptly to real-world traffic situations like recognizing speed signs, traffic lights, and other road users.