A collection of historic traffic count data and guidelines for how to collect new data for Massachusetts Department of Transportation (MassDOT) projects.

Code:

Packet_Features_Generator.py & Features.py

To run this code:

pkt_features.py [-h] -i TXTFILE [-x X] [-y Y] [-z Z] [-ml] [-s S] -j

-h, --help show this help message and exit -i TXTFILE input text file -x X Add first X number of total packets as features. -y Y Add first Y number of negative packets as features. -z Z Add first Z number of positive packets as features. -ml Output to text file all websites in the format of websiteNumber1,feature1,feature2,... -s S Generate samples using size s. -j

Purpose:

Turns a text file containing lists of incomeing and outgoing network packet sizes into separate website objects with associative features.

Uses Features.py to calcualte the features.

startMachineLearning.sh & machineLearning.py

To run this code:

bash startMachineLearning.sh

This code then runs machineLearning.py in a tmux session with the nessisary file paths and flags

Options (to be edited within this file):

--evaluate-only to test 5 fold cross validation accuracy

--test-scaling-normalization to test 6 different combinations of scalers and normalizers

Note: once the best combination is determined, it should be added to the data_preprocessing function in machineLearning.py for future use

--grid-search to test the best grid search hyperparameters - note: the possible hyperparameters must be added to train_model under 'if not evaluateOnly:' - once best hyperparameters are determined, add them to train_model under 'if evaluateOnly:'

Purpose:

Using the .ml file generated by Packet_Features_Generator.py & Features.py, this program trains a RandomForest Classifier on the provided data and provides results using cross validation. These results include the best scaling and normailzation options for each data set as well as the best grid search hyperparameters based on the provided ranges.

Data

Encrypted network traffic was collected on an isolated computer visiting different Wikipedia and New York Times articles, different Google search queres (collected in the form of their autocomplete results and their results page), and different actions taken on a Virtual Reality head set.

Data for this experiment was stored and analyzed in the form of a txt file for each experiment which contains:

First number is a classification number to denote what website, query, or vr action is taking place.

The remaining numbers in each line denote:

The size of a packet,

and the direction it is traveling.

negative numbers denote incoming packets

positive numbers denote outgoing packets

Figure 4 Data

This data uses specific lines from the Virtual Reality.txt file.

The action 'LongText Search' refers to a user searching for "Saint Basils Cathedral" with text in the Wander app.

The action 'ShortText Search' refers to a user searching for "Mexico" with text in the Wander app.

The .xlsx and .csv file are identical

Each file includes (from right to left):

The origional packet data,

each line of data organized from smallest to largest packet size in order to calculate the mean and standard deviation of each packet capture,

and the final Cumulative Distrubution Function (CDF) caluclation that generated the Figure 4 Graph.

From October 2024 to February 2025, ChatGPT outperformed competing AI-powered search engines in traffic referral, achieving a total growth of 155.52 percent. Perplexity placed second, despite experiencing more significant fluctuations, with a total growth of 54.78 percent by the conclusion of the analyzed period. With a 43.64 percent overall growth, Google's Gemini ranked third among other engines and maintained the most consistent traffic referral rate. Artificial intelligence-driven trends, notably AI-powered search, are changing online traffic patterns. This suggests a more significant change in the way users find information online and is expected to have a knock-on effect on the digital advertising sector.

In the second quarter of 2025, mobile devices (excluding tablets) accounted for 62.54 percent of global website traffic. Since consistently maintaining a share of around 50 percent beginning in 2017, mobile usage surpassed this threshold in 2020 and has demonstrated steady growth in its dominance of global web access. Mobile traffic Due to low infrastructure and financial restraints, many emerging digital markets skipped the desktop internet phase entirely and moved straight onto mobile internet via smartphone and tablet devices. India is a prime example of a market with a significant mobile-first online population. Other countries with a significant share of mobile internet traffic include Nigeria, Ghana and Kenya. In most African markets, mobile accounts for more than half of the web traffic. By contrast, mobile only makes up around 45.49 percent of online traffic in the United States. Mobile usage The most popular mobile internet activities worldwide include watching movies or videos online, e-mail usage and accessing social media. Apps are a very popular way to watch video on the go and the most-downloaded entertainment apps in the Apple App Store are Netflix, Tencent Video and Amazon Prime Video.

This dataset originates from DataCamp. Many users have reposted copies of the CSV on Kaggle, but most of those uploads omit the original instructions, business context, and problem framing. In this upload, I’ve included that missing context in the About Dataset so the reader of my notebook or any other notebook can fully understand how the data was intended to be used and the intended problem framing.

Note: I have also uploaded a visualization of the workflow I personally took to tackle this problem, but it is not part of the dataset itself. Additionally, I created a PowerPoint presentation based on my work in the notebook, which you can download from here:
PPTX Presentation

Recipe Site Traffic

From: Head of Data Science
Received: Today
Subject: New project from the product team

Hey!

I have a new project for you from the product team. Should be an interesting challenge. You can see the background and request in the email below.

I would like you to perform the analysis and write a short report for me. I want to be able to review your code as well as read your thought process for each step. I also want you to prepare and deliver the presentation for the product team - you are ready for the challenge!

They want us to predict which recipes will be popular 80% of the time and minimize the chance of showing unpopular recipes. I don't think that is realistic in the time we have, but do your best and present whatever you find.

You can find more details about what I expect you to do here. And information on the data here.

I will be on vacation for the next couple of weeks, but I know you can do this without my support. If you need to make any decisions, include them in your work and I will review them when I am back.

Good Luck!

From: Product Manager - Recipe Discovery
To: Head of Data Science
Received: Yesterday
Subject: Can you help us predict popular recipes?

Hi,

We haven't met before but I am responsible for choosing which recipes to display on the homepage each day. I have heard about what the data science team is capable of and I was wondering if you can help me choose which recipes we should display on the home page?

At the moment, I choose my favorite recipe from a selection and display that on the home page. We have noticed that traffic to the rest of the website goes up by as much as 40% if I pick a popular recipe. But I don't know how to decide if a recipe will be popular. More traffic means more subscriptions so this is really important to the company.

Can your team: - Predict which recipes will lead to high traffic? - Correctly predict high traffic recipes 80% of the time?

We need to make a decision on this soon, so I need you to present your results to me by the end of the month. Whatever your results, what do you recommend we do next?

Look forward to seeing your presentation.

About Tasty Bytes

Tasty Bytes was founded in 2020 in the midst of the Covid Pandemic. The world wanted inspiration so we decided to provide it. We started life as a search engine for recipes, helping people to find ways to use up the limited supplies they had at home.

Now, over two years on, we are a fully fledged business. For a monthly subscription we will put together a full meal plan to ensure you and your family are getting a healthy, balanced diet whatever your budget. Subscribe to our premium plan and we will also deliver the ingredients to your door.

Example Recipe

This is an example of how a recipe may appear on the website, we haven't included all of the steps but you should get an idea of what visitors to the site see.

Tomato Soup

Servings: 4
Time to make: 2 hours
Category: Lunch/Snack
Cost per serving: $

Nutritional Information (per serving) - Calories 123 - Carbohydrate 13g - Sugar 1g - Protein 4g

Ingredients: - Tomatoes - Onion - Carrot - Vegetable Stock

Method: 1. Cut the tomatoes into quarters….

Data Information

The product manager has tried to make this easier for us and provided data for each recipe, as well as whether there was high traffic when the recipe was featured on the home page.

As you will see, they haven't given us all of the information they have about each recipe.

You can find the data here.

I will let you decide how to process it, just make sure you include all your decisions in your report.

Don't forget to double check the data really does match what they say - it might not.

This dataset contains the Department of Transport and Main Roads road location details (both spatial and through distance) as well as associated traffic data.

It allows users to locate themselves with respect to road section number and through distance using the spatial coordinates on the state-controlled road network.

Through distance – the distance in kilometres measured from the gazetted start point of the road section.

Note: "Road location and traffic data" resource has been updated as of June 2025.

Context

The Google Merchandise Store sells Google branded merchandise. The data is typical of what you would see for an ecommerce website.

Content

The sample dataset contains Google Analytics 360 data from the Google Merchandise Store, a real ecommerce store. The Google Merchandise Store sells Google branded merchandise. The data is typical of what you would see for an ecommerce website. It includes the following kinds of information:

Traffic source data: information about where website visitors originate. This includes data about organic traffic, paid search traffic, display traffic, etc. Content data: information about the behavior of users on the site. This includes the URLs of pages that visitors look at, how they interact with content, etc. Transactional data: information about the transactions that occur on the Google Merchandise Store website.

Fork this kernel to get started.

Acknowledgements

Data from: https://bigquery.cloud.google.com/table/bigquery-public-data:google_analytics_sample.ga_sessions_20170801

Banner Photo by Edho Pratama from Unsplash.

Inspiration

What is the total number of transactions generated per device browser in July 2017?

The real bounce rate is defined as the percentage of visits with a single pageview. What was the real bounce rate per traffic source?

What was the average number of product pageviews for users who made a purchase in July 2017?

What was the average number of product pageviews for users who did not make a purchase in July 2017?

What was the average total transactions per user that made a purchase in July 2017?

What is the average amount of money spent per session in July 2017?

What is the sequence of pages viewed?

The map layers in this service provide color-coded maps of the traffic conditions you can expect for the present time (the default). The map shows present traffic as a blend of live and typical information. Live speeds are used wherever available and are established from real-time sensor readings. Typical speeds come from a record of average speeds, which are collected over several weeks within the last year or so. Layers also show current incident locations where available. By changing the map time, the service can also provide past and future conditions. Live readings from sensors are saved for 12 hours, so setting the map time back within 12 hours allows you to see a actual recorded traffic speeds, supplemented with typical averages by default. You can choose to turn off the average speeds and see only the recorded live traffic speeds for any time within the 12-hour window. Predictive traffic conditions are shown for any time in the future.The color-coded traffic map layer can be used to represent relative traffic speeds; this is a common type of a map for online services and is used to provide context for routing, navigation, and field operations. A color-coded traffic map can be requested for the current time and any time in the future. A map for a future request might be used for planning purposes.The map also includes dynamic traffic incidents showing the location of accidents, construction, closures, and other issues that could potentially impact the flow of traffic. Traffic incidents are commonly used to provide context for routing, navigation and field operations. Incidents are not features; they cannot be exported and stored for later use or additional analysis.Data sourceEsri’s typical speed records and live and predictive traffic feeds come directly from HERE (www.HERE.com). HERE collects billions of GPS and cell phone probe records per month and, where available, uses sensor and toll-tag data to augment the probe data collected. An advanced algorithm compiles the data and computes accurate speeds. The real-time and predictive traffic data is updated every five minutes through traffic feeds.Data coverageThe service works globally and can be used to visualize traffic speeds and incidents in many countries. Check the service coverage web map to determine availability in your area of interest. Look at the coverage map to learn whether a country currently supports traffic. The support for traffic incidents can be determined by identifying a country. For detailed information on this service, visit the directions and routing documentation and the ArcGIS Help.SymbologyTraffic speeds are displayed as a percentage of free-flow speeds, which is frequently the speed limit or how fast cars tend to travel when unencumbered by other vehicles. The streets are color coded as follows:Green (fast): 85 - 100% of free flow speedsYellow (moderate): 65 - 85%Orange (slow); 45 - 65%Red (stop and go): 0 - 45%To view live traffic only—that is, excluding typical traffic conditions—enable the Live Traffic layer and disable the Traffic layer. (You can find these layers under World/Traffic > [region] > [region] Traffic). To view more comprehensive traffic information that includes live and typical conditions, disable the Live Traffic layer and enable the Traffic layer.ArcGIS Online organization subscriptionImportant Note:The World Traffic map service is available for users with an ArcGIS Online organizational subscription. To access this map service, you'll need to sign in with an account that is a member of an organizational subscription. If you don't have an organizational subscription, you can create a new account and then sign up for a 30-day trial of ArcGIS Online.

A dataset explaining organic traffic, its importance for SEO, and methods to track it in Google Analytics 4.

The traffic management system map contains major and minor roadways. With relation to continuous counting stations and TMS sensors.Raw and aggregated speed, volume and classification data in 5-minute bins for all Traffic Monitoring Station (TMS) sensors on UDOT roads. Data is available within five minutes using the Freeway PeMS website. See training videos for additional information.This layer contains traffic statistics collected or obtained by the Transportation Monitoring Unit and developed and analyzed by the Traffic Analysis section of the Technology & Innovation Division at the Utah Department of Transportation. These statistics are intended for transportation management, private businesses, and public use. The traffic information is used in planning, programming, highway design, maintenance, traffic control, and general administration of highway systems.The traffic information contained represents the Annual Average Daily Traffic (AADT) on road sections of State Highway and Federal-Aided roads. These statistics are developed by the Traffic Analysis section through the use of the following count site types:1. Continuous Count Stations (CCS) provided by the Utah Department of Transportation. A statewide system of permanent vehicle count stations that collect volume, vehicle classification (length), and speed, 24 hours per day, 365 days per year.2. Short-term counts administered by the Utah Department of Transportation, each traffic section is generally counted about every 3 years for a duration of 48 hours, and seasonally factored using factors created by CCS stations.AADT represents traffic for both directions of travel. The routes are sectioned according to:a. Major intersectionsb. Sections where traffic volumes show a significant difference in AADT. Generally 20% under 5,000 AADT to 5% for over 50,000 AADT.Truck Statistics are available for State routes only. Single-Unit Truck (FHWA Vehicle Axle Classifications 4-7), and Combination Unit Trucks (FHWA Vehicle Axle Classifications 8-13). The values are represented by a percentage of the AADT reported for the same year.Example: A roadway with 1000 AADT has 0.1500 SUTRK reported, is 15% of the AADT are Single-Unit Trucks (or 150 SUTRK AADT).For AADT reported as Rounded, reports and maps use the AASHTO recommendation as follows:AADT = 0 - 999: Round to the nearest 10thAADT = 1,000 - 9,999: Round to the nearest 100thAADT >= 10,000: Round to the nearest 1000thNOTE: If a statistical calculation is needed to be applied to the AADT, (such as when calculating Vehicle Miles Traveled), it is not recommended to use rounded AADT’s. Unrounded AADT’s and truck percentages for are available at UDOT's Open Data Portal for download.This service is specially configured for use with UDOT's Open Data Portal. Please see the Data Assessment Form for more information. To download either the Rounded or Unrounded AADT for this data please visit the UDOT Open Data Portal and download the available Shapefile and CSV's.

In 2024, most of the global website traffic was still generated by humans, but bot traffic is constantly growing. Fraudulent traffic through bad bot actors accounted for 37 percent of global web traffic in the most recently measured period, representing an increase of 12 percent from the previous year. Sophistication of Bad Bots on the rise The complexity of malicious bot activity has dramatically increased in recent years. Advanced bad bots have doubled in prevalence over the past 2 years, indicating a surge in the sophistication of cyber threats. Simultaneously, the share of simple bad bots drastically increased over the last years, suggesting a shift in the landscape of automated threats. Meanwhile, areas like food and groceries, sports, gambling, and entertainment faced the highest amount of advanced bad bots, with more than 70 percent of their bot traffic affected by evasive applications. Good and bad bots across industries The impact of bot traffic varies across different sectors. Bad bots accounted for over 50 percent of the telecom and ISPs, community and society, and computing and IT segments web traffic. However, not all bot traffic is considered bad. Some of these applications help index websites for search engines or monitor website performance, assisting users throughout their online search. Therefore, areas like entertainment, food and groceries, and even areas targeted by bad bots themselves experienced notable levels of good bot traffic, demonstrating the diverse applications of benign automated systems across different sectors.

Due to the COVID-19 pandemic, many governments imposed lockdowns that forced hundreds of millions to stay at home. As a result of these measures, Internet traffic of residential users increased, in particular, for remote working, entertainment, commerce, and education. In turn, traffic demands in the Internet core shifted as well. In this paper, using data from a diverse set of vantage points (one ISP, three IXPs, and one metropolitan educational network), we study the effect of these lockdowns on traffic shifts. We find that the traffic volume increased by 15-20% almost within a week---while overall still modest, this constitutes a large increase within this short time period. The Internet infrastructure is able to handle this increase, as most traffic shifts occur outside of traditional peak hours. When looking at traffic sources, we find that while hypergiants still contribute a significant fraction of traffic, we see a higher increase in traffic of non-hypergiants. We observe traffic increases in applications that people use when at home, such as Web conferencing, VPN, and gaming. While many networks see increased traffic demands, in particular, residential users, others see major decreases, e.g., in the educational network.

In May 2025, the United States accounted for over 50 percent of traffic to the online search website search.yahoo.com. Taiwan and the United Kingdom ranked second and third, accounting for 6.24 percent and 5.43 percent of web visits to the platform each. Meanwhile, the domain Yahoo.com also received a similar distribution of its traffic from the United States and the countries composing the rest of its ranking.

This statistic presents the distribution of global e-commerce sessions as of October 2019, by source and medium. During the measured period, search traffic generated ** percent of total e-commerce session. Overall, ** percent were generated through organic search traffic and ** percent were generated through paid search.

According to the results of a survey conducted worldwide in 2023, nearly **** of responding digital marketers believed artificial intelligence (AI) would have a positive impact on website search traffic in the next five years. Some ** percent stated AI would have a neutral effect, while ** percent agreed that the technology would negatively impact search traffic.

The map layers in this service provide color-coded maps of the traffic conditions you can expect for the present time (the default). The map shows present traffic as a blend of live and typical information. Live speeds are used wherever available and are established from real-time sensor readings. Typical speeds come from a record of average speeds, which are collected over several weeks within the last year or so. Layers also show current incident locations where available. By changing the map time, the service can also provide past and future conditions. Live readings from sensors are saved for 12 hours, so setting the map time back within 12 hours allows you to see a actual recorded traffic speeds, supplemented with typical averages by default. You can choose to turn off the average speeds and see only the recorded live traffic speeds for any time within the 12-hour window. Predictive traffic conditions are shown for any time in the future.The color-coded traffic map layer can be used to represent relative traffic speeds; this is a common type of a map for online services and is used to provide context for routing, navigation, and field operations. A color-coded traffic map can be requested for the current time and any time in the future. A map for a future request might be used for planning purposes.The map also includes dynamic traffic incidents showing the location of accidents, construction, closures, and other issues that could potentially impact the flow of traffic. Traffic incidents are commonly used to provide context for routing, navigation and field operations. Incidents are not features; they cannot be exported and stored for later use or additional analysis.Data sourceEsri’s typical speed records and live and predictive traffic feeds come directly from HERE (www.HERE.com). HERE collects billions of GPS and cell phone probe records per month and, where available, uses sensor and toll-tag data to augment the probe data collected. An advanced algorithm compiles the data and computes accurate speeds. The real-time and predictive traffic data is updated every five minutes through traffic feeds.Data coverageThe service works globally and can be used to visualize traffic speeds and incidents in many countries. Check the service coverage web map to determine availability in your area of interest. Look at the coverage map to learn whether a country currently supports traffic. The support for traffic incidents can be determined by identifying a country. For detailed information on this service, visit the directions and routing documentation and the ArcGIS Help.SymbologyTraffic speeds are displayed as a percentage of free-flow speeds, which is frequently the speed limit or how fast cars tend to travel when unencumbered by other vehicles. The streets are color coded as follows:Green (fast): 85 - 100% of free flow speedsYellow (moderate): 65 - 85%Orange (slow); 45 - 65%Red (stop and go): 0 - 45%To view live traffic only—that is, excluding typical traffic conditions—enable the Live Traffic layer and disable the Traffic layer. (You can find these layers under World/Traffic > [region] > [region] Traffic). To view more comprehensive traffic information that includes live and typical conditions, disable the Live Traffic layer and enable the Traffic layer.

A dataset providing information of the vehicle types and counts in several locations in Leeds. Purpose of the project The aim of this work was to examine the profile of vehicle types in Leeds, in order to compare local emissions with national predictions. Traffic was monitored for a period of one week at two Inner Ring Road locations in April 2016 and at seven sites around the city in June 2016. The vehicle registration data was then sent to the Department for Transport (Dft), who combined it with their vehicle type data, replacing the registration number with an anonymised ‘Unique ID’. The data is provided in three folders:- Raw Data – contains the data in the format it was received, and a sample of each format. Processed Data – the data after processing by LCC, lookup tables, and sample data. Outputs – Excel spreadsheets summarising the data for each site, for various time/dates. Initially a dataset was received for the Inner Ring Road (see file “IRR ANPR matched to DFT vehicle type list.csv”), with vehicle details, but with missing / uncertain data on the vehicles emissions Eurostandard class. Of the 820,809 recorded journeys, from the pseudo registration number field (UniqueID) it was determined that there were 229,891 unique vehicles, and 31,912 unique “vehicle types” based on the unique concatenated vehicle description fields. It was therefore decided to import the data into an MS Access database, create a table of vehicle types, and to add the necessary fields/data so that combined with the year of manufacture / vehicle registration, the appropriate Eurostandard could be determined for the particular vehicle. The criteria for the Eurostandards was derived mainly from www.dieselnet.com and summarised in a spreadsheet (“EuroStandards.xlsx”). Vehicle types were assigned to a “VehicleClass” (see “Lookup Tables.xlsx”) and “EU class” with additional fields being added for any modified data (Gross Vehicle Weight – “GVM_Mod”; Engine capacity – “EngineCC_mod”; No of passenger seats – “PassSeats”; and Kerb weight – “KerbWt”). Missing data was added from the internet lookups, extrapolation from known data, and by association – eg 99% of cars with an engine size Additional data was then received from the Inner Ring Road site, giving journey date/time and incorporating the Taxi data for licensed taxis in Leeds. Similar data for Sites 1-7 was also then received, and processed to determine the “VehicleClass” and “EU class”. A mixture of Update queries, and VBA processing was then used to provide the Level 1-6 breakdown of vehicle types (see “Lookup Tables.xlsx”). The data was then combined into one database, so that the required Excel spreadsheets could be exported for the required time/date periods (see “outputs” folder).

This is a dynamic traffic map service with capabilities for visualizing traffic speeds relative to free-flow speeds as well as traffic incidents which can be visualized and identified. The traffic data is updated every five minutes. Traffic speeds are displayed as a percentage of free-flow speeds, which is frequently the speed limit or how fast cars tend to travel when unencumbered by other vehicles. The streets are color coded as follows: Green (fast): 85 - 100% of free flow speeds Yellow (moderate): 65 - 85% Orange (slow); 45 - 65% Red (stop and go): 0 - 45%Esri's historical, live, and predictive traffic feeds come directly from TomTom (www.tomtom.com). Historical traffic is based on the average of observed speeds over the past year. The live and predictive traffic data is updated every five minutes through traffic feeds. The color coded traffic map layer can be used to represent relative traffic speeds; this is a common type of a map for online services and is used to provide context for routing, navigation and field operations. The traffic map layer contains two sublayers: Traffic and Live Traffic. The Traffic sublayer (shown by default) leverages historical, live and predictive traffic data; while the Live Traffic sublayer is calculated from just the live and predictive traffic data only. A color coded traffic map image can be requested for the current time and any time in the future. A map image for a future request might be used for planning purposes. The map layer also includes dynamic traffic incidents showing the location of accidents, construction, closures and other issues that could potentially impact the flow of traffic. Traffic incidents are commonly used to provide context for routing, navigation and field operations. Incidents are not features; they cannot be exported and stored for later use or additional analysis. The service works globally and can be used to visualize traffic speeds and incidents in many countries. Check the service coverage web map to determine availability in your area of interest. In the coverage map, the countries color coded in dark green support visualizing live traffic. The support for traffic incidents can be determined by identifying a country. For detailed information on this service, including a data coverage map, visit the directions and routing documentation and ArcGIS Help.

In March 2024, search platform Google.com generated approximately 85.5 billion visits, down from 87 billion platform visits in October 2023. Google is a global search platform and one of the biggest online companies worldwide.