Code:

Packet_Features_Generator.py & Features.py

To run this code:

pkt_features.py [-h] -i TXTFILE [-x X] [-y Y] [-z Z] [-ml] [-s S] -j

-h, --help show this help message and exit -i TXTFILE input text file -x X Add first X number of total packets as features. -y Y Add first Y number of negative packets as features. -z Z Add first Z number of positive packets as features. -ml Output to text file all websites in the format of websiteNumber1,feature1,feature2,... -s S Generate samples using size s. -j

Purpose:

Turns a text file containing lists of incomeing and outgoing network packet sizes into separate website objects with associative features.

Uses Features.py to calcualte the features.

startMachineLearning.sh & machineLearning.py

To run this code:

bash startMachineLearning.sh

This code then runs machineLearning.py in a tmux session with the nessisary file paths and flags

Options (to be edited within this file):

--evaluate-only to test 5 fold cross validation accuracy

--test-scaling-normalization to test 6 different combinations of scalers and normalizers

Note: once the best combination is determined, it should be added to the data_preprocessing function in machineLearning.py for future use

--grid-search to test the best grid search hyperparameters - note: the possible hyperparameters must be added to train_model under 'if not evaluateOnly:' - once best hyperparameters are determined, add them to train_model under 'if evaluateOnly:'

Purpose:

Using the .ml file generated by Packet_Features_Generator.py & Features.py, this program trains a RandomForest Classifier on the provided data and provides results using cross validation. These results include the best scaling and normailzation options for each data set as well as the best grid search hyperparameters based on the provided ranges.

Data

Encrypted network traffic was collected on an isolated computer visiting different Wikipedia and New York Times articles, different Google search queres (collected in the form of their autocomplete results and their results page), and different actions taken on a Virtual Reality head set.

Data for this experiment was stored and analyzed in the form of a txt file for each experiment which contains:

First number is a classification number to denote what website, query, or vr action is taking place.

The remaining numbers in each line denote:

The size of a packet,

and the direction it is traveling.

negative numbers denote incoming packets

positive numbers denote outgoing packets

Figure 4 Data

This data uses specific lines from the Virtual Reality.txt file.

The action 'LongText Search' refers to a user searching for "Saint Basils Cathedral" with text in the Wander app.

The action 'ShortText Search' refers to a user searching for "Mexico" with text in the Wander app.

The .xlsx and .csv file are identical

Each file includes (from right to left):

The origional packet data,

each line of data organized from smallest to largest packet size in order to calculate the mean and standard deviation of each packet capture,

and the final Cumulative Distrubution Function (CDF) caluclation that generated the Figure 4 Graph.

Attributes of sites in Hamilton City which collect anonymised data from a sample of vehicles. Note: A Link is the section of the road between two sites

Column_InfoSite_Id, int : Unique identiferNumber, int : Asset number. Note: If the site is at a signalised intersection, Number will match 'Site_Number' in the table 'Traffic Signal Site Location'Is_Enabled, varchar : Site is currently enabledDisabled_Date, datetime : If currently disabled, the date at which the site was disabledSite_Name, varchar : Description of the site locationLatitude, numeric : North-south geographic coordinatesLongitude, numeric : East-west geographic coordinates

Relationship









Disclaimer

Hamilton City Council does not make any representation or give any warranty as to the accuracy or exhaustiveness of the data released for public download. Levels, locations and dimensions of works depicted in the data may not be accurate due to circumstances not notified to Council. A physical check should be made on all levels, locations and dimensions before starting design or works.

Hamilton City Council shall not be liable for any loss, damage, cost or expense (whether direct or indirect) arising from reliance upon or use of any data provided, or Council's failure to provide this data.

While you are free to crop, export and re-purpose the data, we ask that you attribute the Hamilton City Council and clearly state that your work is a derivative and not the authoritative data source. Please include the following statement when distributing any work derived from this data:

‘This work is derived entirely or in part from Hamilton City Council data; the provided information may be updated at any time, and may at times be out of date, inaccurate, and/or incomplete.'

According to the results of a survey conducted worldwide in 2023, nearly half of responding digital marketers believed artificial intelligence (AI) would have a positive impact on website search traffic in the next five years. Some 20 percent stated AI would have a neutral effect, while 30 percent agreed that the technology would negatively impact search traffic.

Website Traffic Analysis

Website traffic analysis is the process of monitoring and evaluating the visitors to a website. It provides insights into how users are interacting with the site, where they are coming from, which pages they visit most often, and how long they stay. By analyzing this data, businesses can understand user behavior, improve site performance, and optimize content to increase engagement and conversions.

Key metrics include the number of visitors, page views, bounce rate, traffic sources (organic, referral, direct), and geographic location. Website traffic analysis is essential for enhancing SEO, refining marketing strategies, and boosting overall user experience.

Mobile accounts for approximately half of web traffic worldwide. In the last quarter of 2024, mobile devices (excluding tablets) generated 62.54 percent of global website traffic. Mobiles and smartphones consistently hoovered around the 50 percent mark since the beginning of 2017, before surpassing it in 2020. Mobile traffic Due to low infrastructure and financial restraints, many emerging digital markets skipped the desktop internet phase entirely and moved straight onto mobile internet via smartphone and tablet devices. India is a prime example of a market with a significant mobile-first online population. Other countries with a significant share of mobile internet traffic include Nigeria, Ghana and Kenya. In most African markets, mobile accounts for more than half of the web traffic. By contrast, mobile only makes up around 45.49 percent of online traffic in the United States. Mobile usage The most popular mobile internet activities worldwide include watching movies or videos online, e-mail usage and accessing social media. Apps are a very popular way to watch video on the go and the most-downloaded entertainment apps in the Apple App Store are Netflix, Tencent Video and Amazon Prime Video.

📊 Website Performance & SEO Metrics: GSC, Yandex Metrika & Page Parsing

This dataset contains real-world data collected from a live website, integrating insights from three powerful sources:

• Google Search Console (GSC) — search performance metrics including clicks, impressions, and average positions.
• Yandex Metrica — behavioral analytics like user visits, bounce rates, and session depths.
• On-page Parsing — extracted metadata and structural elements directly from the site's pages.

The dataset covers a specific time period, offering a rich ground for analysis, modeling, and discovery.

🔍 What You Can Do:

• Explore correlations between technical SEO and user behavior
• Investigate how search visibility affects on-site interactions
• Build predictive models to estimate the number of clicks based on other metrics
• Perform feature engineering and multisource data blending

Whether you're into digital marketing, data science, or SEO analytics, this dataset provides a hands-on opportunity to dive deep into web performance data and develop actionable insights.

As of 2019, direct traffic accounts for the largest percentage of website traffic worldwide, with a share of 55 percent. Additionally, search traffic accounts for 29 percent of worldwide website traffic.

In March 2024, search platform Google.com generated approximately 85.5 billion visits, down from 87 billion platform visits in October 2023. Google is a global search platform and one of the biggest online companies worldwide.

Context

The Google Merchandise Store sells Google branded merchandise. The data is typical of what you would see for an ecommerce website.

Content

The sample dataset contains Google Analytics 360 data from the Google Merchandise Store, a real ecommerce store. The Google Merchandise Store sells Google branded merchandise. The data is typical of what you would see for an ecommerce website. It includes the following kinds of information:

Traffic source data: information about where website visitors originate. This includes data about organic traffic, paid search traffic, display traffic, etc. Content data: information about the behavior of users on the site. This includes the URLs of pages that visitors look at, how they interact with content, etc. Transactional data: information about the transactions that occur on the Google Merchandise Store website.

Fork this kernel to get started.

Acknowledgements

Data from: https://bigquery.cloud.google.com/table/bigquery-public-data:google_analytics_sample.ga_sessions_20170801

Banner Photo by Edho Pratama from Unsplash.

Inspiration

What is the total number of transactions generated per device browser in July 2017?

The real bounce rate is defined as the percentage of visits with a single pageview. What was the real bounce rate per traffic source?

What was the average number of product pageviews for users who made a purchase in July 2017?

What was the average number of product pageviews for users who did not make a purchase in July 2017?

What was the average total transactions per user that made a purchase in July 2017?

What is the average amount of money spent per session in July 2017?

What is the sequence of pages viewed?

As online shopping experienced an exponential growth during the months of the COVID-19 induced lockdown in France, the source wanted to measure the share of search traffic of the different cosmetic websites. Thus, we note that around 24 percent of visits to Beautysuccess.fr come from search traffic, that is to say, visits made coming from search engines such as Google.

DataForSEO Labs API offers three powerful keyword research algorithms and historical keyword data:

• Related Keywords from the “searches related to” element of Google SERP. • Keyword Suggestions that match the specified seed keyword with additional words before, after, or within the seed key phrase. • Keyword Ideas that fall into the same category as specified seed keywords. • Historical Search Volume with current cost-per-click, and competition values.

Based on in-market categories of Google Ads, you can get keyword ideas from the relevant Categories For Domain and discover relevant Keywords For Categories. You can also obtain Top Google Searches with AdWords and Bing Ads metrics, product categories, and Google SERP data.

You will find well-rounded ways to scout the competitors:

• Domain Whois Overview with ranking and traffic info from organic and paid search. • Ranked Keywords that any domain or URL has positions for in SERP. • SERP Competitors and the rankings they hold for the keywords you specify. • Competitors Domain with a full overview of its rankings and traffic from organic and paid search. • Domain Intersection keywords for which both specified domains rank within the same SERPs. • Subdomains for the target domain you specify along with the ranking distribution across organic and paid search. • Relevant Pages of the specified domain with rankings and traffic data. • Domain Rank Overview with ranking and traffic data from organic and paid search. • Historical Rank Overview with historical data on rankings and traffic of the specified domain from organic and paid search. • Page Intersection keywords for which the specified pages rank within the same SERP.

All DataForSEO Labs API endpoints function in the Live mode. This means you will be provided with the results in response right after sending the necessary parameters with a POST request.

The limit is 2000 API calls per minute, however, you can contact our support team if your project requires higher rates.

We offer well-rounded API documentation, GUI for API usage control, comprehensive client libraries for different programming languages, free sandbox API testing, ad hoc integration, and deployment support.

We have a pay-as-you-go pricing model. You simply add funds to your account and use them to get data. The account balance doesn't expire.

In January 2024, users who reached Reddit.com from links displayed after launching a research on search engines like Google or Yahoo generated over 4.6 billion visits. Between April 2022 and January 2024, search traffic volumes to Reddit experienced a positive trend.

Unlock the Power of Behavioural Data with GDPR-Compliant Clickstream Insights.

Swash clickstream data offers a comprehensive and GDPR-compliant dataset sourced from users worldwide, encompassing both desktop and mobile browsing behaviour. Here's an in-depth look at what sets us apart and how our data can benefit your organisation.

User-Centric Approach: Unlike traditional data collection methods, we take a user-centric approach by rewarding users for the data they willingly provide. This unique methodology ensures transparent data collection practices, encourages user participation, and establishes trust between data providers and consumers.

Wide Coverage and Varied Categories: Our clickstream data covers diverse categories, including search, shopping, and URL visits. Whether you are interested in understanding user preferences in e-commerce, analysing search behaviour across different industries, or tracking website visits, our data provides a rich and multi-dimensional view of user activities.

GDPR Compliance and Privacy: We prioritise data privacy and strictly adhere to GDPR guidelines. Our data collection methods are fully compliant, ensuring the protection of user identities and personal information. You can confidently leverage our clickstream data without compromising privacy or facing regulatory challenges.

Market Intelligence and Consumer Behaviuor: Gain deep insights into market intelligence and consumer behaviour using our clickstream data. Understand trends, preferences, and user behaviour patterns by analysing the comprehensive user-level, time-stamped raw or processed data feed. Uncover valuable information about user journeys, search funnels, and paths to purchase to enhance your marketing strategies and drive business growth.

High-Frequency Updates and Consistency: We provide high-frequency updates and consistent user participation, offering both historical data and ongoing daily delivery. This ensures you have access to up-to-date insights and a continuous data feed for comprehensive analysis. Our reliable and consistent data empowers you to make accurate and timely decisions.

Custom Reporting and Analysis: We understand that every organisation has unique requirements. That's why we offer customisable reporting options, allowing you to tailor the analysis and reporting of clickstream data to your specific needs. Whether you need detailed metrics, visualisations, or in-depth analytics, we provide the flexibility to meet your reporting requirements.

Data Quality and Credibility: We take data quality seriously. Our data sourcing practices are designed to ensure responsible and reliable data collection. We implement rigorous data cleaning, validation, and verification processes, guaranteeing the accuracy and reliability of our clickstream data. You can confidently rely on our data to drive your decision-making processes.

This map contains a dynamic traffic map service with capabilities for visualizing traffic speeds relative to free-flow speeds as well as traffic incidents which can be visualized and identified. The traffic data is updated every five minutes. Traffic speeds are displayed as a percentage of free-flow speeds, which is frequently the speed limit or how fast cars tend to travel when unencumbered by other vehicles. The streets are color coded as follows:Green (fast): 85 - 100% of free flow speedsYellow (moderate): 65 - 85%Orange (slow); 45 - 65%Red (stop and go): 0 - 45%Esri's historical, live, and predictive traffic feeds come directly from TomTom (www.tomtom.com). Historical traffic is based on the average of observed speeds over the past year. The live and predictive traffic data is updated every five minutes through traffic feeds. The color coded traffic map layer can be used to represent relative traffic speeds; this is a common type of a map for online services and is used to provide context for routing, navigation and field operations. The traffic map layer contains two sublayers: Traffic and Live Traffic. The Traffic sublayer (shown by default) leverages historical, live and predictive traffic data; while the Live Traffic sublayer is calculated from just the live and predictive traffic data only. A color coded traffic map can be requested for the current time and any time in the future. A map for a future request might be used for planning purposes. The map also includes dynamic traffic incidents showing the location of accidents, construction, closures and other issues that could potentially impact the flow of traffic. Traffic incidents are commonly used to provide context for routing, navigation and field operations. Incidents are not features; they cannot be exported and stored for later use or additional analysis. The service works globally and can be used to visualize traffic speeds and incidents in many countries. Check the service coverage web map to determine availability in your area of interest. In the coverage map, the countries color coded in dark green support visualizing live traffic. The support for traffic incidents can be determined by identifying a country. For detailed information on this service, including a data coverage map, visit the directions and routing documentation and ArcGIS Help.

In 2025, Google was the most used search engine in Morocco, accounting for nearly 97 percent of the web traffic. The next most used search engine was Bing, which made up over two percent of web traffic in Morocco. The number of people using the internet in Morocco stood at 35.3 million in 2025, the fifth highest amount of internet users in Africa.

Recorded volume data at SCATS intersections or pedestrian crossings in Hamilton. To get data for this dataset, please call the API directly talking to the HCC Data Warehouse: https://api.hcc.govt.nz/OpenData/get_traffic_signal_detector_count?Page=1&Start_Date=2020-10-01&End_Date=2020-10-02. For this API, there are three mandatory parameters: Page, Start_Date, End_Date. Sample values for these parameters are in the link above. When calling the API for the first time, please always start with Page 1. Then from the returned JSON, you can see more information such as the total page count and page size. For help on using the API in your preferred data analysis software, please contact dale.townsend@hcc.govt.nz. NOTE: Anomalies and missing data may be present in the dataset.

Column_InfoSite_Number, int : SCATS ID - Unique identifierDetector_Number, int : Detector number that the count is recorded toDate, datetime : Start of the 15 minute time interval that the count was recorded forCount, int : Number of vehicles that passed over the detector

Relationship

This table reference to table Traffic_Signal_Detector

Analytics

For convenience Hamilton City Council has also built a Quick Analytics Dashboard over this dataset that you can access here.

Disclaimer

Hamilton City Council does not make any representation or give any warranty as to the accuracy or exhaustiveness of the data released for public download. Levels, locations and dimensions of works depicted in the data may not be accurate due to circumstances not notified to Council. A physical check should be made on all levels, locations and dimensions before starting design or works.

Hamilton City Council shall not be liable for any loss, damage, cost or expense (whether direct or indirect) arising from reliance upon or use of any data provided, or Council's failure to provide this data.

While you are free to crop, export and re-purpose the data, we ask that you attribute the Hamilton City Council and clearly state that your work is a derivative and not the authoritative data source. Please include the following statement when distributing any work derived from this data:

‘This work is derived entirely or in part from Hamilton City Council data; the provided information may be updated at any time, and may at times be out of date, inaccurate, and/or incomplete.'

You can also access an API version of this dataset.

TMS

(traffic monitoring system) daily-updated traffic counts API

Important note: due to the size of this dataset, you won't be able to open it fully in Excel. Use notepad / R / any software package which can open more than a million rows.

Data reuse caveats: as per license.

Data quality

statement: please read the accompanying user manual, explaining:

how

 this data is collected identification 

 of count stations traffic 

 monitoring technology monitoring 

 hierarchy and conventions typical 

 survey specification data 

 calculation TMS 

 operation. 

Traffic

monitoring for state highways: user manual

[PDF 465 KB]

The data is at daily granularity. However, the actual update

frequency of the data depends on the contract the site falls within. For telemetry

sites it's once a week on a Wednesday. Some regional sites are fortnightly, and

some monthly or quarterly. Some are only 4 weeks a year, with timing depending

on contractors’ programme of work.

Data quality caveats: you must use this data in

conjunction with the user manual and the following caveats.

The

 road sensors used in data collection are subject to both technical errors and 

 environmental interference.Data 

 is compiled from a variety of sources. Accuracy may vary and the data 

 should only be used as a guide.As 

 not all road sections are monitored, a direct calculation of Vehicle 

 Kilometres Travelled (VKT) for a region is not possible.Data 

 is sourced from Waka Kotahi New Zealand Transport Agency TMS data.For 

 sites that use dual loops classification is by length. Vehicles with a length of less than 5.5m are 

 classed as light vehicles. Vehicles over 11m long are classed as heavy 

 vehicles. Vehicles between 5.5 and 11m are split 50:50 into light and 

 heavy.In September 2022, the National Telemetry contract was handed to a new contractor. During the handover process, due to some missing documents and aged technology, 40 of the 96 national telemetry traffic count sites went offline. Current contractor has continued to upload data from all active sites and have gradually worked to bring most offline sites back online. Please note and account for possible gaps in data from National Telemetry Sites. 

The NZTA Vehicle

Classification Relationships diagram below shows the length classification (typically dual loops) and axle classification (typically pneumatic tube counts),

and how these map to the Monetised benefits and costs manual, table A37,

page 254.

Monetised benefits and costs manual [PDF 9 MB]

For the full TMS

classification schema see Appendix A of the traffic counting manual vehicle

classification scheme (NZTA 2011), below.

Traffic monitoring for state highways: user manual [PDF 465 KB]

State highway traffic monitoring (map)

State highway traffic monitoring sites

The dataset collection is an amalgamation of interconnected data tables sourced from 'Tilastokeskus' (the Statistical Bureau) in Finland. It includes comprehensive information from the year 2011 related to road traffic. The data within the collection is arranged in a tabular format, with each table comprising of rows and columns containing related data. This organized structure allows for easy access, analysis and interpretation of the data. The data source, the Statistical Bureau's web service interface (WFS), provides an authoritative and reliable basis for these traffic related datasets. This dataset is licensed under CC BY 4.0 (Creative Commons Attribution 4.0, https://creativecommons.org/licenses/by/4.0/deed.fi).

In November 2024, Google.com was the most popular website worldwide with 136 billion average monthly visits. The online platform has held the top spot as the most popular website since June 2010, when it pulled ahead of Yahoo into first place. Second-ranked YouTube generated more than 72.8 billion monthly visits in the measured period. The internet leaders: search, social, and e-commerce Social networks, search engines, and e-commerce websites shape the online experience as we know it. While Google leads the global online search market by far, YouTube and Facebook have become the world’s most popular websites for user generated content, solidifying Alphabet’s and Meta’s leadership over the online landscape. Meanwhile, websites such as Amazon and eBay generate millions in profits from the sale and distribution of goods, making the e-market sector an integral part of the global retail scene. What is next for online content? Powering social media and websites like Reddit and Wikipedia, user-generated content keeps moving the internet’s engines. However, the rise of generative artificial intelligence will bring significant changes to how online content is produced and handled. ChatGPT is already transforming how online search is performed, and news of Google's 2024 deal for licensing Reddit content to train large language models (LLMs) signal that the internet is likely to go through a new revolution. While AI's impact on the online market might bring both opportunities and challenges, effective content management will remain crucial for profitability on the web.

Vehicle travel time and delay data on routes in Hamilton City, based on Bluetooth sensor records. To get data for this dataset, please call the API directly talking to the HCC Data Warehouse: https://api.hcc.govt.nz/OpenData/get_traffic_route_stats?Page=1&Start_Date=2021-06-02&End_Date=2021-06-03. For this API, there are three mandatory parameters: Page, Start_Date, End_Date. Sample values for these parameters are in the link above. When calling the API for the first time, please always start with Page 1. Then from the returned JSON, you can see more information such as the total page count and page size. For help on using the API in your preferred data analysis software, please contact dale.townsend@hcc.govt.nz. NOTE: Anomalies and missing data may be present in the dataset.

Column_InfoRoute_Id, int : Unique route identifierTravel_Time, int : Average travel time in seconds to travel along the routeDelay, int : Average travel delay in seconds, calculated as the difference between the free flow travel time and observed travel timeExcess_Delay, int : Excess Delay is similar to Delay, but it ignores recurring (expected) delays associated with peak times of dayDate, varchar : Starting date and time for the recorded delay and travel time, in 15 minute periods

Relationship

This table reference to table Traffic_Route

Analytics

For convenience Hamilton City Council has also built a Quick Analytics Dashboard over this dataset that you can access here.

Disclaimer

Hamilton City Council does not make any representation or give any warranty as to the accuracy or exhaustiveness of the data released for public download. Levels, locations and dimensions of works depicted in the data may not be accurate due to circumstances not notified to Council. A physical check should be made on all levels, locations and dimensions before starting design or works.

Hamilton City Council shall not be liable for any loss, damage, cost or expense (whether direct or indirect) arising from reliance upon or use of any data provided, or Council's failure to provide this data.

While you are free to crop, export and re-purpose the data, we ask that you attribute the Hamilton City Council and clearly state that your work is a derivative and not the authoritative data source. Please include the following statement when distributing any work derived from this data:

‘This work is derived entirely or in part from Hamilton City Council data; the provided information may be updated at any time, and may at times be out of date, inaccurate, and/or incomplete.'