Context Malicious URLs or malicious website is a very serious threat to cybersecurity. Malicious URLs host unsolicited content (spam, phishing, drive-by downloads, etc.) and lure unsuspecting users to become victims of scams (monetary loss, theft of private information, and malware installation), and cause losses of billions of dollars every year. We have collected this dataset to include a large number of examples of Malicious URLs so that a machine learning-based model can be developed to identify malicious urls so that we can stop them in advance before infecting computer system or spreading through inteinternet.

Content we have collected a huge dataset of 651,191 URLs, out of which 428103 benign or safe URLs, 96457 defacement URLs, 94111 phishing URLs, and 32520 malware URLs. Figure 2 depicts their distribution in terms of percentage. As we know one of the most crucial tasks is to curate the dataset for a machine learning project. We have curated this dataset from five different sources.

For collecting benign, phishing, malware and defacement URLs we have used URL dataset (ISCX-URL-2016) For increasing phishing and malware URLs, we have used Malware domain black list dataset. We have increased benign URLs using faizan git repo At last, we have increased more number of phishing URLs using Phishtank dataset and PhishStorm dataset As we have told you that dataset is collected from different sources. So firstly, we have collected the URLs from different sources into a separate data frame and finally merge them to retain only URLs and their class type.

The goal of our research is to identify malicious advertisement URLs and to apply adversarial attack on ensembles. We extract lexical and web-scrapped features from using python code. And then 4 machine learning algorithms are applied for the classification process and then used the K-Means clustering for the visual understanding. We check the vulnerability of the models by the adversarial examples. We applied Zeroth Order Optimization adversarial attack on the models and compute the attack accuracy.

Dataset

This dataset was created by kianindeed

Released under MIT

Contents

Summary of previous works on malicious URL detection.

Dataset

This dataset was created by Shreeshail Chavan

Contents

The dataset contains DNS records, IP-related features, WHOIS/RDAP information, information from TLS handshakes and certificates, and GeoIP information for 368,956 benign domains from Cisco Umbrella, 461,338 benign domains from the actual CESNET network traffic, 164,425 phishing domains from PhishTank and OpenPhish services, and 100,809 malware domains from various sources like ThreatFox, The Firebog, MISP threat intelligence platform, and other sources. The ground truth for the phishing dataset was double-check with the VirusTotal (VT) service. Domain names not considered malicious by VT have been removed from phishing and malware datasets. Similarly, benign domain names that were considered risky by VT have been removed from the benign datasets. The data was collected between March 2023 and July 2024. The final assessment of the data was conducted in August 2024.

The dataset is useful for cybersecurity research, e.g. statistical analysis of domain data or feature extraction for training machine learning-based classifiers, e.g. for phishing and malware website detection.

Data Files

• The data is located in the following individual files:

  • benign_umbrella.json - data for 368,956 benign domains from Cisco Umbrella,
  • benign_cesnet.json - data for 461,338 benign domains from the CESNET network,
  • phishing.json - data for 164,425 phishing domains, and
  • malware.json - data for 100,809 malware domains.

Data Structure

In the second half of 2021, websites regarding manufacturing were the most common websites to be targeted by malicious URL redirections, with 39 percent of detected cases being found on these sites. Although manufacturing websites have been a common target for malware attacks before, finds on these sites have largely increased compared to the first half of the year, which recorded around 23 percent of cases redirecting through that industry.

The Web has long become a major platform for online criminal activities. URLs are used as the main vehicle in this domain. To counter this issues security community focused its efforts on developing techniques for mostly blacklisting of malicious URLs.

Dataset for Detecting Phishing URLs

This dataset contains URLs labeled as 'Safe' (0) or 'Not Safe' (1) for phishing detection tasks.

  Dataset Summary

This dataset contains URLs labeled for phishing detection tasks. It's designed to help train and evaluate models that can identify potentially malicious URLs.

  Dataset Creation

The dataset was synthetically generated using a custom script that creates both legitimate and potentially phishing URLs. This approach… See the full description on the dataset page: https://huggingface.co/datasets/darshan8950/phishing_url_classification.

Dataset

This dataset was created by Zihan ZHAO_qq

Released under Apache 2.0

Contents

Web applications are important for various online businesses and operations because of their platform stability and low operation cost. The increasing usage of Internet-of-Things (IoT) devices within a network has contributed to the rise of network intrusion issues due to malicious Uniform Resource Locators (URLs). Generally, malicious URLs are initiated to promote scams, attacks, and frauds which can lead to high-risk intrusion. Several methods have been developed to detect malicious URLs in previous works. There has been a good amount of work done to detect malicious URLs using various methods such as random forest, regression, LightGBM, and more as reported in the literature. However, most of the previous works focused on the binary classification of malicious URLs and are tested on limited URL datasets. Nevertheless, the detection of malicious URLs remains a challenging task that remains open to research. Hence, this work proposed a stacking-based ensemble classifier to perform multi-class classification of malicious URLs on larger URL datasets to justify the robustness of the proposed method. This study focuses on obtaining lexical features directly from the URL to identify malicious websites. Then, the proposed stacking-based ensemble classifier is developed by integrating Random Forest, XGBoost, LightGBM, and CatBoost. In addition, hyperparameter tuning was performed using the Randomized Search method to optimize the proposed classifier. The proposed stacking-based ensemble classifier aims to take advantage of the performance of each machine learning model and aggregate the output to improve prediction accuracy. The classification accuracies of the machine learning model when applied individually are 93.6%, 95.2%, 95.7% and 94.8% for random forest, XGBoost, LightGBM, and CatBoost respectively. The proposed stacking-based ensemble classifier has shown significant results in classifying four classes of malicious URLs (phishing, malware, defacement, and benign) with an average accuracy of 96.8% when benchmarked with previous works.

Dataset

This dataset was created by SethMDoty

Contents

Web applications are important for various online businesses and operations because of their platform stability and low operation cost. The increasing usage of Internet-of-Things (IoT) devices within a network has contributed to the rise of network intrusion issues due to malicious Uniform Resource Locators (URLs). Generally, malicious URLs are initiated to promote scams, attacks, and frauds which can lead to high-risk intrusion. Several methods have been developed to detect malicious URLs in previous works. There has been a good amount of work done to detect malicious URLs using various methods such as random forest, regression, LightGBM, and more as reported in the literature. However, most of the previous works focused on the binary classification of malicious URLs and are tested on limited URL datasets. Nevertheless, the detection of malicious URLs remains a challenging task that remains open to research. Hence, this work proposed a stacking-based ensemble classifier to perform multi-class classification of malicious URLs on larger URL datasets to justify the robustness of the proposed method. This study focuses on obtaining lexical features directly from the URL to identify malicious websites. Then, the proposed stacking-based ensemble classifier is developed by integrating Random Forest, XGBoost, LightGBM, and CatBoost. In addition, hyperparameter tuning was performed using the Randomized Search method to optimize the proposed classifier. The proposed stacking-based ensemble classifier aims to take advantage of the performance of each machine learning model and aggregate the output to improve prediction accuracy. The classification accuracies of the machine learning model when applied individually are 93.6%, 95.2%, 95.7% and 94.8% for random forest, XGBoost, LightGBM, and CatBoost respectively. The proposed stacking-based ensemble classifier has shown significant results in classifying four classes of malicious URLs (phishing, malware, defacement, and benign) with an average accuracy of 96.8% when benchmarked with previous works.

An October 2023 phishing simulation carried out at worldwide organizations found that *** percent of employees submitted passwords in the form embedded in the malicious webpage. On the other hand, *** percent of them clicked only the link, and **** percent did not click the link.

Context

Common Crawl project has fascinated me ever since I learned about it. It provides a large number of data formats and presents challenges across skill and interest areas. I am particularly interested in URL analysis for applications such as typosquatting, malicious URLs, and just about anything interesting that can be done with domain names.

Content

I have sampled 1% of the domains from the Common Crawl Index dataset that is available on AWS in Parquet format. You can read more about how I extracted this dataset @ https://harshsinghal.dev/create-a-url-dataset-for-nlp/

Acknowledgements

Thanks a ton to the folks at https://commoncrawl.org/ for making this immensely valuable resource available to the world for free. Please find their Terms of Use here.

Inspiration

My interests are in working with string similarity functions and I continue to find scalable ways of doing this. I wrote about using a Postgres extension to compute string distances and used Common Crawl URL domains as the input dataset (you can read more @ https://harshsinghal.dev/postgres-text-similarity-with-commoncrawl-domains/).

I am also interested in identifying fraudulent domains and understanding malicious URL patterns.

Dataset from: bgspaditya/byt-malicious-url-treatment

One important topic to work is to create a good set of malicious web characteristics

The global market for suspicious file and URL analysis is experiencing robust growth, projected to reach $88 million in 2025 and maintain a Compound Annual Growth Rate (CAGR) of 6.4% from 2025 to 2033. This expansion is driven by the escalating sophistication of cyber threats, the increasing reliance on digital infrastructure across various sectors, and the growing need for proactive security measures to mitigate risks associated with malicious files and URLs. The market's segmentation reveals a strong preference for cloud-based solutions, offering scalability and accessibility to organizations of all sizes. Large enterprises are the primary consumers, reflecting their higher vulnerability to advanced cyberattacks and their greater capacity for investment in robust security solutions. However, the market is also seeing significant adoption among SMEs, driven by the increasing affordability and ease of use of cloud-based solutions and a rising awareness of the risks associated with malicious online content. Several factors contribute to market growth. The development and proliferation of advanced malware necessitates continuous improvement in threat detection and analysis capabilities. Furthermore, the expanding attack surface due to remote work and the increasing use of IoT devices are contributing to a heightened demand for effective file and URL analysis tools. Regulatory compliance requirements, particularly within sensitive industries like finance and healthcare, further incentivize organizations to invest in these solutions. Conversely, challenges such as the emergence of obfuscated malware, the high cost of advanced solutions, and the need for specialized expertise pose some restraints to broader market penetration. The competitive landscape is diverse, with established cybersecurity players and innovative startups offering a range of solutions catering to specific needs and budgets. This competitive pressure is ultimately beneficial for consumers, driving innovation and fostering a more efficient and effective market for suspicious file and URL analysis.

The market for suspicious file and URL analysis is experiencing robust growth, driven by the escalating sophistication of cyber threats and the increasing reliance on digital infrastructure across various sectors. The $92 million market size in 2025, coupled with a compound annual growth rate (CAGR) of 6.7%, projects substantial expansion to approximately $150 million by 2033. This growth is fueled by several key factors. The rising frequency and severity of ransomware attacks, phishing campaigns, and malware distribution necessitate robust security solutions for proactive threat detection and response. Furthermore, the expanding adoption of cloud-based services and the increasing interconnectedness of devices amplify the attack surface, thereby increasing the demand for advanced file and URL analysis capabilities. The growing awareness of data privacy regulations, such as GDPR and CCPA, also incentivizes organizations to enhance their security posture and invest in solutions that can effectively identify and mitigate potential threats. The market landscape is highly competitive, with a diverse range of players, from established cybersecurity giants like CrowdStrike, McAfee, and Symantec to specialized providers like Any.Run and Joe Sandbox. The market's segmentation likely includes solutions based on different analysis techniques (static, dynamic, sandbox-based), deployment models (cloud, on-premise), and target users (enterprise, SMB, individuals). While the provided data lacks regional specifics, it's reasonable to expect that North America and Europe will initially dominate market share, given their advanced cybersecurity infrastructure and high rates of digital adoption. However, emerging markets in Asia-Pacific and Latin America are poised for significant growth in the coming years, driven by rising digital literacy and economic expansion. Competition will intensify as vendors strive to offer innovative features, such as AI-powered threat detection and improved integration with existing security ecosystems.

In 2023, the total detection cases of web-based malware sites in South Korea amounted to roughly 12.7 thousand, a slight decrease compared to the previous year. The highest number of detected web-based malware sites in South Korea was 47,703 cases in 2014. The type of web-based malware sites was comprised of distribution sites and staging sties.