Dirty Cafe Sales Dataset

Overview

The Dirty Cafe Sales dataset contains 10,000 rows of synthetic data representing sales transactions in a cafe. This dataset is intentionally "dirty," with missing values, inconsistent data, and errors introduced to provide a realistic scenario for data cleaning and exploratory data analysis (EDA). It can be used to practice cleaning techniques, data wrangling, and feature engineering.

File Information

• File Name: dirty_cafe_sales.csv
• Number of Rows: 10,000
• Number of Columns: 8

Columns Description

Data Characteristics

1. Missing Values:

   • Some columns (e.g., Item, Payment Method, Location) may contain missing values represented as None or empty cells.

2. Invalid Values:

   • Some rows contain invalid entries like "ERROR" or "UNKNOWN" to simulate real-world data issues.

3. Price Consistency:

   • Prices for menu items are consistent but may have missing or incorrect values introduced.

Menu Items

The dataset includes the following menu items with their respective price ranges:

Use Cases

This dataset is suitable for: - Practicing data cleaning techniques such as handling missing values, removing duplicates, and correcting invalid entries. - Exploring EDA techniques like visualizations and summary statistics. - Performing feature engineering for machine learning workflows.

Cleaning Steps Suggestions

To clean this dataset, consider the following steps: 1. Handle Missing Values: - Fill missing numeric values with the median or mean. - Replace missing categorical values with the mode or "Unknown."

1. Handle Invalid Values:

   • Replace invalid entries like "ERROR" and "UNKNOWN" with NaN or appropriate values.

2. Date Consistency:

   • Ensure all dates are in a consistent format.
   • Fill missing dates with plausible values based on nearby records.

3. Feature Engineering:

   • Create new columns, such as Day of the Week or Transaction Month, for further analysis.

License

This dataset is released under the CC BY-SA 4.0 License. You are free to use, share, and adapt it, provided you give appropriate credit.

Feedback

If you have any questions or feedback, feel free to reach out through the dataset's discussion board on Kaggle.

This dataset includes all of the data downloaded from GBIF (DOIs provided in README.md as well as below, downloaded Feb 2021) as well as data downloaded from SCAN. This dataset has 2,808,432 records and can be used as a reference to the verbatim data before it underwent the cleaning process. The only modifications made to this datset after direct download from the data portals are the following:

1) for GBIF records, I renamed the countryCode column to be "country" so that the column title is consistent across both GBIF and SCAN 2) A source column was added where I specify if the record came from GBIF or SCAN 3) Duplicate records across SCAN and GBIF were removed by identifying identical instances "catalogNumber" and "institutionCode" 4) Only the Darwin core columns (DwC) that were shared across downloaded datasets were retained. GBIF contained ~249 DwC variables, and SCAN data contained fewer, so this combined dataset only includes the ~80 columns shared between the two datasets

For GBIF, we downloaded the data in three separate chunks, therefore there are three DOIs. See below:

GBIF.org (3 February 2021) GBIF Occurrence Downloadhttps://doi.org/10.15468/dl.6cxfsw GBIF.org (3 February 2021) GBIF Occurrence Downloadhttps://doi.org/10.15468/dl.b9rfa7 GBIF.org (3 February 2021) GBIF Occurrence Downloadhttps://doi.org/10.15468/dl.w2nndm

Sample data for exercises in Further Adventures in Data Cleaning.

All data are prone to error and require data cleaning prior to analysis. An important example is longitudinal growth data, for which there are no universally agreed standard methods for identifying and removing implausible values and many existing methods have limitations that restrict their usage across different domains. A decision-making algorithm that modified or deleted growth measurements based on a combination of pre-defined cut-offs and logic rules was designed. Five data cleaning methods for growth were tested with and without the addition of the algorithm and applied to five different longitudinal growth datasets: four uncleaned canine weight or height datasets and one pre-cleaned human weight dataset with randomly simulated errors. Prior to the addition of the algorithm, data cleaning based on non-linear mixed effects models was the most effective in all datasets and had on average a minimum of 26.00% higher sensitivity and 0.12% higher specificity than other methods. Data cleaning methods using the algorithm had improved data preservation and were capable of correcting simulated errors according to the gold standard; returning a value to its original state prior to error simulation. The algorithm improved the performance of all data cleaning methods and increased the average sensitivity and specificity of the non-linear mixed effects model method by 7.68% and 0.42% respectively. Using non-linear mixed effects models combined with the algorithm to clean data allows individual growth trajectories to vary from the population by using repeated longitudinal measurements, identifies consecutive errors or those within the first data entry, avoids the requirement for a minimum number of data entries, preserves data where possible by correcting errors rather than deleting them and removes duplications intelligently. This algorithm is broadly applicable to data cleaning anthropometric data in different mammalian species and could be adapted for use in a range of other domains.

Netflix is a popular streaming service that offers a vast catalog of movies, TV shows, and original contents. This dataset is a cleaned version of the original version which can be found here. The data consist of contents added to Netflix from 2008 to 2021. The oldest content is as old as 1925 and the newest as 2021. This dataset will be cleaned with PostgreSQL and visualized with Tableau. The purpose of this dataset is to test my data cleaning and visualization skills. The cleaned data can be found below and the Tableau dashboard can be found here .

Data Cleaning

We are going to: 1. Treat the Nulls 2. Treat the duplicates 3. Populate missing rows 4. Drop unneeded columns 5. Split columns Extra steps and more explanation on the process will be explained through the code comments

--View dataset

SELECT * 
FROM netflix;

--The show_id column is the unique id for the dataset, therefore we are going to check for duplicates
                                  
SELECT show_id, COUNT(*)                                                                                      
FROM netflix 
GROUP BY show_id                                                                                              
ORDER BY show_id DESC;

--No duplicates

--Check null values across columns

SELECT COUNT(*) FILTER (WHERE show_id IS NULL) AS showid_nulls,
    COUNT(*) FILTER (WHERE type IS NULL) AS type_nulls,
    COUNT(*) FILTER (WHERE title IS NULL) AS title_nulls,
    COUNT(*) FILTER (WHERE director IS NULL) AS director_nulls,
    COUNT(*) FILTER (WHERE movie_cast IS NULL) AS movie_cast_nulls,
    COUNT(*) FILTER (WHERE country IS NULL) AS country_nulls,
    COUNT(*) FILTER (WHERE date_added IS NULL) AS date_addes_nulls,
    COUNT(*) FILTER (WHERE release_year IS NULL) AS release_year_nulls,
    COUNT(*) FILTER (WHERE rating IS NULL) AS rating_nulls,
    COUNT(*) FILTER (WHERE duration IS NULL) AS duration_nulls,
    COUNT(*) FILTER (WHERE listed_in IS NULL) AS listed_in_nulls,
    COUNT(*) FILTER (WHERE description IS NULL) AS description_nulls
FROM netflix;

We can see that there are NULLS. 
director_nulls = 2634
movie_cast_nulls = 825
country_nulls = 831
date_added_nulls = 10
rating_nulls = 4
duration_nulls = 3 

The director column nulls is about 30% of the whole column, therefore I will not delete them. I will rather find another column to populate it. To populate the director column, we want to find out if there is relationship between movie_cast column and director column

-- Below, we find out if some directors are likely to work with particular cast

WITH cte AS
(
SELECT title, CONCAT(director, '---', movie_cast) AS director_cast 
FROM netflix
)

SELECT director_cast, COUNT(*) AS count
FROM cte
GROUP BY director_cast
HAVING COUNT(*) > 1
ORDER BY COUNT(*) DESC;

With this, we can now populate NULL rows in directors 
using their record with movie_cast 

UPDATE netflix 
SET director = 'Alastair Fothergill'
WHERE movie_cast = 'David Attenborough'
AND director IS NULL ;

--Repeat this step to populate the rest of the director nulls
--Populate the rest of the NULL in director as "Not Given"

UPDATE netflix 
SET director = 'Not Given'
WHERE director IS NULL;

--When I was doing this, I found a less complex and faster way to populate a column which I will use next

Just like the director column, I will not delete the nulls in country. Since the country column is related to director and movie, we are going to populate the country column with the director column

--Populate the country using the director column

SELECT COALESCE(nt.country,nt2.country) 
FROM netflix AS nt
JOIN netflix AS nt2 
ON nt.director = nt2.director 
AND nt.show_id <> nt2.show_id
WHERE nt.country IS NULL;
UPDATE netflix
SET country = nt2.country
FROM netflix AS nt2
WHERE netflix.director = nt2.director and netflix.show_id <> nt2.show_id 
AND netflix.country IS NULL;


--To confirm if there are still directors linked to country that refuse to update

SELECT director, country, date_added
FROM netflix
WHERE country IS NULL;

--Populate the rest of the NULL in director as "Not Given"

UPDATE netflix 
SET country = 'Not Given'
WHERE country IS NULL;

The date_added rows nulls is just 10 out of over 8000 rows, deleting them cannot affect our analysis or visualization

--Show date_added nulls

SELECT show_id, date_added
FROM netflix_clean
WHERE date_added IS NULL;

--DELETE nulls

DELETE F...

Dirty Retail Store Sales Dataset

Overview

The Dirty Retail Store Sales dataset contains 12,575 rows of synthetic data representing sales transactions from a retail store. The dataset includes eight product categories with 25 items per category, each having static prices. It is designed to simulate real-world sales data, including intentional "dirtiness" such as missing or inconsistent values. This dataset is suitable for practicing data cleaning, exploratory data analysis (EDA), and feature engineering.

File Information

• File Name: retail_store_sales.csv
• Number of Rows: 12,575
• Number of Columns: 11

Columns Description

Categories and Items

The dataset includes the following categories, each containing 25 items with corresponding codes, names, and static prices:

Electric Household Essentials

Furniture

A data cleaning tool customised for cleaning and sorting the data generated during the Enviro-Champs pilot study as they are downloaded from Formshare, the platform capturing data sent from a customised ODK Collect form collection app. The dataset inclues the latest data from the pilot study as at 14 May 2024.

Project Overview : This project demonstrates a thorough data cleaning process for the Nashville Housing dataset using SQL. The script performs various data cleaning and transformation operations to improve the quality and usability of the data for further analysis.

Technologies Used : SQL Server T-SQL

Dataset: The project uses the Nashville Housing dataset, which contains information about property sales in Nashville, Tennessee. The original dataset includes various fields such as property addresses, sale dates, sale prices, and other relevant real estate information. Data Cleaning Operations The script performs the following data cleaning operations:

Date Standardization: Converts the SaleDate column to a standard Date format for consistency and easier manipulation. Populating Missing Property Addresses: Fills in NULL values in the PropertyAddress field using data from other records with the same ParcelID. Breaking Down Address Components: Separates the PropertyAddress and OwnerAddress fields into individual columns for Address, City, and State, improving data granularity and queryability. Standardizing Values: Converts 'Y' and 'N' values to 'Yes' and 'No' in the SoldAsVacant field for clarity and consistency. Removing Duplicates: Identifies and removes duplicate records based on specific criteria to ensure data integrity. Dropping Unused Columns: Removes unnecessary columns to streamline the dataset.

Key SQL Techniques Demonstrated :

Data type conversion Self joins for data population String manipulation (SUBSTRING, CHARINDEX, PARSENAME) CASE statements Window functions (ROW_NUMBER) Common Table Expressions (CTEs) Data deletion Table alterations (adding and dropping columns)

Important Notes :

The script includes cautionary comments about data deletion and column dropping, emphasizing the importance of careful consideration in a production environment. This project showcases various SQL data cleaning techniques and can serve as a template for similar data cleaning tasks.

Potential Improvements :

Implement error handling and transaction management for more robust execution. Add data validation steps to ensure the cleaned data meets specific criteria. Consider creating indexes on frequently queried columns for performance optimization.

*The data for this dataset is updated daily. The date(s) displayed in the details section on our Open Data Portal is based on the last date the metadata was updated and not the refresh date of the data itself.*The Cleanup Sites layer provides locations and document links for sites currently in the cleanup process and sites awaiting cleanup funding. Cleanup programs include: Brownfields, Petroleum, EPA Superfund (CERCLA), Drycleaning, Responsible Party Cleanup, State Funded Cleanup, State Owned Lands Cleanup and Hazardous Waste Cleanup.Please reference the metadata for contact information.

Autonomous Data Cleaning with AI Market Outlook

According to our latest research, the global Autonomous Data Cleaning with AI market size reached USD 1.68 billion in 2024, with a robust year-on-year growth driven by the surge in enterprise data volumes and the mounting demand for high-quality, actionable insights. The market is projected to expand at a CAGR of 24.2% from 2025 to 2033, which will take the overall market value to approximately USD 13.1 billion by 2033. This rapid growth is fueled by the increasing adoption of artificial intelligence (AI) and machine learning (ML) technologies across industries, aiming to automate and optimize the data cleaning process for improved operational efficiency and decision-making.

The primary growth driver for the Autonomous Data Cleaning with AI market is the exponential increase in data generation across various industries such as BFSI, healthcare, retail, and manufacturing. Organizations are grappling with massive amounts of structured and unstructured data, much of which is riddled with inconsistencies, duplicates, and inaccuracies. Manual data cleaning is both time-consuming and error-prone, leading businesses to seek automated AI-driven solutions that can intelligently detect, correct, and prevent data quality issues. The integration of AI not only accelerates the data cleaning process but also ensures higher accuracy, enabling organizations to leverage clean, reliable data for analytics, compliance, and digital transformation initiatives. This, in turn, translates into enhanced business agility and competitive advantage.

Another significant factor propelling the market is the increasing regulatory scrutiny and compliance requirements in sectors such as banking, healthcare, and government. Regulations such as GDPR, HIPAA, and others mandate strict data governance and quality standards. Autonomous Data Cleaning with AI solutions help organizations maintain compliance by ensuring data integrity, traceability, and auditability. Additionally, the evolution of cloud computing and the proliferation of big data analytics platforms have made it easier for organizations of all sizes to deploy and scale AI-powered data cleaning tools. These advancements are making autonomous data cleaning more accessible, cost-effective, and scalable, further driving market adoption.

The growing emphasis on digital transformation and real-time decision-making is also a crucial growth factor for the Autonomous Data Cleaning with AI market. As enterprises increasingly rely on analytics, machine learning, and artificial intelligence for business insights, the quality of input data becomes paramount. Automated, AI-driven data cleaning solutions enable organizations to process, cleanse, and prepare data in real-time, ensuring that downstream analytics and AI models are fed with high-quality inputs. This not only improves the accuracy of business predictions but also reduces the time-to-insight, helping organizations stay ahead in highly competitive markets.

From a regional perspective, North America currently dominates the Autonomous Data Cleaning with AI market, accounting for the largest share in 2024, followed closely by Europe and Asia Pacific. The presence of leading technology companies, early adopters of AI, and a mature regulatory environment are key factors contributing to North America’s leadership. However, Asia Pacific is expected to witness the highest CAGR over the forecast period, driven by rapid digitalization, expanding IT infrastructure, and increasing investments in AI and data analytics, particularly in countries such as China, India, and Japan. Latin America and the Middle East & Africa are also gradually emerging as promising markets, supported by growing awareness and adoption of AI-driven data management solutions.

Component Analysis

The Autonomous Data Cleaning with AI market is segmented by component into Software and Services. The software segment currently holds the largest market share, driven

AI in Data Cleaning Market Outlook

According to our latest research, the global AI in Data Cleaning market size reached USD 1.82 billion in 2024, demonstrating remarkable momentum driven by the exponential growth of data-driven enterprises. The market is projected to grow at a CAGR of 28.1% from 2025 to 2033, reaching an estimated USD 17.73 billion by 2033. This exceptional growth trajectory is primarily fueled by increasing data volumes, the urgent need for high-quality datasets, and the adoption of artificial intelligence technologies across diverse industries.

The surging demand for automated data management solutions remains a key growth driver for the AI in Data Cleaning market. As organizations generate and collect massive volumes of structured and unstructured data, manual data cleaning processes have become insufficient, error-prone, and costly. AI-powered data cleaning tools address these challenges by leveraging machine learning algorithms, natural language processing, and pattern recognition to efficiently identify, correct, and eliminate inconsistencies, duplicates, and inaccuracies. This automation not only enhances data quality but also significantly reduces operational costs and improves decision-making capabilities, making AI-based solutions indispensable for enterprises aiming to achieve digital transformation and maintain a competitive edge.

Another crucial factor propelling market expansion is the growing emphasis on regulatory compliance and data governance. Sectors such as BFSI, healthcare, and government are subject to stringent data privacy and accuracy regulations, including GDPR, HIPAA, and CCPA. AI in data cleaning enables these industries to ensure data integrity, minimize compliance risks, and maintain audit trails, thereby safeguarding sensitive information and building stakeholder trust. Furthermore, the proliferation of cloud computing and advanced analytics platforms has made AI-powered data cleaning solutions more accessible, scalable, and cost-effective, further accelerating adoption across small, medium, and large enterprises.

The increasing integration of AI in data cleaning with other emerging technologies such as big data analytics, IoT, and robotic process automation (RPA) is unlocking new avenues for market growth. By embedding AI-driven data cleaning processes into end-to-end data pipelines, organizations can streamline data preparation, enable real-time analytics, and support advanced use cases like predictive modeling and personalized customer experiences. Strategic partnerships, investments in R&D, and the rise of specialized AI startups are also catalyzing innovation in this space, making AI in data cleaning a cornerstone of the broader data management ecosystem.

From a regional perspective, North America continues to lead the global AI in Data Cleaning market, accounting for the largest revenue share in 2024, followed closely by Europe and Asia Pacific. The region’s dominance is attributed to the presence of major technology vendors, robust digital infrastructure, and high adoption rates of AI and cloud technologies. Meanwhile, Asia Pacific is witnessing the fastest growth, propelled by rapid digitalization, expanding IT sectors, and increasing investments in AI-driven solutions by enterprises in China, India, and Southeast Asia. Europe remains a significant market, supported by strict data protection regulations and a mature enterprise landscape. Latin America and the Middle East & Africa are emerging as promising markets, albeit at a relatively nascent stage, with growing awareness and gradual adoption of AI-powered data cleaning solutions.

Component Analysis

The AI in Data Cleaning market is broadly segmented by component into software and services, with each segment playing a pivotal role in shaping the industry’s evolution. The software segment dominates the market, driven by the rapid adoption of advanced AI-based data cleaning platforms that automate complex data preparation tasks. These platforms leverage sophisticated algorithms to detect anomalies, standardize formats, and enrich datasets, thereby enabling organizations to maintain high-quality data repositories. The increasing demand for self-service data cleaning software, which empowers business users to cleanse data without extensive IT intervention, is further fueling growth in this segment. Vendors are continuously enhancing their offerings with intuitive interfaces, integration capabilities, and support for diverse data sources to cater to a wide r

Overview and Contents This replication package was assembled in January of 2025. The code in this repository generates the 13 figures and content of the 3 tables for the paper “All Forecasters Are Not the Same: Systematic Patterns in Predictive Performance”. It also generates the 2 figures and content of the 5 tables in the appendix to this paper. The main contents of the repository are the following: Code/: folder of scripts to prepare and clean data as well as generate tables and figures. Functions/: folder of subroutines for use with MATLAB scripts. Data/: data folder. Raw/: ECB SPF forecast data, realizations of target variables, and start and end bins for density forecasts. Intermediate/: Data used at intermediate steps in the cleaning process. These datasets are generated with x01_Raw_Data_Shell.do, x02a_Individual_Uncertainty_GDP.do, x02b_Individual_Uncertainty_HICP.do, x02c_Individual_Uncertainty_Urate.do, x03_Pull_Data.do, x04_Data_Clean_And_Merge, and x05_Drop_Low_Counts.do in the Code/ folder. Ready/: Data used to conduct regressions, statistical tests, and generate figures. Output/: folder of results. Figures/: .jpg files for each figure used in the paper and its appendix. HL Results/: Results from applying the Hounyo and Lahiri (2023) testing procedure for equal predictive performance to ECB SPF forecast data. This folder contains the material for Tables 1A-4A. Regressions/: Regression results, as well as material for Tables 3 and 5A. Simulations/: Results from simulation exercise as well as the datasets used to create Figures 9-12. Statistical Tests/: Results displayed in Tables 1 and 2. The repository also contains the manuscript, appendix, and this read-me file.DisclaimerThis replication package was produced by the authors and is not an official product of the Federal Reserve Bank of Cleveland. The analysis and conclusions set forth are those of the authors and do not indicate concurrence by the Federal Reserve Bank of Cleveland or the Federal Reserve System.

A collection of datasets and python scripts for extraction and analysis of isograms (and some palindromes and tautonyms) from corpus-based word-lists, specifically Google Ngram and the British National Corpus (BNC).Below follows a brief description, first, of the included datasets and, second, of the included scripts.1. DatasetsThe data from English Google Ngrams and the BNC is available in two formats: as a plain text CSV file and as a SQLite3 database.1.1 CSV formatThe CSV files for each dataset actually come in two parts: one labelled ".csv" and one ".totals". The ".csv" contains the actual extracted data, and the ".totals" file contains some basic summary statistics about the ".csv" dataset with the same name.The CSV files contain one row per data point, with the colums separated by a single tab stop. There are no labels at the top of the files. Each line has the following columns, in this order (the labels below are what I use in the database, which has an identical structure, see section below):

Label Data type Description

isogramy int The order of isogramy, e.g. "2" is a second order isogram

length int The length of the word in letters

word text The actual word/isogram in ASCII

source_pos text The Part of Speech tag from the original corpus

count int Token count (total number of occurences)

vol_count int Volume count (number of different sources which contain the word)

count_per_million int Token count per million words

vol_count_as_percent int Volume count as percentage of the total number of volumes

is_palindrome bool Whether the word is a palindrome (1) or not (0)

is_tautonym bool Whether the word is a tautonym (1) or not (0)

The ".totals" files have a slightly different format, with one row per data point, where the first column is the label and the second column is the associated value. The ".totals" files contain the following data:

Label

Data type

Description

!total_1grams

int

The total number of words in the corpus

!total_volumes

int

The total number of volumes (individual sources) in the corpus

!total_isograms

int

The total number of isograms found in the corpus (before compacting)

!total_palindromes

int

How many of the isograms found are palindromes

!total_tautonyms

int

How many of the isograms found are tautonyms

The CSV files are mainly useful for further automated data processing. For working with the data set directly (e.g. to do statistics or cross-check entries), I would recommend using the database format described below.1.2 SQLite database formatOn the other hand, the SQLite database combines the data from all four of the plain text files, and adds various useful combinations of the two datasets, namely:• Compacted versions of each dataset, where identical headwords are combined into a single entry.• A combined compacted dataset, combining and compacting the data from both Ngrams and the BNC.• An intersected dataset, which contains only those words which are found in both the Ngrams and the BNC dataset.The intersected dataset is by far the least noisy, but is missing some real isograms, too.The columns/layout of each of the tables in the database is identical to that described for the CSV/.totals files above.To get an idea of the various ways the database can be queried for various bits of data see the R script described below, which computes statistics based on the SQLite database.2. ScriptsThere are three scripts: one for tiding Ngram and BNC word lists and extracting isograms, one to create a neat SQLite database from the output, and one to compute some basic statistics from the data. The first script can be run using Python 3, the second script can be run using SQLite 3 from the command line, and the third script can be run in R/RStudio (R version 3).2.1 Source dataThe scripts were written to work with word lists from Google Ngram and the BNC, which can be obtained from http://storage.googleapis.com/books/ngrams/books/datasetsv2.html and [https://www.kilgarriff.co.uk/bnc-readme.html], (download all.al.gz).For Ngram the script expects the path to the directory containing the various files, for BNC the direct path to the *.gz file.2.2 Data preparationBefore processing proper, the word lists need to be tidied to exclude superfluous material and some of the most obvious noise. This will also bring them into a uniform format.Tidying and reformatting can be done by running one of the following commands:python isograms.py --ngrams --indir=INDIR --outfile=OUTFILEpython isograms.py --bnc --indir=INFILE --outfile=OUTFILEReplace INDIR/INFILE with the input directory or filename and OUTFILE with the filename for the tidied and reformatted output.2.3 Isogram ExtractionAfter preparing the data as above, isograms can be extracted from by running the following command on the reformatted and tidied files:python isograms.py --batch --infile=INFILE --outfile=OUTFILEHere INFILE should refer the the output from the previosu data cleaning process. Please note that the script will actually write two output files, one named OUTFILE with a word list of all the isograms and their associated frequency data, and one named "OUTFILE.totals" with very basic summary statistics.2.4 Creating a SQLite3 databaseThe output data from the above step can be easily collated into a SQLite3 database which allows for easy querying of the data directly for specific properties. The database can be created by following these steps:1. Make sure the files with the Ngrams and BNC data are named “ngrams-isograms.csv” and “bnc-isograms.csv” respectively. (The script assumes you have both of them, if you only want to load one, just create an empty file for the other one).2. Copy the “create-database.sql” script into the same directory as the two data files.3. On the command line, go to the directory where the files and the SQL script are. 4. Type: sqlite3 isograms.db 5. This will create a database called “isograms.db”.See the section 1 for a basic descript of the output data and how to work with the database.2.5 Statistical processingThe repository includes an R script (R version 3) named “statistics.r” that computes a number of statistics about the distribution of isograms by length, frequency, contextual diversity, etc. This can be used as a starting point for running your own stats. It uses RSQLite to access the SQLite database version of the data described above.

This Feature Layer Collection contains publicly shared data from Envirostor, the Department of Toxic Substances Control's Project Management Solution. The data includes Cleanup Sites, Hazardous Waste Sites, and Inspection, Compliance and Enforcement sites. Cleanup Sites: DTSC conducts and supervises investigation and cleanup actions at sites where oil or hazardous chemicals have been or may be released into the environment. Cleanup activities take place at active and abandoned waste sites, federal and state facilities and properties, and where any storage tanks have leaked. DTSC, federal and other state agencies or municipalities, or the company or party responsible for the contamination may perform cleanups. Cleanup can also include site reuse and redevelopment. Hazardous Waste Sites: Hazardous waste management facilities receive hazardous wastes for treatment, storage or disposal. These facilities are often referred to as treatment, storage and disposal facilities, or TSDFs, and their activities are described in more detail below: Treatment - Using various processes, such as incineration or oxidation, to alter the character or composition of hazardous wastes. Some treatment processes enable waste to be recovered and reused in manufacturing settings, while other treatment processes dramatically reduce the amount of hazardous waste. Storage - Temporarily holding hazardous wastes until they are treated or disposed. Hazardous waste is commonly stored prior to treatment or disposal, and must be stored in containers, tanks, containment buildings, drip pads, waste piles, or surface impoundments that comply with the Resource Conservation and Recovery Act (RCRA) regulations. Disposal - Permanently containing hazardous wastes. The most common type of disposal facility is a landfill, where hazardous wastes are disposed of in carefully constructed units designed to protect groundwater and surface water resources. ICE Sites: The Department of Toxic Substances Control (DTSC) regulates the generation, transportation, treatment, storage, and disposal of hazardous wastes. DTSC monitors compliance with state and federal hazardous waste requirements by conducting inspections. DTSC works to ensure compliance with environmental requirements. When warranted, DTSC will take civil or criminal enforcement action against violators of environmental laws. DTSC provides compliance incentives and auditing to encourage facilities to find and disclose violations to the Agency. Violations may also be discovered from tips/complaints received by the Agency from the public. Violations discovered as a result of any of these activities may lead to civil or criminal enforcement. This data is a geospatial representation of data found at https://www.envirostor.dtsc.ca.gov/public/. This dataset is updated daily.

THIS DATA ASSET NO LONGER ACTIVE: This is metadata documentation for the National Priorities List (NPL) Publication Assistance Databsae (PAD), a Lotus Notes application that holds Region 7's universe of NPL site information such as site description, threats and contaminants, cleanup approach, environmental process, community involvement, site repository, and regional contacts. This database used to be updated annually, at different times for different NPLs, but it is currently no longer being used. This work fell under objectives for EPA's 2003-2008 Strategic Plan (Goal 3) for Land Preservation & Restoration, which are to clean up and reuse contaminated land.

The summary from the detailed analysis of the case study in EPA (1988b) is provided in Table 3 of the manuscript, and was used as the data source for the two datasets used in this study. These include a flat and hierarchical structure of the five balancing criteria, shown in Table 4 and Table 5, respectively. Table 4 provides a comprehensive score for each balancing criterion, similar to the summary tables presented in the FS of Superfund sites (e.g., (EPA 2016b, AECOM 2019)). Table 5 uses the same information in Table 3, but in this case, each piece of information is used to define multiple sub-criteria for each balancing criterion, except the cost one. This leads to a much more elaborate information table with the four remaining balancing criteria, now characterized by 13 sub-criteria. It is important to note that the scoring provided in Table 4 and Table 5, with the exception of the cost (c_5), were derived from the author’s interpretation of the descriptive language of the detailed analysis in for the hypothetical case study in presented in Table A-7 in Appendix A of the guidance document of EPA (1988b). It should be noted that the analysis of the three remedy alternatives presented in this hypothetical case study is governed by site-specific characteristics and may not represent potential performance of these remediation alternatives for other sites . The intent of this exercise is to illustrate the flexibility and adaptability of the MCDA process to address both the main, overarching criteria, as well as sub-criteria that may have specific importance in the decision process for a particular site. Ultimately, the sub-criteria can be adapted to address specific stakeholder perspectives or technical factors that may be linked to properties unique to the contaminant or physical characteristics of the site. This dataset is associated with the following publication: Cinelli, M., M.A. Gonzalez, R. Ford, J. McKernan, S. Corrente, M. Kadziński, and R. Słowiński. Supporting contaminated sites management with Multiple Criteria Decision Analysis: Demonstration of a regulation-consistent approach. JOURNAL OF CLEANER PRODUCTION. Elsevier Science Ltd, New York, NY, USA, 316: 128347, (2021).

This folder "XPS data" contains original and evaluated XPS data (.vms) on a p-GaN sample which was treated at various temperatures and underwent Ar+ irradiation.

Furthermore, the folder "REM Images" contains REM images (.tif) and EDX data (.xlsx) on the used excessively treated sample.

All images that are published in the main manuscript are collected as .tif files in the folder "images".

Quadrant provides Insightful, accurate, and reliable mobile location data.

Our privacy-first mobile location data unveils hidden patterns and opportunities, provides actionable insights, and fuels data-driven decision-making at the world's biggest companies.

These companies rely on our privacy-first Mobile Location and Points-of-Interest Data to unveil hidden patterns and opportunities, provide actionable insights, and fuel data-driven decision-making. They build better AI models, uncover business insights, and enable location-based services using our robust and reliable real-world data.

We conduct stringent evaluations on data providers to ensure authenticity and quality. Our proprietary algorithms detect, and cleanse corrupted and duplicated data points – allowing you to leverage our datasets rapidly with minimal processing or cleaning. During the ingestion process, our proprietary Data Filtering Algorithms remove events based on a number of both qualitative factors, as well as latency and other integrity variables to provide more efficient data delivery. The deduplicating algorithm focuses on a combination of four important attributes: Device ID, Latitude, Longitude, and Timestamp. This algorithm scours our data and identifies rows that contain the same combination of these four attributes. Post-identification, it retains a single copy and eliminates duplicate values to ensure our customers only receive complete and unique datasets.

We actively identify overlapping values at the provider level to determine the value each offers. Our data science team has developed a sophisticated overlap analysis model that helps us maintain a high-quality data feed by qualifying providers based on unique data values rather than volumes alone – measures that provide significant benefit to our end-use partners.

Quadrant mobility data contains all standard attributes such as Device ID, Latitude, Longitude, Timestamp, Horizontal Accuracy, and IP Address, and non-standard attributes such as Geohash and H3. In addition, we have historical data available back through 2022.

Through our in-house data science team, we offer sophisticated technical documentation, location data algorithms, and queries that help data buyers get a head start on their analyses. Our goal is to provide you with data that is “fit for purpose”.

The method involves sucking gas mixture from a crude oil tank (10), and supplying inert gas into the tank. The energy content of the sucked gas mixture is determined. The gas mixture is supplied into a compressor (38) for compressing the gas mixture, when the energy content of the gas mixture exceeds a limitation value. The gas mixture is supplied into a load unit (40), when the energy content of the gas mixture is below the limitation value. Independent claims are also included for the following: (1) a device for suction of a gas mixture from a tank (2) a cleaning vehicle for cleaning a tank.

Directory Cleanup Tools Market Outlook

According to our latest research, the directory cleanup tools market size reached USD 1.48 billion in 2024, demonstrating robust demand across global enterprises. The market is projected to expand at a CAGR of 12.1% from 2025 to 2033, reaching an estimated USD 4.13 billion by 2033. This growth is primarily driven by the increasing need for efficient data management, regulatory compliance, and cybersecurity among organizations of all sizes. The adoption of digital transformation initiatives and the proliferation of data-intensive applications are fueling the deployment of advanced directory cleanup solutions across various industries.

One of the primary growth factors for the directory cleanup tools market is the exponential rise in organizational data volumes due to digital transformation and cloud adoption. As businesses migrate to hybrid and cloud environments, directory structures become more complex, leading to redundant, obsolete, or incorrect records. This data sprawl not only impacts operational efficiency but also increases the risk of security breaches and compliance violations. Directory cleanup tools play a crucial role in automating the identification and removal of such records, ensuring data hygiene and streamlined access control. The surge in remote work and the resultant expansion of digital identities further accentuate the need for robust directory management solutions, thereby propelling market growth.

Another significant driver is the tightening regulatory landscape, compelling organizations to maintain accurate and up-to-date directory information. Regulations such as GDPR, HIPAA, and CCPA mandate strict data governance, including the timely deletion of unnecessary or outdated records. Directory cleanup tools help enterprises adhere to these requirements by offering automated workflows, detailed audit trails, and policy-based management. The increasing frequency of audits and the high costs associated with non-compliance are pushing organizations to adopt these solutions as a preventive measure. Moreover, the integration of directory cleanup tools with identity and access management (IAM) platforms is enabling comprehensive data governance, further fueling adoption in highly regulated sectors such as BFSI, healthcare, and government.

Technological advancements are also shaping the growth trajectory of the directory cleanup tools market. The incorporation of artificial intelligence (AI) and machine learning (ML) capabilities in these tools allows for intelligent pattern recognition, anomaly detection, and predictive analytics. These features enhance the effectiveness of directory cleanup processes by reducing manual intervention and minimizing errors. Furthermore, the emergence of cloud-based directory cleanup solutions is lowering the entry barrier for small and medium enterprises (SMEs), enabling them to leverage sophisticated data management tools without significant upfront investments. The growing emphasis on cybersecurity, coupled with the need for operational efficiency, is expected to sustain the demand for directory cleanup tools throughout the forecast period.

From a regional perspective, North America continues to dominate the directory cleanup tools market, accounting for the largest revenue share in 2024. The region's leadership can be attributed to the early adoption of advanced IT infrastructure, stringent regulatory requirements, and the presence of leading technology vendors. Europe follows closely, driven by strong data protection laws and a high concentration of data-driven enterprises. The Asia Pacific region is emerging as a high-growth market, fueled by rapid digitalization, increasing investments in cloud technologies, and the proliferation of SMEs. Latin America and the Middle East & Africa are also witnessing steady growth, supported by ongoing digital transformation initiatives and rising awareness of data management best practices.

In the realm of directory management, File Version Cleanup Tools are becoming increasingly important. As organizations accumulate vast amounts of data, managing file versions efficiently is crucial to maintaining data integrity and reducing storage costs. These tools help automate the process of identifying and removing outdated or redundant file versions, which can otherwise clutter