Data cleaning / Files for practical learning

Randomly generated data

Data source

The data is randomly generated and not from an external source. The data do not represent parameters.

The data in the files have been stored in a structure that allows the use of basic tools created for data cleaning, e.g. 'fillna', 'dropna' functions from the pandas module. Additional information can be found in the file descriptions.

License

CC0: Public Domain

Project Overview : This project demonstrates a thorough data cleaning process for the Nashville Housing dataset using SQL. The script performs various data cleaning and transformation operations to improve the quality and usability of the data for further analysis.

Technologies Used : SQL Server T-SQL

Dataset: The project uses the Nashville Housing dataset, which contains information about property sales in Nashville, Tennessee. The original dataset includes various fields such as property addresses, sale dates, sale prices, and other relevant real estate information. Data Cleaning Operations The script performs the following data cleaning operations:

Date Standardization: Converts the SaleDate column to a standard Date format for consistency and easier manipulation. Populating Missing Property Addresses: Fills in NULL values in the PropertyAddress field using data from other records with the same ParcelID. Breaking Down Address Components: Separates the PropertyAddress and OwnerAddress fields into individual columns for Address, City, and State, improving data granularity and queryability. Standardizing Values: Converts 'Y' and 'N' values to 'Yes' and 'No' in the SoldAsVacant field for clarity and consistency. Removing Duplicates: Identifies and removes duplicate records based on specific criteria to ensure data integrity. Dropping Unused Columns: Removes unnecessary columns to streamline the dataset.

Key SQL Techniques Demonstrated :

Data type conversion Self joins for data population String manipulation (SUBSTRING, CHARINDEX, PARSENAME) CASE statements Window functions (ROW_NUMBER) Common Table Expressions (CTEs) Data deletion Table alterations (adding and dropping columns)

Important Notes :

The script includes cautionary comments about data deletion and column dropping, emphasizing the importance of careful consideration in a production environment. This project showcases various SQL data cleaning techniques and can serve as a template for similar data cleaning tasks.

Potential Improvements :

Implement error handling and transaction management for more robust execution. Add data validation steps to ensure the cleaned data meets specific criteria. Consider creating indexes on frequently queried columns for performance optimization.

python scripts and functions needed to view and clean saccade data

How To Cite?

Alinaghi, N., Giannopoulos, I., Kattenbeck, M., & Raubal, M. (2025). Decoding wayfinding: analyzing wayfinding processes in the outdoor environment. International Journal of Geographical Information Science, 1–31. https://doi.org/10.1080/13658816.2025.2473599

Link to the paper: https://www.tandfonline.com/doi/full/10.1080/13658816.2025.2473599

Folder Structure

The folder named “submission” contains the following:

1. “pythonProject”: This folder contains all the Python files and subfolders needed for analysis.
2. ijgis.yml: This file lists all the Python libraries and dependencies required to run the code.

Setting Up the Environment

1. Use the ijgis.yml file to create a Python project and environment. Ensure you activate the environment before running the code.
2. The pythonProject folder contains several .py files and subfolders, each with specific functionality as described below.

Subfolders

1. Data_4_IJGIS

• This folder contains the data used for the results reported in the paper.
• Note: The data analysis that we explain in this paper already begins with the synchronization and cleaning of the recorded raw data. The published data is already synchronized and cleaned. Both the cleaned files and the merged files with features extracted for them are given in this directory. If you want to perform the segmentation and feature extraction yourself, you should run the respective Python files yourself. If not, you can use the “merged_…csv” files as input for the training.

2. results_[DateTime] (e.g., results_20240906_15_00_13)

• This folder will be generated when you run the code and will store the output of each step.
• The current folder contains results created during code debugging for the submission.
• When you run the code, a new folder with fresh results will be generated.

Python Files

1. helper_functions.py

• Contains reusable functions used throughout the analysis.
• Each function includes a description of its purpose and the input parameters required.

2. create_sanity_plots.py

• Generates scatter plots like those in Figure 3 of the paper.
• Although the code has been run for all 309 trials, it can be used to check the sample data provided.
• Output: A .png file for each column of the raw gaze and IMU recordings, color-coded with logged events.
• Usage: Run this file to create visualizations similar to Figure 3.

3. overlapping_sliding_window_loop.py

• Implements overlapping sliding window segmentation and generates plots like those in Figure 4.
• Output:
  • Two new subfolders, “Gaze” and “IMU”, will be added to the Data_4_IJGIS folder.
  • Segmented files (default: 2–10 seconds with a 1-second step size) will be saved as .csv files.
  • A visualization of the segments, similar to Figure 4, will be automatically generated.

4. gaze_features.py & imu_features.py (Note: there has been an update to the IDT function implementation in the gaze_features.py on 19.03.2025.)

• These files compute features as explained in Tables 1 and 2 of the paper, respectively.
• They process the segmented recordings generated by the overlapping_sliding_window_loop.py.
• Usage: Just to know how the features are calculated, you can run this code after the segmentation with the sliding window and run these files to calculate the features from the segmented data.

5. training_prediction.py

• This file contains the main machine learning analysis of the paper. This file contains all the code for the training of the model, its evaluation, and its use for the inference of the “monitoring part”. It covers the following steps:

a. Data Preparation (corresponding to Section 5.1.1 of the paper)

• Prepares the data according to the research question (RQ) described in the paper. Since this data was collected with several RQs in mind, we remove parts of the data that are not related to the RQ of this paper.
• A function named plot_labels_comparison(df, save_path, x_label_freq=10, figsize=(15, 5)) in line 116 visualizes the data preparation results. As this visualization is not used in the paper, the line is commented out, but if you want to see visually what has been changed compared to the original data, you can comment out this line.

b. Training/Validation/Test Split

• Splits the data for machine learning experiments (an explanation can be found in Section 5.1.1. Preparation of data for training and inference of the paper).
• Make sure that you follow the instructions in the comments to the code exactly.
• Output: The split data is saved as .csv files in the results folder.

c. Machine and Deep Learning Experiments

This part contains three main code blocks:

iii. One for the XGboost code with correct hyperparameter tuning:
Please read the instructions for each block carefully to ensure that the code works smoothly. Regardless of which block you use, you will get the classification results (in the form of scores) for unseen data. The way we empirically test the confidence threshold of

• MLP Network (Commented Out): This code was used for classification with the MLP network, and the results shown in Table 3 are from this code. If you wish to use this model, please comment out the following blocks accordingly.
• XGBoost without Hyperparameter Tuning: If you want to run the code but do not want to spend time on the full training with hyperparameter tuning (as was done for the paper), just uncomment this part. This will give you a simple, untuned model with which you can achieve at least some results.
• XGBoost with Hyperparameter Tuning: If you want to train the model the way we trained it for the analysis reported in the paper, use this block (the plots in Figure 7 are from this block). We ran this block with different feature sets and different segmentation files and created a simple bar chart from the saved results, shown in Figure 6.

Note: Please read the instructions for each block carefully to ensure that the code works smoothly. Regardless of which block you use, you will get the classification results (in the form of scores) for unseen data. The way we empirically calculated the confidence threshold of the model (explained in the paper in Section 5.2. Part II: Decoding surveillance by sequence analysis) is given in this block in lines 361 to 380.

d. Inference (Monitoring Part)

• Final inference is performed using the monitoring data. This step produces a .csv file containing inferred labels.
• Figure 8 in the paper is generated using this part of the code.

6. sequence_analysis.py

• Performs analysis on the inferred data, producing Figures 9 and 10 from the paper.
• This file reads the inferred data from the previous step and performs sequence analysis as described in Sections 5.2.1 and 5.2.2.

Licenses

The data is licensed under CC-BY, the code is licensed under MIT.

Overview and Contents This replication package was assembled in January of 2025. The code in this repository generates the 13 figures and content of the 3 tables for the paper “All Forecasters Are Not the Same: Systematic Patterns in Predictive Performance”. It also generates the 2 figures and content of the 5 tables in the appendix to this paper. The main contents of the repository are the following: Code/: folder of scripts to prepare and clean data as well as generate tables and figures. Functions/: folder of subroutines for use with MATLAB scripts. Data/: data folder. Raw/: ECB SPF forecast data, realizations of target variables, and start and end bins for density forecasts. Intermediate/: Data used at intermediate steps in the cleaning process. These datasets are generated with x01_Raw_Data_Shell.do, x02a_Individual_Uncertainty_GDP.do, x02b_Individual_Uncertainty_HICP.do, x02c_Individual_Uncertainty_Urate.do, x03_Pull_Data.do, x04_Data_Clean_And_Merge, and x05_Drop_Low_Counts.do in the Code/ folder. Ready/: Data used to conduct regressions, statistical tests, and generate figures. Output/: folder of results. Figures/: .jpg files for each figure used in the paper and its appendix. HL Results/: Results from applying the Hounyo and Lahiri (2023) testing procedure for equal predictive performance to ECB SPF forecast data. This folder contains the material for Tables 1A-4A. Regressions/: Regression results, as well as material for Tables 3 and 5A. Simulations/: Results from simulation exercise as well as the datasets used to create Figures 9-12. Statistical Tests/: Results displayed in Tables 1 and 2. The repository also contains the manuscript, appendix, and this read-me file.DisclaimerThis replication package was produced by the authors and is not an official product of the Federal Reserve Bank of Cleveland. The analysis and conclusions set forth are those of the authors and do not indicate concurrence by the Federal Reserve Bank of Cleveland or the Federal Reserve System.

Initial data analysis checklist for data screening in longitudinal studies.

The global Data Preparation Platform market is poised for substantial growth, estimated to reach $15,600 million by the study's end in 2033, up from $6,000 million in the base year of 2025. This trajectory is fueled by a Compound Annual Growth Rate (CAGR) of approximately 12.5% over the forecast period. The proliferation of big data and the increasing need for clean, usable data across all business functions are primary drivers. Organizations are recognizing that effective data preparation is foundational to accurate analytics, informed decision-making, and successful AI/ML initiatives. This has led to a surge in demand for platforms that can automate and streamline the complex, time-consuming process of data cleansing, transformation, and enrichment. The market's expansion is further propelled by the growing adoption of cloud-based solutions, offering scalability, flexibility, and cost-efficiency, particularly for Small & Medium Enterprises (SMEs). Key trends shaping the Data Preparation Platform market include the integration of AI and machine learning for automated data profiling and anomaly detection, enhanced collaboration features to facilitate teamwork among data professionals, and a growing focus on data governance and compliance. While the market exhibits robust growth, certain restraints may temper its pace. These include the complexity of integrating data preparation tools with existing IT infrastructures, the shortage of skilled data professionals capable of leveraging advanced platform features, and concerns around data security and privacy. Despite these challenges, the market is expected to witness continuous innovation and strategic partnerships among leading companies like Microsoft, Tableau, and Alteryx, aiming to provide more comprehensive and user-friendly solutions to meet the evolving demands of a data-driven world. Here's a comprehensive report description on Data Preparation Platforms, incorporating the requested information, values, and structure:

Directory Cleanup Tools Market Outlook

According to our latest research, the global Directory Cleanup Tools market size reached USD 1.47 billion in 2024, reflecting a robust demand for efficient data hygiene and security solutions across industries. The market is experiencing a strong compound annual growth rate (CAGR) of 11.2% and is forecasted to expand to USD 4.06 billion by 2033. This growth trajectory is primarily driven by the increasing adoption of digital transformation initiatives, the proliferation of data across enterprise environments, and heightened concerns over data privacy and compliance with evolving regulatory frameworks.

The rapid expansion of digital infrastructures, coupled with the exponential growth in unstructured data, is a central growth factor for the Directory Cleanup Tools market. Organizations are facing unprecedented challenges in managing, organizing, and securing vast volumes of directory data, particularly as remote and hybrid work models become the norm. The need to streamline directory structures, remove redundant or obsolete accounts, and ensure that only authorized personnel have access to sensitive resources is driving enterprises toward automated directory cleanup solutions. These tools not only improve operational efficiency but also play a critical role in minimizing security risks, reducing storage costs, and ensuring compliance with global data protection regulations such as GDPR, HIPAA, and CCPA.

Another significant driver is the increasing integration of artificial intelligence (AI) and machine learning (ML) capabilities into directory cleanup tools. Advanced analytics, predictive modeling, and automation features enable organizations to proactively identify anomalies, automate repetitive cleanup tasks, and generate actionable insights for IT administrators. This technological evolution is transforming directory cleanup from a labor-intensive, manual process into a strategic, automated function that supports broader IT governance and risk management objectives. Furthermore, the rise of cloud computing and the proliferation of SaaS applications have necessitated robust directory management solutions that can operate seamlessly across on-premises and cloud environments, further fueling market demand.

Additionally, the growing awareness of the risks associated with stale, orphaned, or misconfigured directory entries is prompting organizations to prioritize directory hygiene as part of their overall cybersecurity strategy. Data breaches, unauthorized access, and insider threats often exploit vulnerabilities in directory structures, making cleanup tools an essential component of any defense-in-depth approach. As organizations continue to invest in digital transformation and cloud migration, the need for continuous, automated directory cleanup will only intensify, ensuring sustained market growth through the forecast period.

From a regional perspective, North America currently dominates the Directory Cleanup Tools market, accounting for the largest revenue share in 2024 due to its advanced IT infrastructure, stringent regulatory environment, and high adoption rates of cloud and hybrid IT models. However, Asia Pacific is emerging as the fastest-growing region, driven by rapid digitalization, increasing investments in IT security, and the proliferation of small and medium enterprises (SMEs) seeking to modernize their directory management practices. Europe, Latin America, and the Middle East & Africa are also witnessing steady growth, supported by rising awareness of data hygiene and compliance requirements. Each region presents unique opportunities and challenges, shaping the competitive dynamics and innovation landscape of the global Directory Cleanup Tools market.

Component Analysis

The Directory Cleanup Tools market is segmented by component into software and services, each playing a pivotal role in the overall ecosystem. The software segment, encompassing standalone cleanup solutions, integrated platforms, and automation tools, holds the largest market share. This dominance is attributed to the increasing demand for robust, scalable, and user-friendly software that can automate directory cleanup processes, identify redundant or obsolete entries, and ensure compliance with organizational policies. Software vendors are continuously innovating, integrating advanced features such as AI-powered analytics, real-time monitoring, and customizable reporting dashboards t

Context

The 2019 Kaggle ML & DS Survey data like its predecessors was a wonderful repository of data that helped understand the data science landscape of the world in better sense. However, this analysis was not so apparent because of the significant amount of cleaning needed to convert the data into a format that would aid in quick exploratory analysis. This was especially daunting for beginners like me. So, I took up the chance to try and clean the data up a bit so that it could be beneficial to other beginners like me. In this way, people can save up a great deal of time in the data cleaning process.

This was my aim. Hope it helps 😄

P.S : This is also my first core messy-data-cleaning project.

Content

Original Survey Data : The multiple_choice_responses.csv file in 2019 Kaggle ML and DS Survey Data

Sequence of Cleaning : I followed a bit of a sequential process in data cleaning : * Step 1. Removed all the features from the dataset that were "OTHER_TEXT". These features were encoded with -1 or 1, so it was logical to remove these * Step 2. Grouped all the features belonging to a similar question. This was needed as certain questions that had the "Select all that apply" choice, were split as multiple features(each feature corresponded to one of the choices selected by a respondent). * Step 3. Combined all the responses for a given question from multiple features and group them together as a list. * Step 4. Finally, re-arranged the headers in appropriate positions and saved the data.

Notebook where the Data Cleaning was performed : Kaggle DS and ML Survey 2019 - Data Cleaning

Bug : There is a slight extra column in the final dataset that was generated due to a small inaccuracy in generating it. The first column is Unnamed: 0. However, this can easily be gotten rid off while you use it. Just use the following code block to load the data :
```

loading data

df = pd.read_csv(file_path)

first column is an extra; remove it

df = df.drop(["Unnamed: 0"], axis=1) ```

Acknowledgements

I thank the Kaggle Team for conducting the survey and making the data open. It was great fun working on this data cleaning project.

Image Credits : Photo by pan xiaozhen on Unsplash

Inspiration

Hopefully, you can use this dataset to unearth deeper patterns within it and understand the data science scenario in the world in greater perspective, all by not having to spend too much time on data cleaning!

Analyzing Coffee Shop Sales: Excel Insights 📈

In my first Data Analytics Project, I Discover the secrets of a fictional coffee shop's success with my data-driven analysis. By Analyzing a 5-sheet Excel dataset, I've uncovered valuable sales trends, customer preferences, and insights that can guide future business decisions. 📊☕

DATA CLEANING 🧹

• REMOVED DUPLICATES OR IRRELEVANT ENTRIES: Thoroughly eliminated duplicate records and irrelevant data to refine the dataset for analysis.

• FIXED STRUCTURAL ERRORS: Rectified any inconsistencies or structural issues within the data to ensure uniformity and accuracy.

• CHECKED FOR DATA CONSISTENCY: Verified the integrity and coherence of the dataset by identifying and resolving any inconsistencies or discrepancies.

DATA MANIPULATION 🛠️

• UTILIZED LOOKUPS: Used Excel's lookup functions for efficient data retrieval and analysis.

• IMPLEMENTED INDEX MATCH: Leveraged the Index Match function to perform advanced data searches and matches.

• APPLIED SUMIFS FUNCTIONS: Utilized SumIFs to calculate totals based on specified criteria.

• CALCULATED PROFITS: Used relevant formulas and techniques to determine profit margins and insights from the data.

PIVOTING THE DATA 𝄜

• CREATED PIVOT TABLES: Utilized Excel's PivotTable feature to pivot the data for in-depth analysis.

• FILTERED DATA: Utilized pivot tables to filter and analyze specific subsets of data, enabling focused insights. Specially used in “PEAK HOURS” and “TOP 3 PRODUCTS” charts.

VISUALIZATION 📊

• KEY INSIGHTS: Unveiled the grand total sales revenue while also analyzing the average bill per person, offering comprehensive insights into the coffee shop's performance and customer spending habits.

• SALES TREND ANALYSIS: Used Line chart to compute total sales across various time intervals, revealing valuable insights into evolving sales trends.

• PEAK HOUR ANALYSIS: Leveraged Clustered Column chart to identify peak sales hours, shedding light on optimal operating times and potential staffing needs.

• TOP 3 PRODUCTS IDENTIFICATION: Utilized Clustered Bar chart to determine the top three coffee types, facilitating strategic decisions regarding inventory management and marketing focus.

*I also used a Timeline to visualize chronological data trends and identify key patterns over specific times.

While it's a significant milestone for me, I recognize that there's always room for growth and improvement. Your feedback and insights are invaluable to me as I continue to refine my skills and tackle future projects. I'm eager to hear your thoughts and suggestions on how I can make my next endeavor even more impactful and insightful.

THANKS TO: WsCube Tech Mo Chen Alex Freberg

TOOLS USED: Microsoft Excel

DataAnalytics #DataAnalyst #ExcelProject #DataVisualization #BusinessIntelligence #SalesAnalysis #DataAnalysis #DataDrivenDecisions

Duplicate Folder Cleanup Tools Market Outlook

According to our latest research, the global Duplicate Folder Cleanup Tools market size reached USD 1.24 billion in 2024, with a robust growth trajectory expected throughout the forecast period. The market is projected to expand at a CAGR of 11.2% from 2025 to 2033, reaching a forecasted value of USD 3.13 billion by 2033. This significant growth is fueled by the increasing demand for efficient data management solutions across enterprises and individuals, driven by the exponential rise in digital content and the need to optimize storage resources.

The primary growth factor for the Duplicate Folder Cleanup Tools market is the unprecedented surge in digital data generation across all sectors. Organizations and individuals alike are grappling with vast amounts of redundant files and folders that not only consume valuable storage space but also hinder operational efficiency. As businesses undergo digital transformation and migrate to cloud platforms, the risk of data duplication escalates, necessitating advanced duplicate folder cleanup tools. These solutions play a pivotal role in reducing storage costs, enhancing data accuracy, and streamlining workflows, making them indispensable in today’s data-driven landscape.

Another critical driver contributing to the market’s expansion is the increasing adoption of cloud computing and hybrid IT environments. As enterprises shift their infrastructure to cloud-based platforms, the complexity of managing and organizing data multiplies. Duplicate folder cleanup tools, especially those with robust automation and AI-powered features, are being rapidly integrated into cloud ecosystems to address these challenges. The ability to seamlessly identify, analyze, and remove redundant folders across diverse environments is a compelling value proposition for organizations aiming to maintain data hygiene and regulatory compliance.

Furthermore, the growing emphasis on data security and compliance is accelerating the uptake of duplicate folder cleanup solutions. Regulatory frameworks such as GDPR, HIPAA, and CCPA mandate stringent data management practices, including the elimination of unnecessary or duplicate records. Failure to comply can result in substantial penalties and reputational damage. As a result, organizations are investing in advanced duplicate folder cleanup tools that not only enhance storage efficiency but also ensure adherence to legal and industry standards. The integration of these tools with enterprise data governance strategies is expected to further propel market growth in the coming years.

Regionally, North America continues to dominate the Duplicate Folder Cleanup Tools market, accounting for the largest share in 2024, followed closely by Europe and Asia Pacific. The high adoption rate of digital technologies, coupled with the presence of leading software vendors and tech-savvy enterprises, positions North America as a key growth engine. Meanwhile, Asia Pacific is witnessing the fastest CAGR, driven by rapid digitalization, expanding IT infrastructure, and increasing awareness about efficient data management solutions. Latin America and Middle East & Africa are also emerging as promising markets, supported by growing investments in digital transformation initiatives.

Component Analysis

The Component segment of the Duplicate Folder Cleanup Tools market is bifurcated into Software and Services, both of which play integral roles in addressing the challenges of data redundancy. Software solutions form the backbone of this segment, encompassing standalone applications, integrated modules, and AI-powered platforms designed to automate the detection and removal of duplicate folders. The software segment leads the market, owing to its scalability, ease of deployment, and continuous innovation in features such as real-time monitoring, advanced analytics, and seamless integration with existing IT ecosystems. Organizations are increasingly prioritizing software that offers intuitive user interfaces and robust security protocols, ensuring both efficiency and compliance.

On the other hand, the Services segment includes consulting, implementation, customization, and support services that complement software offerings. As enterprises grapple with complex IT environments, the demand for specialized services to tailor duplicate folder cleanup solutions to uniqu

The global multi-function cleaning cars market is projected to reach a market size of USD XXX million by 2033, growing at a CAGR of XX% during the forecast period (2025-2033). The growth of the market is attributed to the increasing demand for efficient and versatile cleaning solutions in various industries, including healthcare, hospitality, and manufacturing. The adoption of smart cleaning technologies and the rising awareness of hygiene and cleanliness standards are also driving the market growth. Key trends shaping the multi-function cleaning cars market include the integration of artificial intelligence (AI) and automation, the development of eco-friendly and sustainable cleaning solutions, and the emergence of on-demand cleaning services. The growing emphasis on workplace safety and employee well-being is expected to further fuel the demand for multi-function cleaning cars that can effectively disinfect and clean large areas. The market is expected to be competitive, with established players such as Carlisle, Aosom, Sitoo, and Janico dominating the landscape. Regional variations in cleaning practices and the availability of local manufacturers are also likely to influence the market dynamics. The global multi-function cleaning car market is projected to grow from a valuation of USD 6.2 billion in 2023 to a colossal USD 12 billion by 2030, exhibiting a robust CAGR of 9.2% throughout the forecast period.

The Multi-function Cleaning Cars market is witnessing significant growth as industries seek efficient and versatile solutions to maintain cleanliness and hygiene in various environments. These specialized vehicles are designed to tackle multiple cleaning tasks, from litter collection to road washing, making them ind

klib library enables us to quickly visualize missing data, perform data cleaning, visualize data distribution plot, visualize correlation plot and visualize categorical column values. klib is a Python library for importing, cleaning, analyzing and preprocessing data. Explanations on key functionalities can be found on Medium / TowardsDataScience in the examples section or on YouTube (Data Professor).

Original Github repo

https://raw.githubusercontent.com/akanz1/klib/main/examples/images/header.png" alt="klib Header">

Usage

!pip install klib

import klib
import pandas as pd

df = pd.DataFrame(data)

# klib.describe functions for visualizing datasets
- klib.cat_plot(df) # returns a visualization of the number and frequency of categorical features
- klib.corr_mat(df) # returns a color-encoded correlation matrix
- klib.corr_plot(df) # returns a color-encoded heatmap, ideal for correlations
- klib.dist_plot(df) # returns a distribution plot for every numeric feature
- klib.missingval_plot(df) # returns a figure containing information about missing values

Examples

Take a look at this starter notebook.

Further examples, as well as applications of the functions can be found here.

Contributing

Pull requests and ideas, especially for further functions are welcome. For major changes or feedback, please open an issue first to discuss what you would like to change. Take a look at this Github repo.

License

MIT

The National Health and Nutrition Examination Survey (NHANES) provides data and have considerable potential to study the health and environmental exposure of the non-institutionalized US population. However, as NHANES data are plagued with multiple inconsistencies, processing these data is required before deriving new insights through large-scale analyses. Thus, we developed a set of curated and unified datasets by merging 614 separate files and harmonizing unrestricted data across NHANES III (1988-1994) and Continuous (1999-2018), totaling 135,310 participants and 5,078 variables. The variables conveydemographics (281 variables),dietary consumption (324 variables),physiological functions (1,040 variables),occupation (61 variables),questionnaires (1444 variables, e.g., physical activity, medical conditions, diabetes, reproductive health, blood pressure and cholesterol, early childhood),medications (29 variables),mortality information linked from the National Death Index (15 variables),survey weights (857 variables),environmental exposure biomarker measurements (598 variables), andchemical comments indicating which measurements are below or above the lower limit of detection (505 variables).csv Data Record: The curated NHANES datasets and the data dictionaries includes 23 .csv files and 1 excel file.The curated NHANES datasets involves 20 .csv formatted files, two for each module with one as the uncleaned version and the other as the cleaned version. The modules are labeled as the following: 1) mortality, 2) dietary, 3) demographics, 4) response, 5) medications, 6) questionnaire, 7) chemicals, 8) occupation, 9) weights, and 10) comments."dictionary_nhanes.csv" is a dictionary that lists the variable name, description, module, category, units, CAS Number, comment use, chemical family, chemical family shortened, number of measurements, and cycles available for all 5,078 variables in NHANES."dictionary_harmonized_categories.csv" contains the harmonized categories for the categorical variables.“dictionary_drug_codes.csv” contains the dictionary for descriptors on the drugs codes.“nhanes_inconsistencies_documentation.xlsx” is an excel file that contains the cleaning documentation, which records all the inconsistencies for all affected variables to help curate each of the NHANES modules.R Data Record: For researchers who want to conduct their analysis in the R programming language, only cleaned NHANES modules and the data dictionaries can be downloaded as a .zip file which include an .RData file and an .R file.“w - nhanes_1988_2018.RData” contains all the aforementioned datasets as R data objects. We make available all R scripts on customized functions that were written to curate the data.“m - nhanes_1988_2018.R” shows how we used the customized functions (i.e. our pipeline) to curate the original NHANES data.Example starter codes: The set of starter code to help users conduct exposome analysis consists of four R markdown files (.Rmd). We recommend going through the tutorials in order.“example_0 - merge_datasets_together.Rmd” demonstrates how to merge the curated NHANES datasets together.“example_1 - account_for_nhanes_design.Rmd” demonstrates how to conduct a linear regression model, a survey-weighted regression model, a Cox proportional hazard model, and a survey-weighted Cox proportional hazard model.“example_2 - calculate_summary_statistics.Rmd” demonstrates how to calculate summary statistics for one variable and multiple variables with and without accounting for the NHANES sampling design.“example_3 - run_multiple_regressions.Rmd” demonstrates how run multiple regression models with and without adjusting for the sampling design.

Datasets (all) for this work, provided in.csv format for direct import into R. The data collection consists of the following datasets:

All.data.csv

This dataset contains the data used for the first behavioural model in PhD chapter 3, and the associated manuscript accepted in Marine Biology entitled: Cleaner shrimp are true cleaners of injured fish [authors: David B Vaughan, Alexandra S Grutter, Hugh W Ferguson, Rhondda Jones, Kate S Hutson]. This dataset informed the initial exploratory mixed effects random intercept model using all cleaning contact locations (fish sides, oral, and ventral) recorded on the fish per day testing the response variable ‘cleaning time’ as a function of the fixed effects ‘day’, ‘cleaning contact locations’, and interaction ‘day x cleaning contact locations’, and ‘fish’ and ‘shrimp’ as random effects.

All.dataR.14.csv

This dataset contains the data used for the second to fifth behavioural models model in PhD chapter 3, and the associated manuscript accepted in Marine Biology entitled: Cleaner shrimp are true cleaners of injured fish [authors: David B Vaughan, Alexandra S Grutter, Hugh W Ferguson, Rhondda Jones, Kate S Hutson]. This is a subset of All.data.csv which excludes oral and ventral cleaning contact locations (scenarios 5 and 6). The analysis for All.data.csv was repeated using this analysis initially, and then two alternative approaches were used to model temporal change in cleaning times. In the first, day was treated as a numeric variable, included in the model as either a quadratic or a linear function to test for curvature testing the response variable ‘cleaning time’ as a function of the fixed effects ‘cleaning contact locations’, ‘day’, ‘day2’, and the interactions ‘cleaning contact locations with day’, ‘cleaning contact locations with day2’, and ‘fish’ and ‘shrimp’ as random effects. This analysis was carried out twice, once including all of the data, and once excluding day 0, to determine whether any temporal changes in behaviour extended beyond the initial establishment period of injury. In the second approach, based on the results of the first, the data were re-analysed with day treated as a category having two binary classes, ‘day0’ and ‘>day0’.

Jolts.data1.csv

This dataset was used for the analysis of jolting in PhD chapter 3, and the associated manuscript accepted in Marine Biology entitled: Cleaner shrimp are true cleaners of injured fish [authors: David B Vaughan, Alexandra S Grutter, Hugh W Ferguson, Rhondda Jones, Kate S Hutson]. The number of ‘jolts’ were analysed using a random-intercept mixed effects model with ‘fish’ and ‘shrimp’ as random effects, and ‘treatment’ (two levels: Injured_with_shrimp; Uninjured_with_shrimp), and ‘day’ as fixed effects.

Red.csv

This dataset was used for the analysis of injury redness (rubor) in PhD chapter 3, and the associated manuscript accepted in Marine Biology entitled: Cleaner shrimp are true cleaners of injured fish [authors: David B Vaughan, Alexandra S Grutter, Hugh W Ferguson, Rhondda Jones, Kate S Hutson]. The analysis examined spectral differences between groups with and without shrimp over the subsequent period to examine whether the presence of shrimp affected the spectral properties of the injury site as the injury healed. For this analysis, ‘day’ (either 4 or 6), ‘shrimp presence’ and the ‘shrimp x day’ interaction were all included as potential explanatory variables.

Yellow.csv

As for Red.csv.

UV1.csv

This dataset was used for the Nonspecific tissue damage analysis in PhD chapter 3, and the associated manuscript accepted in Marine Biology entitled: Cleaner shrimp are true cleaners of injured fish [authors: David B Vaughan, Alexandra S Grutter, Hugh W Ferguson, Rhondda Jones, Kate S Hutson]. Nonspecific tissue damage area was investigated between two levels of four treatment groups (With shrimp and Without shrimp; Injured fish and Uninjured fish) over time to determine their effects on tissue damage. Mixed effects random-intercept models were employed, with the ‘fish’ as the random effect to allow for photographic sampling on both sides of the same fish. The response variable ‘tissue damage area’ was tested as a function of the fixed effects ‘treatment’, ‘side’, ‘day’ (as a factor). Two levels of fish sides were included in the analyses representing injured and uninjured sides.

Dialog Cleanup Tools Market Outlook

According to our latest research, the global Dialog Cleanup Tools market size reached USD 1.12 billion in 2024, demonstrating robust expansion in response to the surging demand for high-quality audio and text outputs across industries. The market is expected to grow at a CAGR of 18.4% from 2025 to 2033, resulting in a forecasted market size of USD 5.85 billion by 2033. Key growth factors include the rapid adoption of advanced AI and machine learning technologies for speech and text processing, increasing reliance on virtual communications, and a heightened emphasis on customer experience and compliance in regulated sectors.

The growth trajectory of the Dialog Cleanup Tools market is primarily driven by the exponential rise in virtual communication channels, especially post-pandemic, which has underscored the need for accurate, clear, and contextually relevant dialog in both audio and text formats. Enterprises are increasingly investing in dialog cleanup tools to enhance customer interactions, ensure compliance, and extract actionable insights from vast volumes of conversational data. The proliferation of digital transformation initiatives across sectors such as healthcare, legal, and media & entertainment further accelerates the adoption of these solutions. The integration of natural language processing (NLP), deep learning, and real-time noise reduction capabilities is enabling dialog cleanup tools to deliver superior accuracy and efficiency, making them indispensable for organizations aiming to optimize communication workflows and improve service delivery.

Another significant growth factor is the evolution of customer service paradigms, where dialog cleanup tools play a pivotal role in refining both automated and human-assisted interactions. With the increasing prevalence of chatbots, voice assistants, and contact center solutions, businesses are leveraging dialog cleanup technologies to ensure clarity, relevance, and compliance in every customer touchpoint. The surge in remote work and global collaboration has also heightened the need for transcription and translation services powered by dialog cleanup tools, especially in multinational enterprises and SMEs. Furthermore, regulatory requirements in sectors such as healthcare and legal mandate the accurate documentation and archiving of conversations, further fueling market demand.

Technological advancements in dialog cleanup tools, including the deployment of cloud-based solutions and the integration of AI-powered analytics, are reshaping the competitive landscape. Vendors are focusing on enhancing product capabilities such as real-time processing, multi-language support, and seamless integration with existing enterprise systems. The emergence of customizable and scalable dialog cleanup solutions is enabling organizations of all sizes to address unique communication challenges, thereby expanding the addressable market. Additionally, the growing recognition of the importance of data privacy and security is prompting solution providers to incorporate robust encryption and compliance features, making dialog cleanup tools more attractive to regulated industries.

From a regional perspective, North America continues to dominate the Dialog Cleanup Tools market, accounting for the largest revenue share in 2024, followed by Europe and Asia Pacific. The presence of leading technology vendors, high digital adoption rates, and stringent regulatory frameworks in North America are key contributors to this leadership. Meanwhile, Asia Pacific is expected to witness the fastest CAGR during the forecast period, driven by rapid digitalization, the expansion of the BPO sector, and increasing investments in AI and automation technologies. While Latin America and the Middle East & Africa are still emerging markets, they present substantial growth opportunities due to rising enterprise adoption and the gradual modernization of communication infrastructures.

Component Analysis

The Dialog Cleanup Tools market is segmented by component into software and services, each playing a critical role in the overall ecosystem. The software segment, comprising standalone applications and integrated platforms, commands the majority share of the market due to its scalability, flexibility, and continuous innovation in AI-driven features. Modern dialog cleanup software leverages advanced algorithms for noise reduction, speech enhancement, and contextual understanding, e

The Cleaning Combination Machines market has emerged as a vital segment within the industrial cleaning sector, providing multifunctional solutions designed to enhance efficiency and effectiveness in various cleaning operations. These machines combine multiple cleaning functions-such as scrubbing, sweeping, and vacuu

HiSeq raw data, and processed representative sequences files:- Mothur output files: Taxonomy_file, shared_file, summary, otu_rep_output, otu_rep_fasta_associated- Pipit output files: otu_table_mod_biom, repseqs.fasta- CD-HIT output files: 11_Ac_Plant_H2NJJBC11_Ac_Plant_H2NJJBC.clstr47_Ac_Plant_H2NJJBC47_Ac_Plant_H2NJJBC.clstr

Contamination of body surfaces can negatively affect many physiological functions. Insects have evolved different adaptations for removing contamination, including surfaces that allow passive self-cleaning and structures for active cleaning. Here, we study the function of the antenna cleaner in Camponotus rufifemur ants, a clamp-like structure consisting of a notch on the basitarsus facing a spur on the tibia, both bearing cuticular 'combs' and 'brushes'. The ants clamp one antenna tightly between notch and spur, pull it through, and subsequently clean the antenna cleaner itself with the mouthparts. We simulated cleaning strokes by moving notch or spur over antennae contaminated with fluorescent particles. The notch removed particles more efficiently than the spur, but both components eliminated more than 60% of the particles with the first stroke. Ablation of bristles, brush and comb strongly reduced the efficiency, indicating that they are essential for cleaning. To study how comb and brush remove particles of different sizes, we contaminated antennae of living ants, and anaesthetized them immediately after they had performed the first cleaning stroke. Different-sized beads were trapped in distinct zones of the notch, consistent with the gap widths between cuticular outgrowths. This suggests that the antenna cleaner operates like a series of sieves that remove the largest objects first, followed by smaller ones, down to the smallest particles that get caught by adhesion.