Data Cleansing Software Market Outlook

The global data cleansing software market size was valued at approximately USD 1.5 billion in 2023 and is projected to reach around USD 4.2 billion by 2032, exhibiting a compound annual growth rate (CAGR) of 12.5% during the forecast period. This substantial growth can be attributed to the increasing importance of maintaining clean and reliable data for business intelligence and analytics, which are driving the adoption of data cleansing solutions across various industries.

The proliferation of big data and the growing emphasis on data-driven decision-making are significant growth factors for the data cleansing software market. As organizations collect vast amounts of data from multiple sources, ensuring that this data is accurate, consistent, and complete becomes critical for deriving actionable insights. Data cleansing software helps organizations eliminate inaccuracies, inconsistencies, and redundancies, thereby enhancing the quality of their data and improving overall operational efficiency. Additionally, the rising adoption of advanced analytics and artificial intelligence (AI) technologies further fuels the demand for data cleansing software, as clean data is essential for the accuracy and reliability of these technologies.

Another key driver of market growth is the increasing regulatory pressure for data compliance and governance. Governments and regulatory bodies across the globe are implementing stringent data protection regulations, such as the General Data Protection Regulation (GDPR) in Europe and the California Consumer Privacy Act (CCPA) in the United States. These regulations mandate organizations to ensure the accuracy and security of the personal data they handle. Data cleansing software assists organizations in complying with these regulations by identifying and rectifying inaccuracies in their data repositories, thus minimizing the risk of non-compliance and hefty penalties.

The growing trend of digital transformation across various industries also contributes to the expanding data cleansing software market. As businesses transition to digital platforms, they generate and accumulate enormous volumes of data. To derive meaningful insights and maintain a competitive edge, it is imperative for organizations to maintain high-quality data. Data cleansing software plays a pivotal role in this process by enabling organizations to streamline their data management practices and ensure the integrity of their data. Furthermore, the increasing adoption of cloud-based solutions provides additional impetus to the market, as cloud platforms facilitate seamless integration and scalability of data cleansing tools.

Regionally, North America holds a dominant position in the data cleansing software market, driven by the presence of numerous technology giants and the rapid adoption of advanced data management solutions. The region is expected to continue its dominance during the forecast period, supported by the strong emphasis on data quality and compliance. Europe is also a significant market, with countries like Germany, the UK, and France showing substantial demand for data cleansing solutions. The Asia Pacific region is poised for significant growth, fueled by the increasing digitalization of businesses and the rising awareness of data quality's importance. Emerging economies in Latin America and the Middle East & Africa are also expected to witness steady growth, driven by the growing adoption of data-driven technologies.

The role of Data Quality Tools cannot be overstated in the context of data cleansing software. These tools are integral in ensuring that the data being processed is not only clean but also of high quality, which is crucial for accurate analytics and decision-making. Data Quality Tools help in profiling, monitoring, and cleansing data, thereby ensuring that organizations can trust their data for strategic decisions. As organizations increasingly rely on data-driven insights, the demand for robust Data Quality Tools is expected to rise. These tools offer functionalities such as data validation, standardization, and enrichment, which are essential for maintaining the integrity of data across various platforms and applications. The integration of these tools with data cleansing software enhances the overall data management capabilities of organizations, enabling them to achieve greater operational efficiency and compliance with data regulations.

Component Analysis

The data cle

Graph and download economic data for Expenditures: Laundry and Cleaning Supplies by Income Before Taxes: Total Complete Income Reporters (CXULAUNDRYLB02A2M) from 1984 to 2003 about laundry, cleaning, supplies, tax, expenditures, income, and USA.

Context

This notebook serves to showcase my problem solving ability, knowledge of the data analysis process, proficiency with Excel and its various tools and functions, as well as my strategic mindset and statistical prowess. This project consist of an auditing prompt provided by Hive Data, a raw Excel data set, a cleaned and audited version of the raw Excel data set, and my description of my thought process and knowledge used during completion of the project. The prompt can be found below:

Hive Data Audit Prompt

The raw data that accompanies the prompt can be found below:

Hive Annotation Job Results - Raw Data

^ These are the tools I was given to complete my task. The rest of the work is entirely my own.

To summarize broadly, my task was to audit the dataset and summarize my process and results. Specifically, I was to create a method for identifying which "jobs" - explained in the prompt above - needed to be rerun based on a set of "background facts," or criteria. The description of my extensive thought process and results can be found below in the Content section.

Content

Brendan Kelley April 23, 2021

Hive Data Audit Prompt Results

This paper explains the auditing process of the “Hive Annotation Job Results” data. It includes the preparation, analysis, visualization, and summary of the data. It is accompanied by the results of the audit in the excel file “Hive Annotation Job Results – Audited”.

Observation

The “Hive Annotation Job Results” data comes in the form of a single excel sheet. It contains 7 columns and 5,001 rows, including column headers. The data includes “file”, “object id”, and the pseudonym for five questions that each client was instructed to answer about their respective table: “tabular”, “semantic”, “definition list”, “header row”, and “header column”. The “file” column includes non-unique (that is, there are multiple instances of the same value in the column) numbers separated by a dash. The “object id” column includes non-unique numbers ranging from 5 to 487539. The columns containing the answers to the five questions include Boolean values - TRUE or FALSE – which depend upon the yes/no worker judgement.

Use of the COUNTIF() function reveals that there are no values other than TRUE or FALSE in any of the five question columns. The VLOOKUP() function reveals that the data does not include any missing values in any of the cells.

Assumptions

Based on the clean state of the data and the guidelines of the Hive Data Audit Prompt, the assumption is that duplicate values in the “file” column are acceptable and should not be removed. Similarly, duplicated values in the “object id” column are acceptable and should not be removed. The data is therefore clean and is ready for analysis/auditing.

Preparation

The purpose of the audit is to analyze the accuracy of the yes/no worker judgement of each question according to the guidelines of the background facts. The background facts are as follows:

• A table that is a definition list should automatically be tabular and also semantic • Semantic tables should automatically be tabular • If a table is NOT tabular, then it is definitely not semantic nor a definition list • A tabular table that has a header row OR header column should definitely be semantic

These background facts serve as instructions for how the answers to the five questions should interact with one another. These facts can be re-written to establish criteria for each question:

For tabular column: - If the table is a definition list, it is also tabular - If the table is semantic, it is also tabular

For semantic column: - If the table is a definition list, it is also semantic - If the table is not tabular, it is not semantic - If the table is tabular and has either a header row or a header column...

The LScDC (Leicester Scientific Dictionary-Core Dictionary)April 2020 by Neslihan Suzen, PhD student at the University of Leicester (ns433@leicester.ac.uk/suzenneslihan@hotmail.com)Supervised by Prof Alexander Gorban and Dr Evgeny Mirkes[Version 3] The third version of LScDC (Leicester Scientific Dictionary-Core) is formed using the updated LScD (Leicester Scientific Dictionary) - Version 3*. All steps applied to build the new version of core dictionary are the same as in Version 2** and can be found in description of Version 2 below. We did not repeat the explanation. The files provided with this description are also same as described as for LScDC Version 2. The numbers of words in the 3rd versions of LScD and LScDC are summarized below. # of wordsLScD (v3) 972,060LScDC (v3) 103,998 * Suzen, Neslihan (2019): LScD (Leicester Scientific Dictionary). figshare. Dataset. https://doi.org/10.25392/leicester.data.9746900.v3 ** Suzen, Neslihan (2019): LScDC (Leicester Scientific Dictionary-Core). figshare. Dataset. https://doi.org/10.25392/leicester.data.9896579.v2[Version 2] Getting StartedThis file describes a sorted and cleaned list of words from LScD (Leicester Scientific Dictionary), explains steps for sub-setting the LScD and basic statistics of words in the LSC (Leicester Scientific Corpus), to be found in [1, 2]. The LScDC (Leicester Scientific Dictionary-Core) is a list of words ordered by the number of documents containing the words, and is available in the CSV file published. There are 104,223 unique words (lemmas) in the LScDC. This dictionary is created to be used in future work on the quantification of the sense of research texts. The objective of sub-setting the LScD is to discard words which appear too rarely in the corpus. In text mining algorithms, usage of enormous number of text data brings the challenge to the performance and the accuracy of data mining applications. The performance and the accuracy of models are heavily depend on the type of words (such as stop words and content words) and the number of words in the corpus. Rare occurrence of words in a collection is not useful in discriminating texts in large corpora as rare words are likely to be non-informative signals (or noise) and redundant in the collection of texts. The selection of relevant words also holds out the possibility of more effective and faster operation of text mining algorithms.To build the LScDC, we decided the following process on LScD: removing words that appear in no more than 10 documents (

The dynamics and prevalence of mutualistic interactions, which are responsible for the maintenance and structuring of all ecological communities, are vulnerable to changes in abiotic and biotic environmental conditions. Mutualistic outcomes can quickly shift from cooperation to conflict, but it unclear how resilient and stable mutualistic outcomes are to more variable conditions. Tidally controlled coral atoll lagoons that experience extreme diurnal environmental shifts thus provide a model from which to test plasticity in mutualistic behavior of dedicated (formerly obligate) cleaner fish, which acquire all their food resources through client interactions. Here we investigated cleaning patterns of a model cleaner fish species, the bluestreak wrasse (Labroides dimidiatus), in an isolated tidal lagoon on the Great Barrier Reef. Under tidally restricted conditions, uniquely both adults and juveniles were part-time facultative cleaners, pecking on Isopora palifera coral. The mutualism was n...

1. Cleaning symbioses on coral reefs are mutually beneficial interactions between two individuals, in which a 'cleaner' removes and eats parasites from the surface of a 'client' fish. A suite of behavioural and morphological traits of cleaners signal cooperation with co-evolved species, thus protecting the cleaner from being eaten by otherwise predatory clients. However, it is unclear whether cooperation between cleaners and predatory clients is innate or learned, and therefore whether an introduced predator might consume, cooperate with, or alter the behaviour of cleaners.
2. We explored the role of learning in cleaning symbioses by comparing the interactions of native cleaner fishes with both naïve and experienced, non-native and native fish predators. In so doing, we tested the vulnerability of the predominant cleaners on Atlantic coral reefs, cleaning gobies (Elacatinus spp.), to the recent introduction of a generalist predator, the Indo-Pacific red lionfish (Pterois volitans).
3. Naïve juveniles of both invasive (P. volitans) and native predators (Cephalopholis spp. groupers) initially attacked cleaning gobies and hyperventilated from a putative toxin on the gobies' skin during laboratory experiments. After 1 to 5 such encounters, invasive lionfish often approached the cleaner closely, then turned away without striking. Consistent with learned avoidance, invasive lionfish rarely interacted with cleaning gobies in the wild, either antagonistically or cooperatively, and did not affect gobies' abundance. Native predators showed little evidence of learning during early encounters; they repeatedly attacked the cleaner during laboratory experiments and hyperventilated less violently than did lionfish. However, consistent with learned cooperation, native predators rarely antagonised and were frequently cleaned by gobies in the wild.
4. We demonstrate that rapid, learned avoidance protects a distasteful cleaning mutualist from an invasive predator. The behavioural plasticity of this invader likely contributes to its success across its invaded range. Additionally, our results suggest that the cleaner's chemical defence most likely evolved as a way to deter predation and reinforce cooperation with naïve individuals of native species.

These are time-averaged velocity datasets for the clean Newtonian and non-Newtonian simulations.

Schedule and Monitor of Gully Cleaning for Dublin City These datasets show the gully cleaning statistics from 2004 to September 14th 2011. It consists attached 6 No. Excel Spreadsheets with the datasets from the Daily Returns section of the Gully Cleaning Application and one dataset from the Gully Repairs section of the gully application. They are divided into the five Dublin City Council administrative areas; Central, North Central, North West, Southeast, South Central. There is also a dataset containing details of all Gully repairs pending (all areas included).The datasets cover all Daily Returns since the gully cleaning programme commenced in 2004. Daily Returns are lists of the work that the gully cleaning crews carry out daily. All gullies on a street are cleaned where possible. A list of Omissions is recorded where some gullies may not have been cleaned due to lack of access or other reasons. Also, the gullies that required repair were noted. The Daily Returns datasets record only the number of gullies requiring repair on a particular street, not the details of the repair. Information in the fields is as follows: .Road name: street name or laneway denoted by nearest house or lamp post etc. If a road name is followed by the letters pl in capital letters than it means that either this road or a section of this road has been placed on the priority list due to a history of flooding or a higher potential of the gully blocking due to location etc. If a road name is followed by a number of zeros in the gullies inspected - gullies cleaned columns etc then it is very probable that this road was travelled during heavy rain as part of our flood zones and there was no flooding noted along this road at the time of travelling. Gullies inspected: number of gullies inspected along road/lane .A road name followed by lower case road names denotes a road that is part of more than one area in our gully cleaning areas and these lower case names denote the starting point and finishing point for the crews working in the particular area i.e. Howth Road All Saints Rd-Fairview denotes that the section of the Howth road between all saints road and Fairview are within the area that the crew have been asked to work in. Gullies cleaned :number of gullies cleaned from total inspected .Gully omissions :number of gullies missed i.e. Unable to put boom or shovel into gully pot due to parked cars / unable to lift grids / hoarding over gullies etc .Gully repairs: number of repairs based on inspections-note not all repairs prevent the gully from being cleaned. Comments box: this box is used to provide any additional information that may be of benefit and it can be noted that results of work carried out by the mini jet is placed in this box.

Abstract

The do-file marital_spouselinks.do combines all data on people's marital statuses and reported spouses to create the following datasets: 1. all_marital_reports - a listing of all the times an individual has reported their current marital status with the id numbers of the reported spouse(s); this listing is as reported so may include discrepancies (i.e. a 'Never married' status following a 'Married' one) 2. all_spouse_pairs_full - a listing of each time each spouse pair has been reported plus summary information on co-residency for each pair 3. all_spouse_pairs_clean_summarised - this summarises the data from all_spouse_pairs_full to give start and end dates of unions 4. marital_status_episodes - this combines data from all the sources to create episodes of marital status, each has a start and end date and a marital status, and if currently married, the spouse ids of the current spouse(s) if reported. There are several variables to indicate where each piece of information is coming from.

The first 2 datasets are made available in case people need the 'raw' data for any reason (i.e. if they only want data from one study) or if they wish to summarise the data in a different way to what is done for the last 2 datasets.

The do-file is quite complicated with many sources of data going through multiple processes to create variables in the datasets so it is not always straightforward to explain where each variable come from on the documentation. The 4 datasets build on each other and the do-file is documented throughout so anyone wanting to understand in great detail may be better off examining that. However, below is a brief description of how the datasets are created:

Marital status data are stored in the tables of the study they were collected in: AHS Adult Health Study [ahs_ahs1] CEN Census (initial CRS census) [cen_individ] CENM In-migration (CRS migration form) [crs_cenm] GP General form (filled for various reasons) [gp_gpform] SEI Socio-economic individual (annual survey from 2007 onwards) [css_sei] TBH TB household (study of household contacts of TB patients) [tb_tbh] TBO TB controls (matched controls for TB patients) [tb_tbo & tb_tboto2007] TBX TB cases (TB patients) [tb_tbx & tb_tbxto2007] In many of the above surveys as well as their current marital status, people were asked to report their current and past spouses along with (sometimes) some information about the marriage (start/end year etc.). These data are stored all together on the table gen_spouse, with variables indicating which study the data came from. Further evidence of spousal relationships is taken from gen_identity (if a couple appear as co-parents to a CRS member) and from crs_residency_episodes_clean_poly, a combined dataset (if they are living in the same household at the same time). Note that co-parent couples who are not reported in gen_spouse are only retained in the datasets if they have co-resident episodes.

The marital status data are appended together and the spouse id data merged in. Minimal data editing/cleaning is carried out. As the spouse data are in long format, this dataset is reshaped wide to have one line per marital status report (polygamy in the area allows for men to have multiple spouses at one time): this dataset is saved as all_marital_reports.

The list of reported spouses on gen_spouse is appended to a list of co-parents (from gen_identity) and this list is cleaned to try to identify and remove obvious id errors (incestuous links, same sex [these are not reported in this culture] and large age difference). Data reported by men and women are compared and variables created to show whether one or both of the couple report the union. Many records have information on start and end year of marriage, and all have the date the union was reported. This listing is compared to data from residency episodes to add dates that couples were living together (not all have start/end dates so this is to try to supplement this), in addition the dates that each member of the couple was last known to be alive or first known to be dead are added (from the residency data as well). This dataset with all the records available for each spouse pair is saved as all_spouse_pairs_full.

The date data from all_spouse_pairs_full are then summarised to get one line per couple with earliest and latest known married date for all, and, if available, marriage and separation date. For each date there are also variables created to indicate the source of the data.
As culture only allows for women having one spouse at a time, records for women with 'overlapping' husbands are cleaned. This dataset is then saved as all_spouse_pairs_clean_summarised.

Both the cleaned spouse pairs and the cleaned marital status datasets are converted into episodes: the spouse listing using the marriage or first known married date as the beginning and the last known married plus a year or separation date as the end, the marital status data records collapsed into periods of the same status being reported (following some cleaning to remove impossible reports) and the start date being the first of these reports, the end date being the last of the reports plus a year. These episodes are appended together and a series of processes run several times to remove overalapping episodes. To be able to assign specific spouse ids to each married episode, some episodes need to be 'split' into more than one (i.e. if a man is married to one woman from 2005 to 2017 and then marries another woman in 2008 and remains married to her till 2017 his intial married episode would be from 2005 to 2017, but this would need to be split into one from 2005 to 2008 which would just have 1 idspouse attached and another from 2008 to 2017, which would have 2 idspouse attached). After this splitting process the spouse ids are merged in.
The final episode dataset is saved as marital_status_episodes.

Analysis unit

Individual

Mode of data collection

Face-to-face [f2f]

India Imports: USD: HS: 34029052: Cleaning or Degreasing Preparations Not Having Basis of Soap or Other Organic Surface Active (Washing Preparations) data was reported at 0.080 USD mn in 2018. This stayed constant from the previous number of 0.080 USD mn for 2017. India Imports: USD: HS: 34029052: Cleaning or Degreasing Preparations Not Having Basis of Soap or Other Organic Surface Active (Washing Preparations) data is updated yearly, averaging 0.030 USD mn from Mar 2004 (Median) to 2018, with 15 observations. The data reached an all-time high of 0.080 USD mn in 2018 and a record low of 0.010 USD mn in 2008. India Imports: USD: HS: 34029052: Cleaning or Degreasing Preparations Not Having Basis of Soap or Other Organic Surface Active (Washing Preparations) data remains active status in CEIC and is reported by Ministry of Commerce and Industry. The data is categorized under India Premium Database’s Chemical and Petrochemical Sector – Table IN.RHG053: Chemical and Petrochemical Imports: Harmonized System: HS34: Soap, Organic Surface-Active Agents, Washing and Lubricating Preparations: USD.

India Exports: Volume: HS: 34022020: Cleaning or Degreasing Preparations not Having Basis of Soap or Other organic Surface Active data was reported at 174.820 kg th in 2018. This records an increase from the previous number of 142.570 kg th for 2017. India Exports: Volume: HS: 34022020: Cleaning or Degreasing Preparations not Having Basis of Soap or Other organic Surface Active data is updated yearly, averaging 68.630 kg th from Mar 2004 (Median) to 2018, with 15 observations. The data reached an all-time high of 2,989.110 kg th in 2014 and a record low of 8.870 kg th in 2006. India Exports: Volume: HS: 34022020: Cleaning or Degreasing Preparations not Having Basis of Soap or Other organic Surface Active data remains active status in CEIC and is reported by Ministry of Commerce and Industry. The data is categorized under India Premium Database’s Chemical and Petrochemical Sector – Table IN.RHG018: Chemical and Petrochemical Exports: Harmonized System: HS34: Soap, Organic Surface-Active Agents, Washing and Lubricating Preparations: Volume.

Cleaned data for producing results for the paper Functional diversity of the Australian flora: strong links to species richness and climate published in the Journal of Vegetation Science. The R scripts for reproducing the results can be found in Appendix S3 which is an Rmarkdown file. In the Rmarkdown file the first for chunks of code show the process for cleaning the data and after the heading Hypervolume results Appendix S4 can be used. Full datasets are available upon reasonable request.

Abstract

The main objective of this project is to collect household data for the ongoing assessment and monitoring of the socio-economic impacts of COVID-19 on households and family businesses in Vietnam. The estimated field work and sample size of households in each round is as follows:

Round 1 June fieldwork- approximately 6300 households (at least 1300 minority households) Round 2 August fieldwork - approximately 4000 households (at least 1000 minority households) Round 3 September fieldwork- approximately 4000 households (at least 1000 minority households) Round 4 December- approximately 4000 households (at least 1000 minority households) Round 5 - pending discussion

Geographic coverage

National, regional

Analysis unit

Households

Kind of data

Sample survey data [ssd]

Sampling procedure

The 2020 Vietnam COVID-19 High Frequency Phone Survey of Households (VHFPS) uses a nationally representative household survey from 2018 as the sampling frame. The 2018 baseline survey includes 46980 households from 3132 communes (about 25% of total communes in Vietnam). In each commune, one EA is randomly selected and then 15 households are randomly selected in each EA for interview. Out of the 15 households, 3 households have information collected on both income and expenditure (large module) as well as many other aspects. The remaining 12 other households have information collected on income, but do not have information collected on expenditure (small module). Therefore, estimation of large module includes 9396 households and are representative at regional and national levels, while the whole sample is representative at the provincial level.

We use the large module of to select the households for official interview of the VHFPS survey and the small module households as reserve for replacement. The sample size of large module has 9396 households, of which, there are 7951 households having phone number (cell phone or line phone).

After data processing, the final sample size is 6,213 households.

Mode of data collection

Computer Assisted Telephone Interview [cati]

Research instrument

The questionnaire for Round 1 consisted of the following sections Section 2. Behavior Section 3. Health Section 4. Education & Child caring Section 5A. Employment (main respondent) Section 5B. Employment (other household member) Section 6. Coping Section 7. Safety Nets Section 8. FIES

Cleaning operations

Data cleaning began during the data collection process. Inputs for the cleaning process include available interviewers’ note following each question item, interviewers’ note at the end of the tablet form as well as supervisors’ note during monitoring. The data cleaning process was conducted in following steps: • Append households interviewed in ethnic minority languages with the main dataset interviewed in Vietnamese. • Remove unnecessary variables which were automatically calculated by SurveyCTO • Remove household duplicates in the dataset where the same form is submitted more than once. • Remove observations of households which were not supposed to be interviewed following the identified replacement procedure. • Format variables as their object type (string, integer, decimal, etc.) • Read through interviewers’ note and make adjustment accordingly. During interviews, whenever interviewers find it difficult to choose a correct code, they are recommended to choose the most appropriate one and write down respondents’ answer in detail so that the survey management team will justify and make a decision which code is best suitable for such answer. • Correct data based on supervisors’ note where enumerators entered wrong code. • Recode answer option “Other, please specify”. This option is usually followed by a blank line allowing enumerators to type or write texts to specify the answer. The data cleaning team checked thoroughly this type of answers to decide whether each answer needed recoding into one of the available categories or just keep the answer originally recorded. In some cases, that answer could be assigned a completely new code if it appeared many times in the survey dataset.
• Examine data accuracy of outlier values, defined as values that lie outside both 5th and 95th percentiles, by listening to interview recordings. • Final check on matching main dataset with different sections, where information is asked on individual level, are kept in separate data files and in long form. • Label variables using the full question text. • Label variable values where necessary.

Response rate

The target for Round 1 is to complete interviews for 6300 households, of which 1888 households are located in urban area and 4475 households in rural area. In addition, at least 1300 ethnic minority households are to be interviewed. A random selection of 6300 households was made out of 7951 households for official interview and the rest as for replacement. However, the refusal rate of the survey was about 27 percent, and households from the small module in the same EA were contacted for replacement and these households are also randomly selected.

Scripts to clean EVI2 data obtained from the VIP lab (University of Arizona) website (https://vip.arizona.edu/about.php and https://vip.arizona.edu/viplab_data_explorer.php). Data obtained in 2012.- outlier detection and removal/replacement- alignment of 2 periodsThe manuscript detailing the methods and resulting data sets has been accepted for publication in Nature Scientific Data (05/11/2019).Instructions: use the R Markdown html file for instructions!Code last manipulated and tested in R 3.4.3 ("Kite-Eating Tree")

Abstract

The MENA region grapples with intensified climate challenges and mounting energy issues. Access to energy is becoming more challenging, particularly for energy importing countries in the region. This makes the transition to clean energy in MENA a vital one. Luckily, the region has inherent comparative advantages given the natural endowments of high solar radiation over much of the year and strong wind nodes.

This data set, collected over one round, covers a spectrum of company-specific details, including sector categorization, employee count, regulatory compliance, experiences with grid-based electricity, and the extent of clean energy transition among enterprises in Tunisia. The data is collected under a comprehensive cross-sectional survey (from December 2023 to February 2024) that investigates how Micro, Small, and Medium Enterprises (MSMEs) in Tunisia navigate their transition towards clean energy.

This survey comes under the activities of ERF newly launched project “The role of MSMEs in fostering inclusive and equitable economic growth in the context of the clean energy transition in MENA” project funded by IDRC. The project launches a series of quantitative national surveys in the 5 targeted counties namely Egypt, Jordan, Morocco, Lebanon and Tunisia

This initiative aims to gather crucial data reflecting the ongoing energy transition in these countries. The objective of this survey data aims at enhancing knowledge and contributing to strategic policy initiatives, seeking to pave the way for sustainable, efficient, and equitable energy management while addressing mitigation of emission and ensuring energy security and equity.

All Transitions to Clean Energy in MENA Enterprises surveys incorporate relatively comparable survey designs, with data on enterprises within the Arab countries (Egypt, Jordan, Morocco, Tunisia, and Lebanon).The harmonization was designed to create comparable data that can facilitate cross-country and comparative research between the five Arab countries.

Geographic coverage

National

Analysis unit

Enterprises

Universe

The target population is the non-governmental micro, small, and medium enterprises that commenced business operations before 2023.

Kind of data

Sample Survey Data [ssd]

Sampling procedure

The target population of the surveys was businesses with less than 100 employees that started business operations before 2023. An ideal sampling frame for a probability sample should cover all target population units, i.e., a list of all working businesses of size 100 employes or less that started operations before 2023 with their telephone numbers. Unfortunately, we could not find such list in Tunisia. Therefore, we used data from Tunisia Yellow Pages, and data from the Institut National de la Statistique (INS). We had access to a list of about 185,966 businesses from INS and 8,685 businesses from the Yellow Pages. Although the sampling frames does not necessarily cover all businesses in Tunisia, it is large enough and cover many business sectors. Unfortunately, we could not assess the coverage of the sampling frame especially with the lack of official numbers about our target population in Tunisia. Four sample were selected for the survey of Renewable Energy in Tunisia, three sample from the INS frame and one sample from the Yellow Pages. The three samples of INS were stratified by sample sectors and business size; those samples were selected from businesses with telephone numbers (businesses without telephone number were not considered for selection). All businesses from the Yellow Pages were included in the fourth sample; no selection was done.

Mode of data collection

Computer Assisted Telephone Interview [cati]

Research instrument

Note: The questionnaire can be seen in the documentation materials tab.

Response rate

Response rate is 3.5%, after excluding those phones that were not in service and firms that were not eligible from the response rate.

python scripts and functions needed to view and clean saccade data

Data Cleaning Tools Market Outlook

As of 2023, the global market size for data cleaning tools is estimated at $2.5 billion, with projections indicating that it will reach approximately $7.1 billion by 2032, reflecting a robust CAGR of 12.1% during the forecast period. This growth is primarily driven by the increasing importance of data quality in business intelligence and analytics workflows across various industries.

The growth of the data cleaning tools market can be attributed to several critical factors. Firstly, the exponential increase in data generation across industries necessitates efficient tools to manage data quality. Poor data quality can result in significant financial losses, inefficient business processes, and faulty decision-making. Organizations recognize the value of clean, accurate data in driving business insights and operational efficiency, thereby propelling the adoption of data cleaning tools. Additionally, regulatory requirements and compliance standards also push companies to maintain high data quality standards, further driving market growth.

Another significant growth factor is the rising adoption of AI and machine learning technologies. These advanced technologies rely heavily on high-quality data to deliver accurate results. Data cleaning tools play a crucial role in preparing datasets for AI and machine learning models, ensuring that the data is free from errors, inconsistencies, and redundancies. This surge in the use of AI and machine learning across various sectors like healthcare, finance, and retail is driving the demand for efficient data cleaning solutions.

The proliferation of big data analytics is another critical factor contributing to market growth. Big data analytics enables organizations to uncover hidden patterns, correlations, and insights from large datasets. However, the effectiveness of big data analytics is contingent upon the quality of the data being analyzed. Data cleaning tools help in sanitizing large datasets, making them suitable for analysis and thus enhancing the accuracy and reliability of analytics outcomes. This trend is expected to continue, fueling the demand for data cleaning tools.

In terms of regional growth, North America holds a dominant position in the data cleaning tools market. The region's strong technological infrastructure, coupled with the presence of major market players and a high adoption rate of advanced data management solutions, contributes to its leadership. However, the Asia Pacific region is anticipated to witness the highest growth rate during the forecast period. The rapid digitization of businesses, increasing investments in IT infrastructure, and a growing focus on data-driven decision-making are key factors driving the market in this region.

As organizations strive to maintain high data quality standards, the role of an Email List Cleaning Service becomes increasingly vital. These services ensure that email databases are free from invalid addresses, duplicates, and outdated information, thereby enhancing the effectiveness of marketing campaigns and communications. By leveraging sophisticated algorithms and validation techniques, email list cleaning services help businesses improve their email deliverability rates and reduce the risk of being flagged as spam. This not only optimizes marketing efforts but also protects the reputation of the sender. As a result, the demand for such services is expected to grow alongside the broader data cleaning tools market, as companies recognize the importance of maintaining clean and accurate contact lists.

Component Analysis

The data cleaning tools market can be segmented by component into software and services. The software segment encompasses various tools and platforms designed for data cleaning, while the services segment includes consultancy, implementation, and maintenance services provided by vendors.

The software segment holds the largest market share and is expected to continue leading during the forecast period. This dominance can be attributed to the increasing adoption of automated data cleaning solutions that offer high efficiency and accuracy. These software solutions are equipped with advanced algorithms and functionalities that can handle large volumes of data, identify errors, and correct them without manual intervention. The rising adoption of cloud-based data cleaning software further bolsters this segment, as it offers scalability and ease of

Abstract

The Quarterly Labour Force Survey (QLFS) is a household-based sample survey conducted by Statistics South Africa (Stats SA). It collects data on the labour market activity of individuals aged 15 years or older who live in South Africa.

The objective of the QLFS is to collect quarterly information about persons in the labour market , i.e., those who are employed by sector(formal,informal,agriculture and Private households); those who are unemployed and those who are not economically active

Geographic coverage

the QLFS has national coverage

Analysis unit

individuals

Universe

Households in the nine provinces of South Africa

Kind of data

Données échantillonées [ssd]

Sampling procedure

The Quarterly Labour Force Survey (QLFS) frame has been developed as a general-purpose household survey frame that can be used by all other household surveys irrespective of the sample size requirement of the survey. The sample size for the QLFS is roughly 30 000 dwellings per quarter.

The sample is based on information collected during the 2001 Population Census conducted by Stats SA. In preparation for the 2001 Census, the country was divided into 80 787 enumeration areas (EAs). Stats SA's household-based surveys use a master sample of primary sampling units (PSUs) which comprises EAs that are drawn from across the country.

The sample is designed to be representative at provincial level and within provinces at metro/nonmetro level. Within the metros, the sample is further distributed by geography type. The four geography types are: urban formal, urban informal, farms and tribal. This implies, for example, that within a metropolitan area the sample is representative at the different geography types that may exist within that metro.

The current sample size is 3 080 PSUs. It is divided equally into four subgroups or panels called rotation groups. The rotation groups are designed in such a way that each of these groups has the same distribution pattern as that which is observed in the whole sample. They are numbered from one to four and these numbers also correspond to the quarters of the year in which the sample will be rotated for the particular group.

The sample for the redesigned Labour Force Survey (i.e. the QLFS) is based on a stratified twostage design with probability proportional to size (PPS) sampling of primary sampling units (PSUs) in the first stage, and sampling of dwelling units (DUs) with systematic sampling in the second stage.

Sample rotation Each quarter, a ¼ of the sampled dwellings rotate out of the sample and are replaced by new dwellings from the same PSU or the next PSU on the list. Thus, sampled dwellings will remain in the sample for four consecutive quarters. It should be noted that the sampling unit is the dwelling, and the unit of observation is the household. Therefore, if a household moves out of a dwelling after being in the sample for, say two quarters and a new household moves in then the new household will be enumerated for the next two quarters. If no household moves into the sampled dwelling, the dwelling will be classified as vacant (unoccupied).

Mode of data collection

Interview face à face [f2f]

Research instrument

the questionnaire is composed by 5 sections: - Section1, Biographical information (marital status, language, migration, education, training, literacy, etc.)
- Section2, Economic activities in the last week : The questions in this section determine those individuals, aged 15-64 years, who are employed and those who are not employed.
- Section 3, Unemployment and economic inactivity : This section determines which respondents are unemployed and which respondents are not economically active. - Section 4, Main work activities in the last week : This section contains questions about the work situation of respondents who are employed. It includes questions about the number of jobs at which the respondent works, the hours of work, the industry and occupation of the respondent as well as whether or not the person is employed in the formal or informal sector etc., - Section 5 covers earnings in the main job for employees and own-account workers aged 15 years and above.

Cleaning operations

Automated editing and imputation QLFS uses the editing and imputation module to ensure that output data is both clean and complete. There are three basic components, called functions, in the Edit and Imputation Module:

Function A: Record acceptance Function B: Edit and imputation Function C: Clean up, derived variables and preparation for weighting

Function A: Record acceptance This function is divided into three phases:

First phase: Pre-function A The first phase ensures that the records contain valid information in selected Cover Page questions required during edit and imputation and during the subsequent weighting and variance estimation. Any blanks or other errors that need to be corrected are done here before processing of the record can proceed.

Second phase: Function A record acceptance The second phase ensures that there is enough demographic and labour market activity information to ensure that editing and imputation can be successfully completed.

Third phase: Post Function A clean up This phase ensures that certain data are present where there is evidence that they should be. This for example, involves: · Ensuring that if there is written material in the job description questions then there are corresponding industry and occupation codes for them. · Ensuring that partial blanks or non-numeric characters that appear in questions where the Survey Officer is required to enter numbers are validated.
· Ensuring that where there is written material in the space provided for “Other - specify” that the corresponding option is marked.

Function B: Edit and imputation Having determined in Function A that the content of the record would support extensive editing and imputation, this function carries out those activities. Editing is the detection of errors in the captured questionnaire. Imputation is the correction of the detected errors.

Function C: Clean up, derived variables and preparation for weighting Function C includes all of the “post E&I clean up” functions such as “Off-path cleaning”, “Result Code validation”, verification of the presence of industry and occupation codes, and the generation of all derived variables.

Response rate

Response rates: first Quarter: 93.1 second quarter: 91.4 third quarter: 90.1 forth quarter: 90.7

Sampling error estimates

Because estimates are based on sample data, they differ from figures that would have been obtained from complete enumeration of the population using the same instrument. Results are subject to both sampling and non-sampling errors. Non-sampling errors include biases from inaccurate reporting, processing, and tabulation etc., as well as errors from non-response and incomplete reporting. These types of errors cannot be measured readily. However, to the extent possible, non-sampling errors can be minimised through the procedures used for data collection, editing, quality control, and non-response adjustment. The variances of the survey estimates are used to measure sampling errors. The variance estimation methodology is discussed below.

(i) Variance estimation The most commonly used methods for estimating variances of survey estimates from complex surveys, such as the QLFS, are the Taylor-series Linearization, Jackknife Replication, Balanced Repeated Replication (BRR), and Bootstrap methods (Wolter, 2007). The Fay’s BRR method has been used for variance estimation in the QLFS because of its simplicity.

(ii) Coefficient of variation It is more useful in many situations to assess the size of the standard error relative to the magnitude of the characteristic being measured (the standard error is defined as the square root of the variance). The coefficient of variation(cv) provides such a measure. It is the ratio of the standard error of the survey estimate to the value of the estimate itself expressed as a percentage. It is very useful in comparing the precision of several different survey estimates, where their sizes or scale differ from one another.

(iii) P-value If p-value <0.01 then the difference is highly significant; if p-value is between 0.01 and 0.05 then the difference is significant; and if p-value >0.05 then the difference is not significant

Many sexually selected ornaments and weapons are elaborations of an animal’s outer body surface, including long feathers, colorful skin, and rigid outgrowths. The time and energy required to keep these traits clean, attractive, and in good condition for signaling may represent an important, but understudied cost of bearing a sexually selected trait. Male fiddler crabs possess an enlarged and brightly colored claw that is used both as a weapon to fight with rival males and also as an ornament to court females. Here, we demonstrate that males benefit from grooming because females prefer males with clean claws over dirty claws, but also that the time spent grooming detracts from the amount of time available for courting females. Males therefore face a temporal tradeoff between attracting the attention of females and maintaining a clean claw. Our study provides rare evidence of the importance of grooming for mediating sexual interactions in an invertebrate, indicating that sexual selection has likely shaped the evolution of self-maintenance behaviors across a broad range of taxa.

Methods Time budgets

We conducted 10-minute focal observations on surface-active individuals (n = 50) to estimate the amount of time males spend grooming their claw. Observations were conducted between 10 am and 3 pm when crabs are most active. We measured the total amount of time a male spent grooming his major claw during the 10-minute sample using a stopwatch. Focal males were haphazardly selected from surface-active individuals on the mudflat, and we alternated between individuals initially identified in the sun and shade to account for potential differences in time budgets between microhabitats. Because crabs were not individually marked, we sampled individuals from different locations on the mudflat to avoid repeat sampling. Males that were not surface-active for the entire 10-minute focal sample (e.g., because they retreated into their burrow) were excluded from the analyses (<10% of males). At the end of the sample, we captured the focal male and measured his claw length to the nearest 0.1 mm using dial calipers.

Male-male competition

To evaluate whether males benefit from having clean claws in the context of male-male fights over burrow ownership, we staged fights between unmanipulated resident males and focal intruder males with either clean (n = 51) or dirty claws (n = 49). To stage fights, we captured males that were defending their territories and released them back into the colony. Captured males were released several meters from their burrow to avoid encounters with familiar neighbors and observed through binoculars as they wandered across the mudflat. Males were measured for claw length and then their claw was painted either with mud from the mudflat (“dirty” claw) or a clean brush (“clean” claw). To minimize handling disturbance, experimenters wore latex gloves and placed a plastic sheet between the crab’s claw and body during the painting treatment. After being released, most males took several seconds before resuming normal activity. We therefore measured a male’s fighting latency as the time elapsed between becoming active and initiating a fight with another resident male. We also recorded whether a male spent any time grooming his claw before initiating a fight (yes/no), and whether the male won or lost the fight. Males were defined as winners if they displaced the resident from his burrow and as losers if they were unable to displace the resident and continued wandering.

Female choice

We conducted female mate choice trials (n = 40) to evaluate whether males benefit from having clean claws in the context of attracting mates. We captured mate-searching females wandering on the mudflat and tested their preference for males with clean versus dirty claws using custom-built robotic crabs. Each robot consisted of a plaster claw replica (21.0 mm long) molded from a real male claw and painted with yellow enamel paint. Because natural A. mjoebergi claws reflect UV light, and females prefer UV-reflecting claws over claws that do not reflect UV light, we also painted the entire surface of the claw replicas with a UV marker. Each claw was attached to a metal arm that was driven by a small motor to mimic the species’ claw waving pattern at a rate of 16.8 waves/min. A pair of identical robots were placed in a 60 x 60 cm level test arena that was covered with a smooth layer of mudflat sediment and positioned 15 cm away from each other and 25 cm away from the choosing female. Once positioned, the claws were painted either with mud from the mudflat (“dirty” claw) or a clean brush (“clean” claw). At the start of each mate-choice trial, a female was placed under a transparent cup in front of the robotic crabs, allowed to observe the robotic crabs for two complete wave cycles, and then released from the cup using a remote trigger. Females were counted as having chosen a male if they moved directly towards and stopped at the base of one of the robotic crab units. Females that gave a startle response (i.e., dashed on release to the perimeter of the arena) or were motionless for > 3 min after release were retested a maximum of three times before being excluded from the data set and released into the population.

Temporal tradeoffs

We tested for a potential cost of grooming due to a tradeoff in the time spent maintaining a clean claw and courting females. Specifically, we conducted two-minute focal observations on males with either clean or dirty claws to measure the amount of time spent grooming versus waving. For each sample, we captured a resident male from his burrow and placed a plastic fence around the entrance to prevent other individuals from entering. Captured males were measured for claw length, painted either with mud (“dirty” claw, n = 23) or a clean brush (“clean” claw, n = 23) as described above, and then returned to their territory. We widened the burrow entrance slightly with a stick to minimize the chances of a “dirty” claw being wiped clean as the male entered or exited his burrow. To encourage the males to emerge from their burrow and engage in courtship activity, we tethered a mate-searching female to a short (~5 cm) piece of cotton thread and anchored the tether near the male’s territory. Focal males were recorded with a video camera (Sony Handycam HDR-PJ410 or DCR-SR45) and observed for two minutes following the initiation of his first wave. We measured the amount of time a male spent either grooming his major claw or waving using a stopwatch.

Abstract

Noncommunicable diseases are the top cause of deaths. In 2008, more than 36 million people worldwide died of such diseases. Ninety per cent of those lived in low-income and middle-income countries.WHO Maps Noncommunicable Disease Trends in All Countries The STEPS Noncommunicable Disease Risk Factor Survey, part of the STEPwise approach to surveillance (STEPS) Adult Risk Factor Surveillance project by the World Health Organization (WHO), is a survey methodology to help countries begin to develop their own surveillance system to monitor and fight against noncommunicable diseases. The methodology prescribes three steps—questionnaire, physical measurements, and biochemical measurements. The steps consist of core items, core variables, and optional modules. Core topics covered by most surveys are demographics, health status, and health behaviors. These provide data on socioeconomic risk factors and metabolic, nutritional, and lifestyle risk factors. Details may differ from country to country and from year to year.

The general objective of the Zimbabwe NCD STEPS survey was to assess the risk factors of selected NCDs in the adult population of Zimbabwe using the WHO STEPwise approach to non-communicable diseases surveillance. The specific objectives were: - To assess the distribution of life-style factors (physical activity, tobacco and alcohol use), and anthropometric measurements (body mass index and central obesity) which may impact on diabetes and cardiovascular risk factors. - To identify dietary practices that are risk factors for selected NCDs. - To determine the prevalence and determinants of hypertension - To determine the prevalence and determinants of diabetes. - To determine the prevalence and determinants of serum lipid profile.

Geographic coverage

Mashonaland Central, Midlands and Matebeleland South Provinces.

Analysis unit

Household Individual

Universe

The survey comprised of individuals aged 25 years and over.

Kind of data

Sample survey data [ssd]

Sampling procedure

A multistage sampling strategy with 3 stages consisting of province, district and health centre was employed. The World Health Organization STEPwise Approach (STEPS) was used as the design basis for the survey. The 3 randomly selected provinces for the survey were Mashonaland Central, Midlands and Matebeleland South. In each Province four districts were chosen and four health centres were surveyed per district. The survey comprised of individuals aged 25 years and over.The survey was carried out on 3,081 respondents consisting of 1,189 from Midlands,944 from Mashonaland Central and 948 from Matebeleland South. A detailed description of the sampling process is provided in sections 3.8 -3.9. if the survey report provided under the related materials tab.

Sampling deviation

Designing a community-based survey such as this one is fraught with difficulties in ensuring representativeness of the sample chosen. In this survey there was a preponderance of female respondents because of the pattern of employment of males and females which also influences urban rural migration.

The response rate in Midlands was lower than the other two provinces in both STEP 2 and 3. This notable difference was due to the fact that Midlands had more respondents sampled from the urban communities. A higher proportion of urban respondents was formally employed and therefore did not complete STEP 2 and 3 due to conflict with work schedules.

Mode of data collection

Face-to-face [f2f]

Research instrument

In this survey all the core and selected expanded and optional variables were collected. In addition a food frequency questionnaire and a UNICEF developed questionnaire, the Fortification Rapid Assessment Tool (FRAT) were administered to elicit relevant dietary information.

Cleaning operations

Data entry for Step 1 and Step 2 data was carried out as soon as data became available to the data management team. Step 3 data became available in October and data entry was carried out when data quality checks were completed in November. Report writing started in September and a preliminary report became available in December 2005.

Training of data entry clerks Five data entry clerks were recruited and trained for one week. The selection of data entry clerks was based on their performance during previous research carried out by the MOH&CW. The training of the data entry clerks involved the following: - Familiarization with the NCD, FRAT and FFQ questionnaires. - Familiarization with the data entry template. - Development of codes for open-ended questions. - Statistical package (EPI Info 6). - Development of a data entry template using EPI6. - Development of check files for each template - Trial runs (mock runs) to check whether template was complete and user friendly for data entry. - Double entry (what it involves and how to do it and why it should be done). - Pre-primary data cleaning (check whether denominators are tallying) of the data entry template was done.

Data Entry for NCD, FRAT and FFQ questionnaires The questionnaires were sequentially numbered and were then divided among the five data entry clerks. Each one of the data entry clerks had a unique identifier for quality control purposes. Hence, the data was entered into five separate files using the statistical package EPI Info version 6.0. The data entry clerks inter-changed their files for double entry and validation of the data. Preliminary data cleaning was done for each of the five files. The five files were then merged to give a single file. The merged file was then transferred to STATA Version 7.0 using Stat Transfer version 5.0.

Data Cleaning A data-cleaning workshop was held with the core research team members. The objectives of the workshop were: 1. To check all data entry errors. 2. To assess any inconsistencies in data filling. 3. To assess any inconsistencies in data entry. 4. To assess completeness of the data entered.

Data Merging There were two datasets (NCD questionnaire dataset and laboratory dataset) after the data entry process. The two files were merged by joining corresponding observations from the NCD questionnaire dataset with those from the laboratory dataset into single observations using a unique identifier. The ID number was chosen as the unique identifier since it appeared in both data sets. The main aim of merging was to combine the two datasets containing information on behaviour of individuals and the NCD laboratory parameters. When the two data sets were merged, a new merge variable was created. The merge variable took values 1, 2 and 3. Merge variable==1 Observation appeared in the NCD questionnaire data set but a corresponding observation was not in the laboratory data set Merge variable==2 Observation appeared in the laboratory data set but a corresponding observation did not appear in the questionnaire data set Merge variable==3 Observation appeared in both data sets and reflects a complete merge of the two data sets.

Data Cleaning After Merging Data cleaning involved identifying the observations where the merge variable values were either 1 or 2. Merge status for each observation was also changed after effecting any corrections. The other two unique variables that were used in the cleaning were Province, district and health centre since they also appeared in both data sets.

Objectives of cleaning: 1. Match common variables in both data sets and identify inconsistencies in other matching variables e.g. province, district and health centre. 2. To check for any data entry errors.

Response rate

A total of 3,081 respondents were included in the survey against an estimated sample size of 3,000. The response rate for Step 1 was 80% for and for Step 2 70% taking Step 1 accrual as being 100%.