Embark on a transformative journey with our Data Cleaning Project, where we meticulously refine and polish raw data into valuable insights. Our project focuses on streamlining data sets, removing inconsistencies, and ensuring accuracy to unlock its full potential.

Through advanced techniques and rigorous processes, we standardize formats, address missing values, and eliminate duplicates, creating a clean and reliable foundation for analysis. By enhancing data quality, we empower organizations to make informed decisions, drive innovation, and achieve strategic objectives with confidence.

Join us as we embark on this essential phase of data preparation, paving the way for more accurate and actionable insights that fuel success."

Data Cleaning Tools Market Outlook

As of 2023, the global market size for data cleaning tools is estimated at $2.5 billion, with projections indicating that it will reach approximately $7.1 billion by 2032, reflecting a robust CAGR of 12.1% during the forecast period. This growth is primarily driven by the increasing importance of data quality in business intelligence and analytics workflows across various industries.

The growth of the data cleaning tools market can be attributed to several critical factors. Firstly, the exponential increase in data generation across industries necessitates efficient tools to manage data quality. Poor data quality can result in significant financial losses, inefficient business processes, and faulty decision-making. Organizations recognize the value of clean, accurate data in driving business insights and operational efficiency, thereby propelling the adoption of data cleaning tools. Additionally, regulatory requirements and compliance standards also push companies to maintain high data quality standards, further driving market growth.

Another significant growth factor is the rising adoption of AI and machine learning technologies. These advanced technologies rely heavily on high-quality data to deliver accurate results. Data cleaning tools play a crucial role in preparing datasets for AI and machine learning models, ensuring that the data is free from errors, inconsistencies, and redundancies. This surge in the use of AI and machine learning across various sectors like healthcare, finance, and retail is driving the demand for efficient data cleaning solutions.

The proliferation of big data analytics is another critical factor contributing to market growth. Big data analytics enables organizations to uncover hidden patterns, correlations, and insights from large datasets. However, the effectiveness of big data analytics is contingent upon the quality of the data being analyzed. Data cleaning tools help in sanitizing large datasets, making them suitable for analysis and thus enhancing the accuracy and reliability of analytics outcomes. This trend is expected to continue, fueling the demand for data cleaning tools.

In terms of regional growth, North America holds a dominant position in the data cleaning tools market. The region's strong technological infrastructure, coupled with the presence of major market players and a high adoption rate of advanced data management solutions, contributes to its leadership. However, the Asia Pacific region is anticipated to witness the highest growth rate during the forecast period. The rapid digitization of businesses, increasing investments in IT infrastructure, and a growing focus on data-driven decision-making are key factors driving the market in this region.

As organizations strive to maintain high data quality standards, the role of an Email List Cleaning Service becomes increasingly vital. These services ensure that email databases are free from invalid addresses, duplicates, and outdated information, thereby enhancing the effectiveness of marketing campaigns and communications. By leveraging sophisticated algorithms and validation techniques, email list cleaning services help businesses improve their email deliverability rates and reduce the risk of being flagged as spam. This not only optimizes marketing efforts but also protects the reputation of the sender. As a result, the demand for such services is expected to grow alongside the broader data cleaning tools market, as companies recognize the importance of maintaining clean and accurate contact lists.

Component Analysis

The data cleaning tools market can be segmented by component into software and services. The software segment encompasses various tools and platforms designed for data cleaning, while the services segment includes consultancy, implementation, and maintenance services provided by vendors.

The software segment holds the largest market share and is expected to continue leading during the forecast period. This dominance can be attributed to the increasing adoption of automated data cleaning solutions that offer high efficiency and accuracy. These software solutions are equipped with advanced algorithms and functionalities that can handle large volumes of data, identify errors, and correct them without manual intervention. The rising adoption of cloud-based data cleaning software further bolsters this segment, as it offers scalability and ease of

The mean, standard deviation, preservation of data (PD), sensitivity and specificity of five data cleaning approaches with and without an algorithm (A) compared to uncleaned longitudinal growth measurements in CLOSER data with and without simulated duplications and 1% errors.

Data Cleansing Software Market Outlook

The global data cleansing software market size was valued at approximately USD 1.5 billion in 2023 and is projected to reach around USD 4.2 billion by 2032, exhibiting a compound annual growth rate (CAGR) of 12.5% during the forecast period. This substantial growth can be attributed to the increasing importance of maintaining clean and reliable data for business intelligence and analytics, which are driving the adoption of data cleansing solutions across various industries.

The proliferation of big data and the growing emphasis on data-driven decision-making are significant growth factors for the data cleansing software market. As organizations collect vast amounts of data from multiple sources, ensuring that this data is accurate, consistent, and complete becomes critical for deriving actionable insights. Data cleansing software helps organizations eliminate inaccuracies, inconsistencies, and redundancies, thereby enhancing the quality of their data and improving overall operational efficiency. Additionally, the rising adoption of advanced analytics and artificial intelligence (AI) technologies further fuels the demand for data cleansing software, as clean data is essential for the accuracy and reliability of these technologies.

Another key driver of market growth is the increasing regulatory pressure for data compliance and governance. Governments and regulatory bodies across the globe are implementing stringent data protection regulations, such as the General Data Protection Regulation (GDPR) in Europe and the California Consumer Privacy Act (CCPA) in the United States. These regulations mandate organizations to ensure the accuracy and security of the personal data they handle. Data cleansing software assists organizations in complying with these regulations by identifying and rectifying inaccuracies in their data repositories, thus minimizing the risk of non-compliance and hefty penalties.

The growing trend of digital transformation across various industries also contributes to the expanding data cleansing software market. As businesses transition to digital platforms, they generate and accumulate enormous volumes of data. To derive meaningful insights and maintain a competitive edge, it is imperative for organizations to maintain high-quality data. Data cleansing software plays a pivotal role in this process by enabling organizations to streamline their data management practices and ensure the integrity of their data. Furthermore, the increasing adoption of cloud-based solutions provides additional impetus to the market, as cloud platforms facilitate seamless integration and scalability of data cleansing tools.

Regionally, North America holds a dominant position in the data cleansing software market, driven by the presence of numerous technology giants and the rapid adoption of advanced data management solutions. The region is expected to continue its dominance during the forecast period, supported by the strong emphasis on data quality and compliance. Europe is also a significant market, with countries like Germany, the UK, and France showing substantial demand for data cleansing solutions. The Asia Pacific region is poised for significant growth, fueled by the increasing digitalization of businesses and the rising awareness of data quality's importance. Emerging economies in Latin America and the Middle East & Africa are also expected to witness steady growth, driven by the growing adoption of data-driven technologies.

The role of Data Quality Tools cannot be overstated in the context of data cleansing software. These tools are integral in ensuring that the data being processed is not only clean but also of high quality, which is crucial for accurate analytics and decision-making. Data Quality Tools help in profiling, monitoring, and cleansing data, thereby ensuring that organizations can trust their data for strategic decisions. As organizations increasingly rely on data-driven insights, the demand for robust Data Quality Tools is expected to rise. These tools offer functionalities such as data validation, standardization, and enrichment, which are essential for maintaining the integrity of data across various platforms and applications. The integration of these tools with data cleansing software enhances the overall data management capabilities of organizations, enabling them to achieve greater operational efficiency and compliance with data regulations.

Component Analysis

The data cle

Restaurant Sales Dataset with Dirt Documentation

Overview

The Restaurant Sales Dataset with Dirt contains data for 17,534 transactions. The data introduces realistic inconsistencies ("dirt") to simulate real-world scenarios where data may have missing or incomplete information. The dataset includes sales details across multiple categories, such as starters, main dishes, desserts, drinks, and side dishes.

Dataset Use Cases

This dataset is suitable for: - Practicing data cleaning tasks, such as handling missing values and deducing missing information. - Conducting exploratory data analysis (EDA) to study restaurant sales patterns. - Feature engineering to create new variables for machine learning tasks.

Columns Description

Key Characteristics

1. Data Dirtiness:

   • Missing values in key columns (Item, Price, Quantity, Order Total, Payment Method) simulate real-world challenges.
   • At least one of the following conditions is ensured for each record to identify an item:
     • Item is present.
     • Price is present.
     • Both Quantity and Order Total are present.
   • If Price or Quantity is missing, the other is used to deduce the missing value (e.g., Order Total / Quantity).

2. Menu Categories and Items:

   • Items are divided into five categories:
     • Starters: E.g., Chicken Melt, French Fries.
     • Main Dishes: E.g., Grilled Chicken, Steak.
     • Desserts: E.g., Chocolate Cake, Ice Cream.
     • Drinks: E.g., Coca Cola, Water.
     • Side Dishes: E.g., Mashed Potatoes, Garlic Bread.

3 Time Range: - Orders span from January 1, 2022, to December 31, 2023.

Cleaning Suggestions

1. Handle Missing Values:

   • Fill missing Order Total or Quantity using the formula: Order Total = Price * Quantity.
   • Deduce missing Price from Order Total / Quantity if both are available.

2. Validate Data Consistency:

   • Ensure that calculated values (Order Total = Price * Quantity) match.

3. Analyze Missing Patterns:

   • Study the distribution of missing values across categories and payment methods.

Menu Map with Prices and Categories

Sample data for exercises in Further Adventures in Data Cleaning.

The percentage of gold standard corrections of errors induced into CLOSER data with simulated duplications and 1% errors using the algorithmic data cleaning methods.

This dataset is about book subjects. It has 3 rows and is filtered where the books is Data cleaning and exploration with machine learning : clean data with machine learning algorithms and techniques. It features 10 columns including number of authors, number of books, earliest publication date, and latest publication date.

Description of the study design, data collection and processing, cohort details and data accessibility for longitudinal height or weight measurements in Dogslife, SAVSNET, Banfield and CLOSER datasets.

The data cleansing software market is experiencing robust growth, driven by the escalating volume and complexity of data generated across various industries. The increasing need for accurate and reliable data for informed decision-making, coupled with stringent data privacy regulations like GDPR and CCPA, is fueling the demand for sophisticated data cleansing solutions. Businesses are increasingly adopting cloud-based solutions due to their scalability, cost-effectiveness, and ease of integration with existing systems. The market is segmented by deployment mode (cloud, on-premise), organization size (small, medium, large), and industry vertical (BFSI, healthcare, retail, etc.). While precise market sizing data is unavailable, considering the presence of major players like IBM, SAS, and SAP, and a projected CAGR (let's assume a conservative 15% based on industry trends), we can estimate the 2025 market size to be around $2 billion (USD) with the potential to exceed $5 billion by 2033. This growth trajectory is supported by the continuous innovation in data cleansing techniques, including AI and machine learning integration, enhancing the speed, accuracy, and automation capabilities of these solutions. Despite the promising outlook, the market faces certain challenges. High initial investment costs for implementing data cleansing solutions can be a barrier for smaller organizations. Furthermore, the lack of skilled professionals proficient in data management and cleansing can hinder widespread adoption. The market’s competitive landscape is characterized by both established players offering comprehensive solutions and smaller niche players focusing on specific functionalities or industries. The success of players in this market hinges on their ability to offer scalable, user-friendly, and highly accurate data cleansing solutions tailored to the specific needs of diverse customer segments, while continually adapting to evolving data formats and regulatory environments. The ongoing development of AI-powered automation within these platforms will prove a key differentiator in the years to come.

OpenRefine (formerly Google Refine) is a powerful free and open source tool for data cleaning, enabling you to correct errors in the data, and make sure that the values and formatting are consistent. In addition, OpenRefine records your processing steps, enabling you to apply the same cleaning procedure to other data, and enhancing the reproducibility of your analysis. This workshop will teach you to use OpenRefine to clean and format data and automatically track any changes that you make.

Snapshot img

Data Science Platform Market Size 2025-2029

The data science platform market size is forecast to increase by USD 763.9 million, at a CAGR of 40.2% between 2024 and 2029.

The market is experiencing significant growth, driven by the increasing integration of Artificial Intelligence (AI) and Machine Learning (ML) technologies. This fusion enables organizations to derive deeper insights from their data, fueling business innovation and decision-making. Another trend shaping the market is the emergence of containerization and microservices in data science platforms. This approach offers enhanced flexibility, scalability, and efficiency, making it an attractive choice for businesses seeking to streamline their data science operations. However, the market also faces challenges. Data privacy and security remain critical concerns, with the increasing volume and complexity of data posing significant risks. Ensuring robust data security and privacy measures is essential for companies to maintain customer trust and comply with regulatory requirements. Additionally, managing the complexity of data science platforms and ensuring seamless integration with existing systems can be a daunting task, requiring significant investment in resources and expertise. Companies must navigate these challenges effectively to capitalize on the market's opportunities and stay competitive in the rapidly evolving data landscape.

What will be the Size of the Data Science Platform Market during the forecast period?

Explore in-depth regional segment analysis with market size data - historical 2019-2023 and forecasts 2025-2029 - in the full report.
Request Free SampleThe market continues to evolve, driven by the increasing demand for advanced analytics and artificial intelligence solutions across various sectors. Real-time analytics and classification models are at the forefront of this evolution, with APIs integrations enabling seamless implementation. Deep learning and model deployment are crucial components, powering applications such as fraud detection and customer segmentation. Data science platforms provide essential tools for data cleaning and data transformation, ensuring data integrity for big data analytics. Feature engineering and data visualization facilitate model training and evaluation, while data security and data governance ensure data privacy and compliance. Machine learning algorithms, including regression models and clustering models, are integral to predictive modeling and anomaly detection. Statistical analysis and time series analysis provide valuable insights, while ETL processes streamline data integration. Cloud computing enables scalability and cost savings, while risk management and algorithm selection optimize model performance. Natural language processing and sentiment analysis offer new opportunities for data storytelling and computer vision. Supply chain optimization and recommendation engines are among the latest applications of data science platforms, demonstrating their versatility and continuous value proposition. Data mining and data warehousing provide the foundation for these advanced analytics capabilities.

How is this Data Science Platform Industry segmented?

The data science platform industry research report provides comprehensive data (region-wise segment analysis), with forecasts and estimates in 'USD million' for the period 2025-2029, as well as historical data from 2019-2023 for the following segments. DeploymentOn-premisesCloudComponentPlatformServicesEnd-userBFSIRetail and e-commerceManufacturingMedia and entertainmentOthersSectorLarge enterprisesSMEsApplicationData PreparationData VisualizationMachine LearningPredictive AnalyticsData GovernanceOthersGeographyNorth AmericaUSCanadaEuropeFranceGermanyUKMiddle East and AfricaUAEAPACChinaIndiaJapanSouth AmericaBrazilRest of World (ROW)

By Deployment Insights

The on-premises segment is estimated to witness significant growth during the forecast period.In the dynamic the market, businesses increasingly adopt solutions to gain real-time insights from their data, enabling them to make informed decisions. Classification models and deep learning algorithms are integral parts of these platforms, providing capabilities for fraud detection, customer segmentation, and predictive modeling. API integrations facilitate seamless data exchange between systems, while data security measures ensure the protection of valuable business information. Big data analytics and feature engineering are essential for deriving meaningful insights from vast datasets. Data transformation, data mining, and statistical analysis are crucial processes in data preparation and discovery. Machine learning models, including regression and clustering, are employed for model training and evaluation. Time series analysis and natural language processing are valuable tools for understanding trends and customer sen

Data Cleansing Tools Market Outlook

The global data cleansing tools market size was valued at approximately USD 1.5 billion in 2023 and is projected to reach USD 4.2 billion by 2032, growing at a CAGR of 12.1% from 2024 to 2032. One of the primary growth factors driving the market is the increasing need for high-quality data in various business operations and decision-making processes.

The surge in big data and the subsequent increased reliance on data analytics are significant factors propelling the growth of the data cleansing tools market. Organizations increasingly recognize the value of high-quality data in driving strategic initiatives, customer relationship management, and operational efficiency. The proliferation of data generated across different sectors such as healthcare, finance, retail, and telecommunications necessitates the adoption of tools that can clean, standardize, and enrich data to ensure its reliability and accuracy.

Furthermore, the rising adoption of Machine Learning (ML) and Artificial Intelligence (AI) technologies has underscored the importance of clean data. These technologies rely heavily on large datasets to provide accurate and reliable insights. Any errors or inconsistencies in data can lead to erroneous outcomes, making data cleansing tools indispensable. Additionally, regulatory and compliance requirements across various industries necessitate the maintenance of clean and accurate data, further driving the market for data cleansing tools.

The growing trend of digital transformation across industries is another critical growth factor. As businesses increasingly transition from traditional methods to digital platforms, the volume of data generated has skyrocketed. However, this data often comes from disparate sources and in various formats, leading to inconsistencies and errors. Data cleansing tools are essential in such scenarios to integrate data from multiple sources and ensure its quality, thus enabling organizations to derive actionable insights and maintain a competitive edge.

In the context of ensuring data reliability and accuracy, Data Quality Software and Solutions play a pivotal role. These solutions are designed to address the challenges associated with managing large volumes of data from diverse sources. By implementing robust data quality frameworks, organizations can enhance their data governance strategies, ensuring that data is not only clean but also consistent and compliant with industry standards. This is particularly crucial in sectors where data-driven decision-making is integral to business success, such as finance and healthcare. The integration of advanced data quality solutions helps businesses mitigate risks associated with poor data quality, thereby enhancing operational efficiency and strategic planning.

Regionally, North America is expected to hold the largest market share due to the early adoption of advanced technologies, robust IT infrastructure, and the presence of key market players. Europe is also anticipated to witness substantial growth due to stringent data protection regulations and the increasing adoption of data-driven decision-making processes. Meanwhile, the Asia Pacific region is projected to experience the highest growth rate, driven by the rapid digitalization of emerging economies, the expansion of the IT and telecommunications sector, and increasing investments in data management solutions.

Component Analysis

The data cleansing tools market is segmented into software and services based on components. The software segment is anticipated to dominate the market due to its extensive use in automating the data cleansing process. The software solutions are designed to identify, rectify, and remove errors in data sets, ensuring data accuracy and consistency. They offer various functionalities such as data profiling, validation, enrichment, and standardization, which are critical in maintaining high data quality. The high demand for these functionalities across various industries is driving the growth of the software segment.

On the other hand, the services segment, which includes professional services and managed services, is also expected to witness significant growth. Professional services such as consulting, implementation, and training are crucial for organizations to effectively deploy and utilize data cleansing tools. As businesses increasingly realize the importance of clean data, the demand for expert

efficacy data against MHV for cleaning surfaces. This dataset is associated with the following publication: Hardison, R., S. Nelson, D. Barriga, J. Ghere, G. Fenton, R. James, M. Stewart, S. Lee, M.W. Calfee, S. Ryan, and M. Howard. Efficacy of Detergent-Based Cleaning Methods Against Coronavirus MHV-A59 on Porous and Non-Porous Surfaces. JOURNAL OF OCCUPATIONAL AND ENVIRONMENTAL HYGIENE. Taylor & Francis, Inc., Philadelphia, PA, USA, 19(2): 91-101, (2022).

Description of the data entries, individuals, data entries per individual, mean and standard deviation of the longitudinal height or weight measurements in Dogslife, SAVSNET, Banfield and CLOSER data with and without simulated duplications and 1% errors before and after removal of duplicated measurement records.

This is a demonstration of the outlier boundary set up across different ML data cleaning techniques.

The LSC (Leicester Scientific Corpus)

April 2020 by Neslihan Suzen, PhD student at the University of Leicester (ns433@leicester.ac.uk) Supervised by Prof Alexander Gorban and Dr Evgeny MirkesThe data are extracted from the Web of Science [1]. You may not copy or distribute these data in whole or in part without the written consent of Clarivate Analytics.[Version 2] A further cleaning is applied in Data Processing for LSC Abstracts in Version 1*. Details of cleaning procedure are explained in Step 6.* Suzen, Neslihan (2019): LSC (Leicester Scientific Corpus). figshare. Dataset. https://doi.org/10.25392/leicester.data.9449639.v1.Getting StartedThis text provides the information on the LSC (Leicester Scientific Corpus) and pre-processing steps on abstracts, and describes the structure of files to organise the corpus. This corpus is created to be used in future work on the quantification of the meaning of research texts and make it available for use in Natural Language Processing projects.LSC is a collection of abstracts of articles and proceeding papers published in 2014, and indexed by the Web of Science (WoS) database [1]. The corpus contains only documents in English. Each document in the corpus contains the following parts:1. Authors: The list of authors of the paper2. Title: The title of the paper 3. Abstract: The abstract of the paper 4. Categories: One or more category from the list of categories [2]. Full list of categories is presented in file ‘List_of _Categories.txt’. 5. Research Areas: One or more research area from the list of research areas [3]. Full list of research areas is presented in file ‘List_of_Research_Areas.txt’. 6. Total Times cited: The number of times the paper was cited by other items from all databases within Web of Science platform [4] 7. Times cited in Core Collection: The total number of times the paper was cited by other papers within the WoS Core Collection [4]The corpus was collected in July 2018 online and contains the number of citations from publication date to July 2018. We describe a document as the collection of information (about a paper) listed above. The total number of documents in LSC is 1,673,350.Data ProcessingStep 1: Downloading of the Data Online

The dataset is collected manually by exporting documents as Tab-delimitated files online. All documents are available online.Step 2: Importing the Dataset to R

The LSC was collected as TXT files. All documents are extracted to R.Step 3: Cleaning the Data from Documents with Empty Abstract or without CategoryAs our research is based on the analysis of abstracts and categories, all documents with empty abstracts and documents without categories are removed.Step 4: Identification and Correction of Concatenate Words in AbstractsEspecially medicine-related publications use ‘structured abstracts’. Such type of abstracts are divided into sections with distinct headings such as introduction, aim, objective, method, result, conclusion etc. Used tool for extracting abstracts leads concatenate words of section headings with the first word of the section. For instance, we observe words such as ConclusionHigher and ConclusionsRT etc. The detection and identification of such words is done by sampling of medicine-related publications with human intervention. Detected concatenate words are split into two words. For instance, the word ‘ConclusionHigher’ is split into ‘Conclusion’ and ‘Higher’.The section headings in such abstracts are listed below:

Background Method(s) Design Theoretical Measurement(s) Location Aim(s) Methodology Process Abstract Population Approach Objective(s) Purpose(s) Subject(s) Introduction Implication(s) Patient(s) Procedure(s) Hypothesis Measure(s) Setting(s) Limitation(s) Discussion Conclusion(s) Result(s) Finding(s) Material (s) Rationale(s) Implications for health and nursing policyStep 5: Extracting (Sub-setting) the Data Based on Lengths of AbstractsAfter correction, the lengths of abstracts are calculated. ‘Length’ indicates the total number of words in the text, calculated by the same rule as for Microsoft Word ‘word count’ [5].According to APA style manual [6], an abstract should contain between 150 to 250 words. In LSC, we decided to limit length of abstracts from 30 to 500 words in order to study documents with abstracts of typical length ranges and to avoid the effect of the length to the analysis.

Step 6: [Version 2] Cleaning Copyright Notices, Permission polices, Journal Names and Conference Names from LSC Abstracts in Version 1Publications can include a footer of copyright notice, permission policy, journal name, licence, author’s right or conference name below the text of abstract by conferences and journals. Used tool for extracting and processing abstracts in WoS database leads to attached such footers to the text. For example, our casual observation yields that copyright notices such as ‘Published by Elsevier ltd.’ is placed in many texts. To avoid abnormal appearances of words in further analysis of words such as bias in frequency calculation, we performed a cleaning procedure on such sentences and phrases in abstracts of LSC version 1. We removed copyright notices, names of conferences, names of journals, authors’ rights, licenses and permission policies identified by sampling of abstracts.Step 7: [Version 2] Re-extracting (Sub-setting) the Data Based on Lengths of AbstractsThe cleaning procedure described in previous step leaded to some abstracts having less than our minimum length criteria (30 words). 474 texts were removed.Step 8: Saving the Dataset into CSV FormatDocuments are saved into 34 CSV files. In CSV files, the information is organised with one record on each line and parts of abstract, title, list of authors, list of categories, list of research areas, and times cited is recorded in fields.To access the LSC for research purposes, please email to ns433@le.ac.uk.References[1]Web of Science. (15 July). Available: https://apps.webofknowledge.com/ [2]WoS Subject Categories. Available: https://images.webofknowledge.com/WOKRS56B5/help/WOS/hp_subject_category_terms_tasca.html [3]Research Areas in WoS. Available: https://images.webofknowledge.com/images/help/WOS/hp_research_areas_easca.html [4]Times Cited in WoS Core Collection. (15 July). Available: https://support.clarivate.com/ScientificandAcademicResearch/s/article/Web-of-Science-Times-Cited-accessibility-and-variation?language=en_US [5]Word Count. Available: https://support.office.com/en-us/article/show-word-count-3c9e6a11-a04d-43b4-977c-563a0e0d5da3 [6]A. P. Association, Publication manual. American Psychological Association Washington, DC, 1983.

Abstract

The Palestinian society's access to information and communication technology tools is one of the main inputs to achieve social development and economic change to the status of Palestinian society; on the basis of its impact on the revolution of information and communications technology that has become a feature of this era. Therefore, and within the scope of the efforts exerted by the Palestinian Central Bureau of Statistics in providing official Palestinian statistics on various areas of life for the Palestinian community, PCBS implemented the household survey for information and communications technology for the year 2019. The main objective of this report is to present the trends of accessing and using information and communication technology by households and individuals in Palestine, and enriching the information and communications technology database with indicators that meet national needs and are in line with international recommendations.

Geographic coverage

Palestine, West Bank, Gaza strip

Analysis unit

Household, Individual

Universe

All Palestinian households and individuals (10 years and above) whose usual place of residence in 2019 was in the state of Palestine.

Kind of data

Sample survey data [ssd]

Sampling procedure

Sampling Frame The sampling frame consists of master sample which were enumerated in the 2017 census. Each enumeration area consists of buildings and housing units with an average of about 150 households. These enumeration areas are used as primary sampling units (PSUs) in the first stage of the sampling selection.

Sample size The estimated sample size is 8,040 households.

Sample Design The sample is three stages stratified cluster (pps) sample. The design comprised three stages: Stage (1): Selection a stratified sample of 536 enumeration areas with (pps) method. Stage (2): Selection a stratified random sample of 15 households from each enumeration area selected in the first stage. Stage (3): Selection one person of the (10 years and above) age group in a random method by using KISH TABLES.

Sample Strata The population was divided by: 1- Governorate (16 governorates, where Jerusalem was considered as two statistical areas) 2- Type of Locality (urban, rural, refugee camps).

Mode of data collection

Computer Assisted Personal Interview [capi]

Research instrument

Questionnaire The survey questionnaire consists of identification data, quality controls and three main sections: Section I: Data on household members that include identification fields, the characteristics of household members (demographic and social) such as the relationship of individuals to the head of household, sex, date of birth and age.

Section II: Household data include information regarding computer processing, access to the Internet, and possession of various media and computer equipment. This section includes information on topics related to the use of computer and Internet, as well as supervision by households of their children (5-17 years old) while using the computer and Internet, and protective measures taken by the household in the home.

Section III: Data on Individuals (10 years and over) about computer use, access to the Internet and possession of a mobile phone.

Cleaning operations

Programming Consistency Check The data collection program was designed in accordance with the questionnaire's design and its skips. The program was examined more than once before the conducting of the training course by the project management where the notes and modifications were reflected on the program by the Data Processing Department after ensuring that it was free of errors before going to the field.

Using PC-tablet devices reduced data processing stages, and fieldworkers collected data and sent it directly to server, and project management withdraw the data at any time.

In order to work in parallel with Jerusalem (J1), a data entry program was developed using the same technology and using the same database used for PC-tablet devices.

Data Cleaning After the completion of data entry and audit phase, data is cleaned by conducting internal tests for the outlier answers and comprehensive audit rules through using SPSS program to extract and modify errors and discrepancies to prepare clean and accurate data ready for tabulation and publishing.

Tabulation After finalizing checking and cleaning data from any errors. Tables extracted according to prepared list of tables.

Response rate

The response rate in the West Bank reached 77.6% while in the Gaza Strip it reached 92.7%.

Sampling error estimates

Sampling Errors Data of this survey affected by sampling errors due to use of the sample and not a complete enumeration. Therefore, certain differences are expected in comparison with the real values obtained through censuses. Variance were calculated for the most important indicators, There is no problem to disseminate results at the national level and at the level of the West Bank and Gaza Strip.

Non-Sampling Errors Non-Sampling errors are possible at all stages of the project, during data collection or processing. These are referred to non-response errors, response errors, interviewing errors and data entry errors. To avoid errors and reduce their effects, strenuous efforts were made to train the field workers intensively. They were trained on how to carry out the interview, what to discuss and what to avoid, as well as practical and theoretical training during the training course.

The implementation of the survey encountered non-response where the case (household was not present at home) during the fieldwork visit become the high percentage of the non response cases. The total non-response rate reached 17.5%. The refusal percentage reached 2.9% which is relatively low percentage compared to the household surveys conducted by PCBS, and the reason is the questionnaire survey is clear.

Dataset used in the publication "Evaluation of chemical free cleaning techniques for RED fed with natural waters and stacks with profiled membranes". This dataset contains data collected during experiment for cleaning techniques in reverse electrodialysis (RED) using natural waters. For explanation of the experimental setup we refer you to the published paper. It is being made public both to act as supplementary data for publication and in order for other researchers to use this data in their own work.

The datasets presented here were partially used in “Formulation and MIP-heuristics for the lot sizing and scheduling problem with temporal cleanings” (Toscano, A., Ferreira, D. , Morabito, R. , Computers & Chemical Engineering) [1], in “A decomposition heuristic to solve the two-stage lot sizing and scheduling problem with temporal cleaning” (Toscano, A., Ferreira, D. , Morabito, R. , Flexible Services and Manufacturing Journal) [2], and in “A heuristic approach to optimize the production scheduling of fruit-based beverages” (Toscano et al., Gestão & Produção, 2020) [3]. In fruit-based production processes, there are two production stages: preparation tanks and production lines. This production process has some process-specific characteristics, such as temporal cleanings and synchrony between the two production stages, which make optimized production planning and scheduling even more difficult. In this sense, some papers in the literature have proposed different methods to solve this problem. To the best of our knowledge, there are no standard datasets used by researchers in the literature in order to verify the accuracy and performance of proposed methods or to be a benchmark for other researchers considering this problem. The authors have been using small data sets that do not satisfactorily represent different scenarios of production. Since the demand in the beverage sector is seasonal, a wide range of scenarios enables us to evaluate the effectiveness of the proposed methods in the scientific literature in solving real scenarios of the problem. The datasets presented here include data based on real data collected from five beverage companies. We presented four datasets that are specifically constructed assuming a scenario of restricted capacity and balanced costs. These dataset is supplementary data for the submitted paper to Data in Brief [4]. [1] Toscano, A., Ferreira, D., Morabito, R., Formulation and MIP-heuristics for the lot sizing and scheduling problem with temporal cleanings, Computers & Chemical Engineering. 142 (2020) 107038. Doi: 10.1016/j.compchemeng.2020.107038. [2] Toscano, A., Ferreira, D., Morabito, R., A decomposition heuristic to solve the two-stage lot sizing and scheduling problem with temporal cleaning, Flexible Services and Manufacturing Journal. 31 (2019) 142-173. Doi: 10.1007/s10696-017-9303-9. [3] Toscano, A., Ferreira, D., Morabito, R., Trassi, M. V. C., A heuristic approach to optimize the production scheduling of fruit-based beverages. Gestão & Produção, 27(4), e4869, 2020. https://doi.org/10.1590/0104-530X4869-20. [4] Piñeros, J., Toscano, A., Ferreira, D., Morabito, R., Datasets for lot sizing and scheduling problems in the fruit-based beverage production process. Data in Brief (2021).