At Thomson Data, we help businesses clean up and manage messy B2B databases to ensure they are up-to-date, correct, and detailed. We believe your sales development representatives and marketing representatives should focus on building meaningful relationships with prospects, not scrubbing through bad data.

Here are the key steps involved in our B2B data cleansing process:

1. Data Auditing: We begin with a thorough audit of the database to identify errors, gaps, and inconsistencies, which majorly revolve around identifying outdated, incomplete, and duplicate information.

2. Data Standardization: Ensuring consistency in the data records is one of our prime services; it includes standardizing job titles, addresses, and company names. It ensures that they can be easily shared and used by different teams.

3. Data Deduplication: Another way we improve efficiency is by removing all duplicate records. Data deduplication is important in a large B2B dataset as multiple records from the same company may exist in the database.

4. Data Enrichment: After the first three steps, we enrich your data, fill in the missing details, and then enhance the database with up-to-date records. This is the step that ensures the database is valuable, providing insights that are actionable and complete.

What are the Key Benefits of Keeping the Data Clean with Thomson Data’s B2B Data Cleansing Service? Once you understand the benefits of our data cleansing service, it will entice you to optimize your data management practices, and it will additionally help you stay competitive in today’s data-driven market.

Here are some advantages of maintaining a clean database with Thomson Data:

1. Better ROI for your Sales and Marketing Campaigns: Our clean data will magnify your precise targeting, enabling you to strategize for effective campaigns, increased conversion rate, and ROI.

2. Compliant with Data Regulations:
   The B2B data cleansing services we provide are compliant to global data norms.

3. Streamline Operations: Your efforts are directed in the right channel when your data is clean and accurate, as your team doesn’t have to spend their valuable time fixing errors.

To summarize, we would again bring your attention to how accurate data is essential for driving sales and marketing in a B2B environment. It enhances your business prowess in the avenues of decision-making and customer relationships. Therefore, it is better to have a proactive approach toward B2B data cleansing service and outsource our offerings to stay competitive by unlocking the full potential of your data.

Send us a request and we will be happy to assist you.

Data Cleaning Tools Market Outlook

As of 2023, the global market size for data cleaning tools is estimated at $2.5 billion, with projections indicating that it will reach approximately $7.1 billion by 2032, reflecting a robust CAGR of 12.1% during the forecast period. This growth is primarily driven by the increasing importance of data quality in business intelligence and analytics workflows across various industries.

The growth of the data cleaning tools market can be attributed to several critical factors. Firstly, the exponential increase in data generation across industries necessitates efficient tools to manage data quality. Poor data quality can result in significant financial losses, inefficient business processes, and faulty decision-making. Organizations recognize the value of clean, accurate data in driving business insights and operational efficiency, thereby propelling the adoption of data cleaning tools. Additionally, regulatory requirements and compliance standards also push companies to maintain high data quality standards, further driving market growth.

Another significant growth factor is the rising adoption of AI and machine learning technologies. These advanced technologies rely heavily on high-quality data to deliver accurate results. Data cleaning tools play a crucial role in preparing datasets for AI and machine learning models, ensuring that the data is free from errors, inconsistencies, and redundancies. This surge in the use of AI and machine learning across various sectors like healthcare, finance, and retail is driving the demand for efficient data cleaning solutions.

The proliferation of big data analytics is another critical factor contributing to market growth. Big data analytics enables organizations to uncover hidden patterns, correlations, and insights from large datasets. However, the effectiveness of big data analytics is contingent upon the quality of the data being analyzed. Data cleaning tools help in sanitizing large datasets, making them suitable for analysis and thus enhancing the accuracy and reliability of analytics outcomes. This trend is expected to continue, fueling the demand for data cleaning tools.

In terms of regional growth, North America holds a dominant position in the data cleaning tools market. The region's strong technological infrastructure, coupled with the presence of major market players and a high adoption rate of advanced data management solutions, contributes to its leadership. However, the Asia Pacific region is anticipated to witness the highest growth rate during the forecast period. The rapid digitization of businesses, increasing investments in IT infrastructure, and a growing focus on data-driven decision-making are key factors driving the market in this region.

As organizations strive to maintain high data quality standards, the role of an Email List Cleaning Service becomes increasingly vital. These services ensure that email databases are free from invalid addresses, duplicates, and outdated information, thereby enhancing the effectiveness of marketing campaigns and communications. By leveraging sophisticated algorithms and validation techniques, email list cleaning services help businesses improve their email deliverability rates and reduce the risk of being flagged as spam. This not only optimizes marketing efforts but also protects the reputation of the sender. As a result, the demand for such services is expected to grow alongside the broader data cleaning tools market, as companies recognize the importance of maintaining clean and accurate contact lists.

Component Analysis

The data cleaning tools market can be segmented by component into software and services. The software segment encompasses various tools and platforms designed for data cleaning, while the services segment includes consultancy, implementation, and maintenance services provided by vendors.

The software segment holds the largest market share and is expected to continue leading during the forecast period. This dominance can be attributed to the increasing adoption of automated data cleaning solutions that offer high efficiency and accuracy. These software solutions are equipped with advanced algorithms and functionalities that can handle large volumes of data, identify errors, and correct them without manual intervention. The rising adoption of cloud-based data cleaning software further bolsters this segment, as it offers scalability and ease of

OpenRefine (formerly Google Refine) is a powerful free and open source tool for data cleaning, enabling you to correct errors in the data, and make sure that the values and formatting are consistent. In addition, OpenRefine records your processing steps, enabling you to apply the same cleaning procedure to other data, and enhancing the reproducibility of your analysis. This workshop will teach you to use OpenRefine to clean and format data and automatically track any changes that you make.

Embark on a transformative journey with our Data Cleaning Project, where we meticulously refine and polish raw data into valuable insights. Our project focuses on streamlining data sets, removing inconsistencies, and ensuring accuracy to unlock its full potential.

Through advanced techniques and rigorous processes, we standardize formats, address missing values, and eliminate duplicates, creating a clean and reliable foundation for analysis. By enhancing data quality, we empower organizations to make informed decisions, drive innovation, and achieve strategic objectives with confidence.

Join us as we embark on this essential phase of data preparation, paving the way for more accurate and actionable insights that fuel success."

This folder "XPS data" contains original and evaluated XPS data (.vms) on a p-GaN sample which was treated at various temperatures and underwent Ar+ irradiation.

Furthermore, the folder "REM Images" contains REM images (.tif) and EDX data (.xlsx) on the used excessively treated sample.

All images that are published in the main manuscript are collected as .tif files in the folder "images".

The mean, standard deviation, preservation of data (PD), sensitivity and specificity of five data cleaning approaches with and without an algorithm (A) compared to uncleaned longitudinal growth measurements in CLOSER data with and without simulated duplications and 1% errors.

The percentage of gold standard corrections of errors induced into CLOSER data with simulated duplications and 1% errors using the algorithmic data cleaning methods.

A data cleaning tool customised for cleaning and sorting the data generated during the Enviro-Champs pilot study as they are downloaded from Formshare, the platform capturing data sent from a customised ODK Collect form collection app. The dataset inclues the latest data from the pilot study as at 14 May 2024.

Data Cleansing Software Market Outlook

The global data cleansing software market size was valued at approximately USD 1.5 billion in 2023 and is projected to reach around USD 4.2 billion by 2032, exhibiting a compound annual growth rate (CAGR) of 12.5% during the forecast period. This substantial growth can be attributed to the increasing importance of maintaining clean and reliable data for business intelligence and analytics, which are driving the adoption of data cleansing solutions across various industries.

The proliferation of big data and the growing emphasis on data-driven decision-making are significant growth factors for the data cleansing software market. As organizations collect vast amounts of data from multiple sources, ensuring that this data is accurate, consistent, and complete becomes critical for deriving actionable insights. Data cleansing software helps organizations eliminate inaccuracies, inconsistencies, and redundancies, thereby enhancing the quality of their data and improving overall operational efficiency. Additionally, the rising adoption of advanced analytics and artificial intelligence (AI) technologies further fuels the demand for data cleansing software, as clean data is essential for the accuracy and reliability of these technologies.

Another key driver of market growth is the increasing regulatory pressure for data compliance and governance. Governments and regulatory bodies across the globe are implementing stringent data protection regulations, such as the General Data Protection Regulation (GDPR) in Europe and the California Consumer Privacy Act (CCPA) in the United States. These regulations mandate organizations to ensure the accuracy and security of the personal data they handle. Data cleansing software assists organizations in complying with these regulations by identifying and rectifying inaccuracies in their data repositories, thus minimizing the risk of non-compliance and hefty penalties.

The growing trend of digital transformation across various industries also contributes to the expanding data cleansing software market. As businesses transition to digital platforms, they generate and accumulate enormous volumes of data. To derive meaningful insights and maintain a competitive edge, it is imperative for organizations to maintain high-quality data. Data cleansing software plays a pivotal role in this process by enabling organizations to streamline their data management practices and ensure the integrity of their data. Furthermore, the increasing adoption of cloud-based solutions provides additional impetus to the market, as cloud platforms facilitate seamless integration and scalability of data cleansing tools.

Regionally, North America holds a dominant position in the data cleansing software market, driven by the presence of numerous technology giants and the rapid adoption of advanced data management solutions. The region is expected to continue its dominance during the forecast period, supported by the strong emphasis on data quality and compliance. Europe is also a significant market, with countries like Germany, the UK, and France showing substantial demand for data cleansing solutions. The Asia Pacific region is poised for significant growth, fueled by the increasing digitalization of businesses and the rising awareness of data quality's importance. Emerging economies in Latin America and the Middle East & Africa are also expected to witness steady growth, driven by the growing adoption of data-driven technologies.

The role of Data Quality Tools cannot be overstated in the context of data cleansing software. These tools are integral in ensuring that the data being processed is not only clean but also of high quality, which is crucial for accurate analytics and decision-making. Data Quality Tools help in profiling, monitoring, and cleansing data, thereby ensuring that organizations can trust their data for strategic decisions. As organizations increasingly rely on data-driven insights, the demand for robust Data Quality Tools is expected to rise. These tools offer functionalities such as data validation, standardization, and enrichment, which are essential for maintaining the integrity of data across various platforms and applications. The integration of these tools with data cleansing software enhances the overall data management capabilities of organizations, enabling them to achieve greater operational efficiency and compliance with data regulations.

Component Analysis

The data cle

Sample data for exercises in Further Adventures in Data Cleaning.

The data center cleaning services market is experiencing robust growth, driven by the increasing demand for high-availability and uptime in data centers worldwide. The rising adoption of cloud computing, the proliferation of edge data centers, and the stringent regulatory requirements for data center hygiene are key factors fueling this expansion. The market is segmented by application (internet industry, finance & insurance, manufacturing, government, others) and cleaning type (equipment, ceiling, floor, others). While precise market sizing data is not provided, a reasonable estimate based on industry reports and the provided CAGR would place the 2025 market value at approximately $2.5 billion (this is an educated estimation and not based on the provided data directly), with a projected CAGR of 8% over the forecast period (2025-2033). North America and Europe currently hold the largest market share due to the high concentration of data centers and stringent regulatory environments. However, Asia-Pacific is expected to witness significant growth in the coming years, fueled by rapid digital transformation and increasing investments in data center infrastructure. The competitive landscape is characterized by a mix of both large multinational corporations and specialized regional service providers. Key players focus on providing specialized cleaning solutions tailored to the unique requirements of data centers, including electrostatic discharge (ESD) protection, contamination control, and specialized equipment. The market's growth faces some restraints, including high service costs, the need for specialized training and expertise, and the potential risks associated with improper cleaning procedures that could damage sensitive equipment. However, increasing awareness of the importance of data center hygiene and the potential consequences of downtime are expected to outweigh these limitations, ensuring continued market expansion throughout the forecast period. The continued adoption of advanced cleaning technologies and the rise of green cleaning practices will further shape the market’s trajectory in the coming years.

Restaurant Sales Dataset with Dirt Documentation

Overview

The Restaurant Sales Dataset with Dirt contains data for 17,534 transactions. The data introduces realistic inconsistencies ("dirt") to simulate real-world scenarios where data may have missing or incomplete information. The dataset includes sales details across multiple categories, such as starters, main dishes, desserts, drinks, and side dishes.

Dataset Use Cases

This dataset is suitable for: - Practicing data cleaning tasks, such as handling missing values and deducing missing information. - Conducting exploratory data analysis (EDA) to study restaurant sales patterns. - Feature engineering to create new variables for machine learning tasks.

Columns Description

Key Characteristics

1. Data Dirtiness:

   • Missing values in key columns (Item, Price, Quantity, Order Total, Payment Method) simulate real-world challenges.
   • At least one of the following conditions is ensured for each record to identify an item:
     • Item is present.
     • Price is present.
     • Both Quantity and Order Total are present.
   • If Price or Quantity is missing, the other is used to deduce the missing value (e.g., Order Total / Quantity).

2. Menu Categories and Items:

   • Items are divided into five categories:
     • Starters: E.g., Chicken Melt, French Fries.
     • Main Dishes: E.g., Grilled Chicken, Steak.
     • Desserts: E.g., Chocolate Cake, Ice Cream.
     • Drinks: E.g., Coca Cola, Water.
     • Side Dishes: E.g., Mashed Potatoes, Garlic Bread.

3 Time Range: - Orders span from January 1, 2022, to December 31, 2023.

Cleaning Suggestions

1. Handle Missing Values:

   • Fill missing Order Total or Quantity using the formula: Order Total = Price * Quantity.
   • Deduce missing Price from Order Total / Quantity if both are available.

2. Validate Data Consistency:

   • Ensure that calculated values (Order Total = Price * Quantity) match.

3. Analyze Missing Patterns:

   • Study the distribution of missing values across categories and payment methods.

Menu Map with Prices and Categories

Description of the study design, data collection and processing, cohort details and data accessibility for longitudinal height or weight measurements in Dogslife, SAVSNET, Banfield and CLOSER datasets.

The data center cleaning service market is experiencing robust growth, driven by the escalating demand for high-availability and fault-tolerant data center infrastructure. The increasing adoption of cloud computing, big data analytics, and the Internet of Things (IoT) is fueling the need for specialized cleaning services that maintain optimal operational efficiency and prevent costly downtime. Stringent regulatory compliance requirements regarding data center hygiene and environmental standards further bolster market expansion. While precise market sizing data was not provided, a reasonable estimate based on industry reports and observed growth in related sectors places the 2025 market value at approximately $2.5 billion USD. Considering a conservative Compound Annual Growth Rate (CAGR) of 8% over the forecast period (2025-2033), the market is projected to exceed $5 billion by 2033. Key market segments include equipment cleaning (servers, networking equipment), ceiling and floor cleaning, and specialized cleaning for controlled environments. North America and Europe currently dominate the market share, benefiting from established data center hubs and robust regulatory frameworks. However, growth in Asia-Pacific, driven by expanding data center infrastructure in countries like China and India, is expected to significantly contribute to overall market expansion in the coming years. Significant restraints include the high cost of specialized cleaning services, the need for highly trained personnel, and the potential disruption to data center operations during cleaning procedures. To mitigate these challenges, innovative cleaning technologies, such as automated cleaning systems and environmentally friendly cleaning agents, are gaining traction. The market is also seeing a trend towards outsourcing cleaning services to specialized firms, as businesses prioritize core competencies and optimize cost management. Companies offering comprehensive data center cleaning solutions, including preventative maintenance and compliance audits, are best positioned to capture market share. The competitive landscape is characterized by a mix of large multinational corporations and specialized regional service providers.

This dataset includes all of the data downloaded from GBIF (DOIs provided in README.md as well as below, downloaded Feb 2021) as well as data downloaded from SCAN. This dataset has 2,808,432 records and can be used as a reference to the verbatim data before it underwent the cleaning process. The only modifications made to this datset after direct download from the data portals are the following:

1) for GBIF records, I renamed the countryCode column to be "country" so that the column title is consistent across both GBIF and SCAN 2) A source column was added where I specify if the record came from GBIF or SCAN 3) Duplicate records across SCAN and GBIF were removed by identifying identical instances "catalogNumber" and "institutionCode" 4) Only the Darwin core columns (DwC) that were shared across downloaded datasets were retained. GBIF contained ~249 DwC variables, and SCAN data contained fewer, so this combined dataset only includes the ~80 columns shared between the two datasets

For GBIF, we downloaded the data in three separate chunks, therefore there are three DOIs. See below:

GBIF.org (3 February 2021) GBIF Occurrence Downloadhttps://doi.org/10.15468/dl.6cxfsw GBIF.org (3 February 2021) GBIF Occurrence Downloadhttps://doi.org/10.15468/dl.b9rfa7 GBIF.org (3 February 2021) GBIF Occurrence Downloadhttps://doi.org/10.15468/dl.w2nndm

Description of the data entries, individuals, data entries per individual, mean and standard deviation of the longitudinal height or weight measurements in Dogslife, SAVSNET, Banfield and CLOSER data with and without simulated duplications and 1% errors before and after removal of duplicated measurement records.

Data Cleansing Tools Market Outlook

The global data cleansing tools market size was valued at approximately USD 1.5 billion in 2023 and is projected to reach USD 4.2 billion by 2032, growing at a CAGR of 12.1% from 2024 to 2032. One of the primary growth factors driving the market is the increasing need for high-quality data in various business operations and decision-making processes.

The surge in big data and the subsequent increased reliance on data analytics are significant factors propelling the growth of the data cleansing tools market. Organizations increasingly recognize the value of high-quality data in driving strategic initiatives, customer relationship management, and operational efficiency. The proliferation of data generated across different sectors such as healthcare, finance, retail, and telecommunications necessitates the adoption of tools that can clean, standardize, and enrich data to ensure its reliability and accuracy.

Furthermore, the rising adoption of Machine Learning (ML) and Artificial Intelligence (AI) technologies has underscored the importance of clean data. These technologies rely heavily on large datasets to provide accurate and reliable insights. Any errors or inconsistencies in data can lead to erroneous outcomes, making data cleansing tools indispensable. Additionally, regulatory and compliance requirements across various industries necessitate the maintenance of clean and accurate data, further driving the market for data cleansing tools.

The growing trend of digital transformation across industries is another critical growth factor. As businesses increasingly transition from traditional methods to digital platforms, the volume of data generated has skyrocketed. However, this data often comes from disparate sources and in various formats, leading to inconsistencies and errors. Data cleansing tools are essential in such scenarios to integrate data from multiple sources and ensure its quality, thus enabling organizations to derive actionable insights and maintain a competitive edge.

In the context of ensuring data reliability and accuracy, Data Quality Software and Solutions play a pivotal role. These solutions are designed to address the challenges associated with managing large volumes of data from diverse sources. By implementing robust data quality frameworks, organizations can enhance their data governance strategies, ensuring that data is not only clean but also consistent and compliant with industry standards. This is particularly crucial in sectors where data-driven decision-making is integral to business success, such as finance and healthcare. The integration of advanced data quality solutions helps businesses mitigate risks associated with poor data quality, thereby enhancing operational efficiency and strategic planning.

Regionally, North America is expected to hold the largest market share due to the early adoption of advanced technologies, robust IT infrastructure, and the presence of key market players. Europe is also anticipated to witness substantial growth due to stringent data protection regulations and the increasing adoption of data-driven decision-making processes. Meanwhile, the Asia Pacific region is projected to experience the highest growth rate, driven by the rapid digitalization of emerging economies, the expansion of the IT and telecommunications sector, and increasing investments in data management solutions.

Component Analysis

The data cleansing tools market is segmented into software and services based on components. The software segment is anticipated to dominate the market due to its extensive use in automating the data cleansing process. The software solutions are designed to identify, rectify, and remove errors in data sets, ensuring data accuracy and consistency. They offer various functionalities such as data profiling, validation, enrichment, and standardization, which are critical in maintaining high data quality. The high demand for these functionalities across various industries is driving the growth of the software segment.

On the other hand, the services segment, which includes professional services and managed services, is also expected to witness significant growth. Professional services such as consulting, implementation, and training are crucial for organizations to effectively deploy and utilize data cleansing tools. As businesses increasingly realize the importance of clean data, the demand for expert

Quadrant provides Insightful, accurate, and reliable mobile location data.

Our privacy-first mobile location data unveils hidden patterns and opportunities, provides actionable insights, and fuels data-driven decision-making at the world's biggest companies.

These companies rely on our privacy-first Mobile Location and Points-of-Interest Data to unveil hidden patterns and opportunities, provide actionable insights, and fuel data-driven decision-making. They build better AI models, uncover business insights, and enable location-based services using our robust and reliable real-world data.

We conduct stringent evaluations on data providers to ensure authenticity and quality. Our proprietary algorithms detect, and cleanse corrupted and duplicated data points – allowing you to leverage our datasets rapidly with minimal processing or cleaning. During the ingestion process, our proprietary Data Filtering Algorithms remove events based on a number of both qualitative factors, as well as latency and other integrity variables to provide more efficient data delivery. The deduplicating algorithm focuses on a combination of four important attributes: Device ID, Latitude, Longitude, and Timestamp. This algorithm scours our data and identifies rows that contain the same combination of these four attributes. Post-identification, it retains a single copy and eliminates duplicate values to ensure our customers only receive complete and unique datasets.

We actively identify overlapping values at the provider level to determine the value each offers. Our data science team has developed a sophisticated overlap analysis model that helps us maintain a high-quality data feed by qualifying providers based on unique data values rather than volumes alone – measures that provide significant benefit to our end-use partners.

Quadrant mobility data contains all standard attributes such as Device ID, Latitude, Longitude, Timestamp, Horizontal Accuracy, and IP Address, and non-standard attributes such as Geohash and H3. In addition, we have historical data available back through 2022.

Through our in-house data science team, we offer sophisticated technical documentation, location data algorithms, and queries that help data buyers get a head start on their analyses. Our goal is to provide you with data that is “fit for purpose”.

Abstract

Within the frame of PCBS' efforts in providing official Palestinian statistics in the different life aspects of Palestinian society and because the wide spread of Computer, Internet and Mobile Phone among the Palestinian people, and the important role they may play in spreading knowledge and culture and contribution in formulating the public opinion, PCBS conducted the Household Survey on Information and Communications Technology, 2014.

The main objective of this survey is to provide statistical data on Information and Communication Technology in the Palestine in addition to providing data on the following: - Prevalence of computers and access to the Internet. - Study the penetration and purpose of Technology use.

Geographic coverage

Palestine (West Bank and Gaza Strip), type of locality (urban, rural, refugee camps) and governorate.

Analysis unit

• Household.
• Persons 10 years and over .

Universe

All Palestinian households and individuals whose usual place of residence in Palestine with focus on persons aged 10 years and over in year 2014.

Kind of data

Sample survey data [ssd]

Sampling procedure

Sampling Frame The sampling frame consists of a list of enumeration areas adopted in the Population, Housing and Establishments Census of 2007. Each enumeration area has an average size of about 124 households. These were used in the first phase as Preliminary Sampling Units in the process of selecting the survey sample.

Sample Size The total sample size of the survey was 7,268 households, of which 6,000 responded.

Sample Design The sample is a stratified clustered systematic random sample. The design comprised three phases:

Phase I: Random sample of 240 enumeration areas. Phase II: Selection of 25 households from each enumeration area selected in phase one using systematic random selection. Phase III: Selection of an individual (10 years or more) in the field from the selected households; KISH TABLES were used to ensure indiscriminate selection.

Sample Strata Distribution of the sample was stratified by: 1- Governorate (16 governorates, J1). 2- Type of locality (urban, rural and camps).

Mode of data collection

Face-to-face [f2f]

Research instrument

The survey questionnaire consists of identification data, quality controls and three main sections: Section I: Data on household members that include identification fields, the characteristics of household members (demographic and social) such as the relationship of individuals to the head of household, sex, date of birth and age.

Section II: Household data include information regarding computer processing, access to the Internet, and possession of various media and computer equipment. This section includes information on topics related to the use of computer and Internet, as well as supervision by households of their children (5-17 years old) while using the computer and Internet, and protective measures taken by the household in the home.

Section III: Data on persons (aged 10 years and over) about computer use, access to the Internet and possession of a mobile phone.

Cleaning operations

Preparation of Data Entry Program: This stage included preparation of the data entry programs using an ACCESS package and defining data entry control rules to avoid errors, plus validation inquiries to examine the data after it had been captured electronically.

Data Entry: The data entry process started on the 8th of May 2014 and ended on the 23rd of June 2014. The data entry took place at the main PCBS office and in field offices using 28 data clerks.

Editing and Cleaning procedures: Several measures were taken to avoid non-sampling errors. These included editing of questionnaires before data entry to check field errors, using a data entry application that does not allow mistakes during the process of data entry, and then examining the data by using frequency and cross tables. This ensured that data were error free; cleaning and inspection of the anomalous values were conducted to ensure harmony between the different questions on the questionnaire.

Response rate

Response Rates: 79%

Sampling error estimates

There are many aspects of the concept of data quality; this includes the initial planning of the survey to the dissemination of the results and how well users understand and use the data. There are three components to the quality of statistics: accuracy, comparability, and quality control procedures.

Checks on data accuracy cover many aspects of the survey and include statistical errors due to the use of a sample, non-statistical errors resulting from field workers or survey tools, and response rates and their effect on estimations. This section includes:

Statistical Errors Data of this survey may be affected by statistical errors due to the use of a sample and not a complete enumeration. Therefore, certain differences can be expected in comparison with the real values obtained through censuses. Variances were calculated for the most important indicators.

Variance calculations revealed that there is no problem in disseminating results nationally or regionally (the West Bank, Gaza Strip), but some indicators show high variance by governorate, as noted in the tables of the main report.

Non-Statistical Errors Non-statistical errors are possible at all stages of the project, during data collection or processing. These are referred to as non-response errors, response errors, interviewing errors and data entry errors. To avoid errors and reduce their effects, strenuous efforts were made to train the field workers intensively. They were trained on how to carry out the interview, what to discuss and what to avoid, and practical and theoretical training took place during the training course. Training manuals were provided for each section of the questionnaire, along with practical exercises in class and instructions on how to approach respondents to reduce refused cases. Data entry staff were trained on the data entry program, which was tested before starting the data entry process.

Several measures were taken to avoid non-sampling errors. These included editing of questionnaires before data entry to check field errors, using a data entry application that does not allow mistakes during the process of data entry, and then examining the data by using frequency and cross tables. This ensured that data were error free; cleaning and inspection of the anomalous values were conducted to ensure harmony between the different questions on the questionnaire.

The sources of non-statistical errors can be summarized as: 1. Some of the households were not at home and could not be interviewed, and some households refused to be interviewed. 2. In unique cases, errors occurred due to the way the questions were asked by interviewers and respondents misunderstood some of the questions.

The LSC (Leicester Scientific Corpus)

April 2020 by Neslihan Suzen, PhD student at the University of Leicester (ns433@leicester.ac.uk) Supervised by Prof Alexander Gorban and Dr Evgeny MirkesThe data are extracted from the Web of Science [1]. You may not copy or distribute these data in whole or in part without the written consent of Clarivate Analytics.[Version 2] A further cleaning is applied in Data Processing for LSC Abstracts in Version 1*. Details of cleaning procedure are explained in Step 6.* Suzen, Neslihan (2019): LSC (Leicester Scientific Corpus). figshare. Dataset. https://doi.org/10.25392/leicester.data.9449639.v1.Getting StartedThis text provides the information on the LSC (Leicester Scientific Corpus) and pre-processing steps on abstracts, and describes the structure of files to organise the corpus. This corpus is created to be used in future work on the quantification of the meaning of research texts and make it available for use in Natural Language Processing projects.LSC is a collection of abstracts of articles and proceeding papers published in 2014, and indexed by the Web of Science (WoS) database [1]. The corpus contains only documents in English. Each document in the corpus contains the following parts:1. Authors: The list of authors of the paper2. Title: The title of the paper 3. Abstract: The abstract of the paper 4. Categories: One or more category from the list of categories [2]. Full list of categories is presented in file ‘List_of _Categories.txt’. 5. Research Areas: One or more research area from the list of research areas [3]. Full list of research areas is presented in file ‘List_of_Research_Areas.txt’. 6. Total Times cited: The number of times the paper was cited by other items from all databases within Web of Science platform [4] 7. Times cited in Core Collection: The total number of times the paper was cited by other papers within the WoS Core Collection [4]The corpus was collected in July 2018 online and contains the number of citations from publication date to July 2018. We describe a document as the collection of information (about a paper) listed above. The total number of documents in LSC is 1,673,350.Data ProcessingStep 1: Downloading of the Data Online

The dataset is collected manually by exporting documents as Tab-delimitated files online. All documents are available online.Step 2: Importing the Dataset to R

The LSC was collected as TXT files. All documents are extracted to R.Step 3: Cleaning the Data from Documents with Empty Abstract or without CategoryAs our research is based on the analysis of abstracts and categories, all documents with empty abstracts and documents without categories are removed.Step 4: Identification and Correction of Concatenate Words in AbstractsEspecially medicine-related publications use ‘structured abstracts’. Such type of abstracts are divided into sections with distinct headings such as introduction, aim, objective, method, result, conclusion etc. Used tool for extracting abstracts leads concatenate words of section headings with the first word of the section. For instance, we observe words such as ConclusionHigher and ConclusionsRT etc. The detection and identification of such words is done by sampling of medicine-related publications with human intervention. Detected concatenate words are split into two words. For instance, the word ‘ConclusionHigher’ is split into ‘Conclusion’ and ‘Higher’.The section headings in such abstracts are listed below:

Background Method(s) Design Theoretical Measurement(s) Location Aim(s) Methodology Process Abstract Population Approach Objective(s) Purpose(s) Subject(s) Introduction Implication(s) Patient(s) Procedure(s) Hypothesis Measure(s) Setting(s) Limitation(s) Discussion Conclusion(s) Result(s) Finding(s) Material (s) Rationale(s) Implications for health and nursing policyStep 5: Extracting (Sub-setting) the Data Based on Lengths of AbstractsAfter correction, the lengths of abstracts are calculated. ‘Length’ indicates the total number of words in the text, calculated by the same rule as for Microsoft Word ‘word count’ [5].According to APA style manual [6], an abstract should contain between 150 to 250 words. In LSC, we decided to limit length of abstracts from 30 to 500 words in order to study documents with abstracts of typical length ranges and to avoid the effect of the length to the analysis.

Step 6: [Version 2] Cleaning Copyright Notices, Permission polices, Journal Names and Conference Names from LSC Abstracts in Version 1Publications can include a footer of copyright notice, permission policy, journal name, licence, author’s right or conference name below the text of abstract by conferences and journals. Used tool for extracting and processing abstracts in WoS database leads to attached such footers to the text. For example, our casual observation yields that copyright notices such as ‘Published by Elsevier ltd.’ is placed in many texts. To avoid abnormal appearances of words in further analysis of words such as bias in frequency calculation, we performed a cleaning procedure on such sentences and phrases in abstracts of LSC version 1. We removed copyright notices, names of conferences, names of journals, authors’ rights, licenses and permission policies identified by sampling of abstracts.Step 7: [Version 2] Re-extracting (Sub-setting) the Data Based on Lengths of AbstractsThe cleaning procedure described in previous step leaded to some abstracts having less than our minimum length criteria (30 words). 474 texts were removed.Step 8: Saving the Dataset into CSV FormatDocuments are saved into 34 CSV files. In CSV files, the information is organised with one record on each line and parts of abstract, title, list of authors, list of categories, list of research areas, and times cited is recorded in fields.To access the LSC for research purposes, please email to ns433@le.ac.uk.References[1]Web of Science. (15 July). Available: https://apps.webofknowledge.com/ [2]WoS Subject Categories. Available: https://images.webofknowledge.com/WOKRS56B5/help/WOS/hp_subject_category_terms_tasca.html [3]Research Areas in WoS. Available: https://images.webofknowledge.com/images/help/WOS/hp_research_areas_easca.html [4]Times Cited in WoS Core Collection. (15 July). Available: https://support.clarivate.com/ScientificandAcademicResearch/s/article/Web-of-Science-Times-Cited-accessibility-and-variation?language=en_US [5]Word Count. Available: https://support.office.com/en-us/article/show-word-count-3c9e6a11-a04d-43b4-977c-563a0e0d5da3 [6]A. P. Association, Publication manual. American Psychological Association Washington, DC, 1983.