Sample data for exercises in Further Adventures in Data Cleaning.

Access and clean an open source herbarium dataset using Excel or RStudio.

Dataset

This dataset was created by Luis Lira

Contents

Dataset

This dataset was created by George M122

Contents

Corpus consisting of 10,000 Facebook posts manually annotated on sentiment (2,587 positive, 5,174 neutral, 1,991 negative and 248 bipolar posts). The archive contains data and statistics in an Excel file (FBData.xlsx) and gold data in two text files with posts (gold-posts.txt) and labels (gols-labels.txt) on corresponding lines.

We describe a bibliometric network characterizing co-authorship collaborations in the entire Italian academic community. The network, consisting of 38,220 nodes and 507,050 edges, is built upon two distinct data sources: faculty information provided by the Italian Ministry of University and Research and publications available in Semantic Scholar. Both nodes and edges are associated with a large variety of semantic data, including gender, bibliometric indexes, authors' and publications' research fields, and temporal information. While linking data between the two original sources posed many challenges, the network has been carefully validated to assess its reliability and to understand its graph-theoretic characteristics. By resembling several features of social networks, our dataset can be profitably leveraged in experimental studies in the wide social network analytics domain as well as in more specific bibliometric contexts. , The proposed network is built starting from two distinct data sources:

the entire dataset dump from Semantic Scholar (with particular emphasis on the authors and papers datasets) the entire list of Italian faculty members as maintained by Cineca (under appointment by the Italian Ministry of University and Research).

By means of a custom name-identity recognition algorithm (details are available in the accompanying paper published in Scientific Data), the names of the authors in the Semantic Scholar dataset have been mapped against the names contained in the Cineca dataset and authors with no match (e.g., because of not being part of an Italian university) have been discarded. The remaining authors will compose the nodes of the network, which have been enriched with node-related (i.e., author-related) attributes. In order to build the network edges, we leveraged the papers dataset from Semantic Scholar: specifically, any two authors are said to be connected if there is at least one pap..., , # Data cleaning and enrichment through data integration: networking the Italian academia

https://doi.org/10.5061/dryad.wpzgmsbwj

Description of the data and file structure

This repository contains two main data files:

• edge_data_AGG.csv, the full network in comma-separated edge list format (this file contains mainly temporal co-authorship information);
• Coauthorship_Network_AGG.graphml, the full network in GraphML format.Â

along with several supplementary data, listed below, useful only to build the network (i.e., for reproducibility only):

• University-City-match.xlsx, an Excel file that maps the name of a university against the city where its respective headquarter is located;
• Areas-SS-CINECA-match.xlsx, an Excel file that maps the research areas in Cineca against the research areas in Semantic Scholar.

Description of the main data files

The Coauthorship_Network_AGG.graphml is intended to be the core file which c...

Global Cleaning Robot Market size was valued at USD 4.19 billion in 2022 and is poised to grow from USD 4.97 billion in 2023 to USD 12.81 billion by 2031, growing at a CAGR of 22.9% in the forecast period (2024-2031).

This data set supports the journal paper "Manipulating the consequences of tests: How Shanghai teens react to different consequences", published in Educational Research and Evaluation, v26 (n5-6), pp.221-251.The data were obtained to test the impact of different levels of consequence for taking a test on student test-taking effort. The data are part of the PhD project of Anran Zhao, supervised by Brown & Meissel.The data set is in MS Excel format. Sheet 1 provides an anonymous wide-format data set post-cleaning and missing value analysis of the data.Sheet 2 provides a description of each variable.

Schedule and Monitor of Gully Cleaning for Dublin City These datasets show the gully cleaning statistics from 2004 to September 14th 2011. It consists attached 6 No. Excel Spreadsheets with the datasets from the Daily Returns section of the Gully Cleaning Application and one dataset from the Gully Repairs section of the gully application. They are divided into the five Dublin City Council administrative areas; Central, North Central, North West, Southeast, South Central. There is also a dataset containing details of all Gully repairs pending (all areas included).The datasets cover all Daily Returns since the gully cleaning programme commenced in 2004. Daily Returns are lists of the work that the gully cleaning crews carry out daily. All gullies on a street are cleaned where possible. A list of Omissions is recorded where some gullies may not have been cleaned due to lack of access or other reasons. Also, the gullies that required repair were noted. The Daily Returns datasets record only the number of gullies requiring repair on a particular street, not the details of the repair. Information in the fields is as follows: .Road name: street name or laneway denoted by nearest house or lamp post etc. If a road name is followed by the letters pl in capital letters than it means that either this road or a section of this road has been placed on the priority list due to a history of flooding or a higher potential of the gully blocking due to location etc. If a road name is followed by a number of zeros in the gullies inspected - gullies cleaned columns etc then it is very probable that this road was travelled during heavy rain as part of our flood zones and there was no flooding noted along this road at the time of travelling. Gullies inspected: number of gullies inspected along road/lane .A road name followed by lower case road names denotes a road that is part of more than one area in our gully cleaning areas and these lower case names denote the starting point and finishing point for the crews working in the particular area i.e. Howth Road All Saints Rd-Fairview denotes that the section of the Howth road between all saints road and Fairview are within the area that the crew have been asked to work in. Gullies cleaned :number of gullies cleaned from total inspected .Gully omissions :number of gullies missed i.e. Unable to put boom or shovel into gully pot due to parked cars / unable to lift grids / hoarding over gullies etc .Gully repairs: number of repairs based on inspections-note not all repairs prevent the gully from being cleaned. Comments box: this box is used to provide any additional information that may be of benefit and it can be noted that results of work carried out by the mini jet is placed in this box.

Abstract

A Knowledge, Attitudes and Practices (KAP) survey was conducted in Ajuong Thok and Pamir Refugee Camps in October 2019 to determine the current Water, Sanitation and Hygiene (WASH) conditions as well as hygiene attitudes and practices within the households (HHs) surveyed. The assessment utilized a systematic random sampling method, and a total of 1,474 HHs (735 HHs in Ajuong Thok and 739 HHs in Pamir) were surveyed using mobile data collection (MDC) within a period of 21 days. Data was cleaned and analyzed in Excel. The summary of the results is presented in this report.

The findings show that the overall average number of liters of water per person per day was 23.4, in both Ajuong Thok and Pamir Camps, which was slightly higher than the recommended United Nations High Commissioner for Refugees (UNHCR) minimum standard of at least 20 liters of water available per person per day. This is a slight improvement from the 21 liters reported the previous year. The average HH size was six people. Women comprised 83% of the surveyed respondents and males 17%. Almost all the respondents were refugees, constituting 99.5% (n=1,466). The refugees were aware of the key health and hygiene practices, possibly as a result of routine health and hygiene messages delivered to them by Samaritan´s Purse (SP) and other health partners. Most refugees had knowledge about keeping the water containers clean, washing hands during critical times, safe excreta disposal and disease prevention.

Geographic coverage

Ajuong Thok and Pamir Refugee Camps

Analysis unit

Households

Universe

All households in Ajuong Thok and Pamir Refugee Camps

Kind of data

Sample survey data [ssd]

Sampling procedure

Households were selected using systematic random sampling. Enumerators systematically walked through the camp block by block, row by row, in such a way as to pass each HH. Within blocks, enumerators started at one corner, then systematically used the sampling interval as they walked up and down each of the rows throughout the block, covering every block in Ajuong Thok and Pamir.

In each location, the first HH sampled in a block was generated using an Excel tool customized by UNHCR which generated a Random Start and Sampling Interval.

Mode of data collection

Face-to-face [f2f]

Research instrument

The survey questionnaire used to collect the data consists of the following sections: - Demographics - Water collection and storage - Drinking water hygiene - Hygiene - Sanitation - Messaging - Distribution (NFI) - Diarrhea prevalence, knowledge and health seeking behaviour - Menstrual hygiene

Cleaning operations

The data collected was uploaded to a server at the end of each day. IFormBuilder generated a Microsoft (MS) Excel spreadsheet dataset which was then cleaned and analyzed using MS Excel.

Given that SP is currently implementing a WASH program in Ajuong Thok and Pamir, the assessment data collected in these camps will not only serve as the endline for UNHCR 2018 programming but also as the baseline for 2019 programming.

Data was anonymized through decoding and local suppression.

Abstract

This dataset was generated from a set of Excel spreadsheets from an Information and Communication Technology Services (ICTS) administrative database on student applications to the University of Cape Town (UCT). This database contains information on applications to UCT between the January 2006 and December 2014. In the original form received by DataFirst the data were ill suited to research purposes. This dataset represents an attempt at cleaning and organizing these data into a more tractable format. To ensure data confidentiality direct identifiers have been removed from the data and the data is only made available to accredited researchers through DataFirst's Secure Data Service.

The dataset was separated into the following data files:

1. Application level information: the "finest" unit of analysis. Individuals may have multiple applications. Uniquely identified by an application ID variable. There are a total of 1,714,669 applications on record.
2. Individual level information: individuals may have multiple applications. Each individual is uniquely identified by an individual ID variable. Each individual is associated with information on "key subjects" from a separate data file also contained in the database. These key subjects are all separate variables in the individual level data file. There are a total of 285,005 individuals on record.
3. Secondary Education Information: individuals can also be associated with row entries for each subject. This data file does not have a unique identifier. Instead, each row entry represents a specific secondary school subject for a specific individual. These subjects are quite specific and the data allows the user to distinguish between, for example, higher grade accounting and standard grade accounting. It also allows the user to identify the educational authority issuing the qualification e.g. Cambridge Internal Examinations (CIE) versus National Senior Certificate (NSC).
4. Tertiary Education Information: the smallest of the four data files. There are multiple entries for each individual in this dataset. Each row entry contains information on the year, institution and transcript information and can be associated with individuals.

Analysis unit

Applications, individuals

Kind of data

Administrative records [adm]

Mode of data collection

Other [oth]

Cleaning operations

The data files were made available to DataFirst as a group of Excel spreadsheet documents from an SQL database managed by the University of Cape Town's Information and Communication Technology Services . The process of combining these original data files to create a research-ready dataset is summarised in a document entitled "Notes on preparing the UCT Student Application Data 2006-2014" accompanying the data.

Data for manuscript “Functional morphology and efficiency of the antenna cleaner in Camponotus rufifemur ants"Excel file includes 3 data sheets. One sheet for each experiment. The corresponding figures from the manuscript are mentioned above the actual data.Manuscript data.xlsx

Abstract

The harmonized data set on health, created and published by the ERF, is a subset of Iraq Household Socio Economic Survey (IHSES) 2012. It was derived from the household, individual and health modules, collected in the context of the above mentioned survey. The sample was then used to create a harmonized health survey, comparable with the Iraq Household Socio Economic Survey (IHSES) 2007 micro data set.

----> Overview of the Iraq Household Socio Economic Survey (IHSES) 2012:

Iraq is considered a leader in household expenditure and income surveys where the first was conducted in 1946 followed by surveys in 1954 and 1961. After the establishment of Central Statistical Organization, household expenditure and income surveys were carried out every 3-5 years in (1971/ 1972, 1976, 1979, 1984/ 1985, 1988, 1993, 2002 / 2007). Implementing the cooperation between CSO and WB, Central Statistical Organization (CSO) and Kurdistan Region Statistics Office (KRSO) launched fieldwork on IHSES on 1/1/2012. The survey was carried out over a full year covering all governorates including those in Kurdistan Region.

The survey has six main objectives. These objectives are:

1. Provide data for poverty analysis and measurement and monitor, evaluate and update the implementation Poverty Reduction National Strategy issued in 2009.
2. Provide comprehensive data system to assess household social and economic conditions and prepare the indicators related to the human development.
3. Provide data that meet the needs and requirements of national accounts.
4. Provide detailed indicators on consumption expenditure that serve making decision related to production, consumption, export and import.
5. Provide detailed indicators on the sources of households and individuals income.
6. Provide data necessary for formulation of a new consumer price index number.

The raw survey data provided by the Statistical Office were then harmonized by the Economic Research Forum, to create a comparable version with the 2006/2007 Household Socio Economic Survey in Iraq. Harmonization at this stage only included unifying variables' names, labels and some definitions. See: Iraq 2007 & 2012- Variables Mapping & Availability Matrix.pdf provided in the external resources for further information on the mapping of the original variables on the harmonized ones, in addition to more indications on the variables' availability in both survey years and relevant comments.

Geographic coverage

National coverage: Covering a sample of urban, rural and metropolitan areas in all the governorates including those in Kurdistan Region.

Analysis unit

1- Household/family. 2- Individual/person.

Universe

The survey was carried out over a full year covering all governorates including those in Kurdistan Region.

Kind of data

Sample survey data [ssd]

Sampling procedure

----> Design:

Sample size was (25488) household for the whole Iraq, 216 households for each district of 118 districts, 2832 clusters each of which includes 9 households distributed on districts and governorates for rural and urban.

----> Sample frame:

Listing and numbering results of 2009-2010 Population and Housing Survey were adopted in all the governorates including Kurdistan Region as a frame to select households, the sample was selected in two stages: Stage 1: Primary sampling unit (blocks) within each stratum (district) for urban and rural were systematically selected with probability proportional to size to reach 2832 units (cluster). Stage two: 9 households from each primary sampling unit were selected to create a cluster, thus the sample size of total survey clusters was 25488 households distributed on the governorates, 216 households in each district.

----> Sampling Stages:

In each district, the sample was selected in two stages: Stage 1: based on 2010 listing and numbering frame 24 sample points were selected within each stratum through systematic sampling with probability proportional to size, in addition to the implicit breakdown urban and rural and geographic breakdown (sub-district, quarter, street, county, village and block). Stage 2: Using households as secondary sampling units, 9 households were selected from each sample point using systematic equal probability sampling. Sampling frames of each stages can be developed based on 2010 building listing and numbering without updating household lists. In some small districts, random selection processes of primary sampling may lead to select less than 24 units therefore a sampling unit is selected more than once , the selection may reach two cluster or more from the same enumeration unit when it is necessary.

Mode of data collection

Face-to-face [f2f]

Research instrument

----> Preparation:

The questionnaire of 2006 survey was adopted in designing the questionnaire of 2012 survey on which many revisions were made. Two rounds of pre-test were carried out. Revision were made based on the feedback of field work team, World Bank consultants and others, other revisions were made before final version was implemented in a pilot survey in September 2011. After the pilot survey implemented, other revisions were made in based on the challenges and feedbacks emerged during the implementation to implement the final version in the actual survey.

----> Questionnaire Parts:

The questionnaire consists of four parts each with several sections: Part 1: Socio – Economic Data: - Section 1: Household Roster - Section 2: Emigration - Section 3: Food Rations - Section 4: housing - Section 5: education - Section 6: health - Section 7: Physical measurements - Section 8: job seeking and previous job

Part 2: Monthly, Quarterly and Annual Expenditures: - Section 9: Expenditures on Non – Food Commodities and Services (past 30 days). - Section 10 : Expenditures on Non – Food Commodities and Services (past 90 days). - Section 11: Expenditures on Non – Food Commodities and Services (past 12 months). - Section 12: Expenditures on Non-food Frequent Food Stuff and Commodities (7 days). - Section 12, Table 1: Meals Had Within the Residential Unit. - Section 12, table 2: Number of Persons Participate in the Meals within Household Expenditure Other Than its Members.

Part 3: Income and Other Data: - Section 13: Job - Section 14: paid jobs - Section 15: Agriculture, forestry and fishing - Section 16: Household non – agricultural projects - Section 17: Income from ownership and transfers - Section 18: Durable goods - Section 19: Loans, advances and subsidies - Section 20: Shocks and strategy of dealing in the households - Section 21: Time use - Section 22: Justice - Section 23: Satisfaction in life - Section 24: Food consumption during past 7 days

Part 4: Diary of Daily Expenditures: Diary of expenditure is an essential component of this survey. It is left at the household to record all the daily purchases such as expenditures on food and frequent non-food items such as gasoline, newspapers…etc. during 7 days. Two pages were allocated for recording the expenditures of each day, thus the roster will be consists of 14 pages.

Cleaning operations

----> Raw Data:

Data Editing and Processing: To ensure accuracy and consistency, the data were edited at the following stages: 1. Interviewer: Checks all answers on the household questionnaire, confirming that they are clear and correct. 2. Local Supervisor: Checks to make sure that questions has been correctly completed. 3. Statistical analysis: After exporting data files from excel to SPSS, the Statistical Analysis Unit uses program commands to identify irregular or non-logical values in addition to auditing some variables. 4. World Bank consultants in coordination with the CSO data management team: the World Bank technical consultants use additional programs in SPSS and STAT to examine and correct remaining inconsistencies within the data files. The software detects errors by analyzing questionnaire items according to the expected parameter for each variable.

----> Harmonized Data:

• The SPSS package is used to harmonize the Iraq Household Socio Economic Survey (IHSES) 2007 with Iraq Household Socio Economic Survey (IHSES) 2012.
• The harmonization process starts with raw data files received from the Statistical Office.
• A program is generated for each dataset to create harmonized variables.
• Data is saved on the household and individual level, in SPSS and then converted to STATA, to be disseminated.

Response rate

Iraq Household Socio Economic Survey (IHSES) reached a total of 25488 households. Number of households refused to response was 305, response rate was 98.6%. The highest interview rates were in Ninevah and Muthanna (100%) while the lowest rates were in Sulaimaniya (92%).

These data were retrospectively collected for the REACH study, funded by the NHMRC GNT1076006, in 11 Australian hospitals. This single dataset is the combined costs associated with the REACH cleaning bundle implementation forthe period between 27/06/2016 and 30/07/2017.

Software used to analyse data: R, Microsoft Excel

Extract the list of locations of the 1,424 cameras as described in the Official Bulletin of the City of Paris of 1 February 2019 To make this list available according to the principles of Open Data (open license, standard data format) To geotagger these locations in order to view them on a map (in progress, 707 of 1424, see map) There are actually two lists of locations: — Annex 1 (1 391 pitches): the Video Protection Plan for the Police Prefecture — Annex 2 (890 locations): the video protection plan for Paris, list of cameras visible by authorised agents of the City of Paris These two lists have many locations in common. NextInpact summarised the situation in this article. The source code is available here: https://github.com/ColinMaudry/geo-videoprotection-paris Modus operandi Convert PDF to MS Excel file using online tool PDFtables.com Cleaning non-data text in LibreOffice

Data content: foreign economy and trade_ Total import and export of goods (1991-2021) Data source and processing method: The original data of foreign trade and investment of the third pole (China region) from 2015 to 2021 were obtained from the official website of the World Bank and Sina.com, and the data set of foreign trade and investment of the third pole (China region) from 1991 to 2021 was obtained through data sorting, screening and cleaning. The data started from 1991 to 2021 in Microsoft Excel (xls) format. Data quality description: excellent Data application achievements and prospects: provide effective reference as socio-economic data

The Survey of Household Spending provides detailed information on household expenditures, dwelling characteristics, and ownership of household equipment such as appliances, audio and video equipment, and vehicles. Expenditure categories include: shelter expenses, furnishings and equipment, cost of running the home, communications, child care, food, alcohol and tobacco products, clothing, gifts, medical and health care, transportation and travel, recreation, reading materials, education, taxes, insurance payments and pension contributions. Dwelling characteristics include: type of dwelling, repairs needed (major, minor, none), tenure, year of move, period of construction, number of rooms, number of bathrooms, principal heating equipment and fuel, age of principal heating equipment, principal heating fuel for hot water, and principal cooking fuel. Household equipment includes: washing machines, dryers, dishwashers, refrigerators, freezers, air conditioners, telephones, cellular phones, compact disc players, cablevision, video cassette recorders, computers, modems, internet use from home, televisions, and vehicles. Characteristics of the household, reference person, and spouse of reference person are also provided. The annual Survey of Household Spending replaces the Family Expenditure (FAMEX) Survey which was conducted approximately every four years. The last FAMEX survey was for the reference year 1996. Content from the former annual Household Facilities and Equipment (HFE) Survey is also included in the Survey of Household Spending. The last HFE survey was for the reference year 1998. Please note that when comparing data to the HIFE files, HIFE Reference Year refers to the year in which the data was collected - based on previous year's income and spending. Therefore HIFE Reference Year 1998 collected data based on the 1997 income year. Conversly, the SHS (Survey of Household Spending) uses the term Reference Year to indicate the year of the income and spending rather than the year the data was collected. Therefore, in SHS, the 1998 Reference Year refers to 1998 income and spending, not the year (1999) in which the data was collected. For current Survey of Household Spending data refer to Statistics Canada Access data here

The Survey of Household Spending provides detailed information on household expenditures, dwelling characteristics, and ownership of household equipment. Conducted since 1997, the Survey of Household Spending integrates most of the content found in the Family Expenditure Survey (FAMEX) (1969-1996) and the Household Facilities and Equipment Survey (apart of the Survey of Consumer Finances) (1973-1998). Many data from these two surveys are comparable to the Survey of Household Spending d ata. However, some differences related to methodology, to data quality and to definitions must be considered before comparing these data. Detailed information was collected about expenditures for consumer goods and services, changes in assets, mortgages and other loans, and annual income. Information was also collected about dwelling characteristics (e.g., type and age of heating equipment) and household equipment (e.g., appliances, communications equipment, and vehicles).