Sample data for exercises in Further Adventures in Data Cleaning.

Access and clean an open source herbarium dataset using Excel or RStudio.

Dataset

This dataset was created by Luis Lira

Contents

Dataset

This dataset was created by George M122

Contents

Corpus consisting of 10,000 Facebook posts manually annotated on sentiment (2,587 positive, 5,174 neutral, 1,991 negative and 248 bipolar posts). The archive contains data and statistics in an Excel file (FBData.xlsx) and gold data in two text files with posts (gold-posts.txt) and labels (gols-labels.txt) on corresponding lines.

We describe a bibliometric network characterizing co-authorship collaborations in the entire Italian academic community. The network, consisting of 38,220 nodes and 507,050 edges, is built upon two distinct data sources: faculty information provided by the Italian Ministry of University and Research and publications available in Semantic Scholar. Both nodes and edges are associated with a large variety of semantic data, including gender, bibliometric indexes, authors' and publications' research fields, and temporal information. While linking data between the two original sources posed many challenges, the network has been carefully validated to assess its reliability and to understand its graph-theoretic characteristics. By resembling several features of social networks, our dataset can be profitably leveraged in experimental studies in the wide social network analytics domain as well as in more specific bibliometric contexts. , The proposed network is built starting from two distinct data sources:

the entire dataset dump from Semantic Scholar (with particular emphasis on the authors and papers datasets) the entire list of Italian faculty members as maintained by Cineca (under appointment by the Italian Ministry of University and Research).

By means of a custom name-identity recognition algorithm (details are available in the accompanying paper published in Scientific Data), the names of the authors in the Semantic Scholar dataset have been mapped against the names contained in the Cineca dataset and authors with no match (e.g., because of not being part of an Italian university) have been discarded. The remaining authors will compose the nodes of the network, which have been enriched with node-related (i.e., author-related) attributes. In order to build the network edges, we leveraged the papers dataset from Semantic Scholar: specifically, any two authors are said to be connected if there is at least one pap..., , # Data cleaning and enrichment through data integration: networking the Italian academia

https://doi.org/10.5061/dryad.wpzgmsbwj

Description of the data and file structure

This repository contains two main data files:

• edge_data_AGG.csv, the full network in comma-separated edge list format (this file contains mainly temporal co-authorship information);
• Coauthorship_Network_AGG.graphml, the full network in GraphML format.Â

along with several supplementary data, listed below, useful only to build the network (i.e., for reproducibility only):

• University-City-match.xlsx, an Excel file that maps the name of a university against the city where its respective headquarter is located;
• Areas-SS-CINECA-match.xlsx, an Excel file that maps the research areas in Cineca against the research areas in Semantic Scholar.

Description of the main data files

The Coauthorship_Network_AGG.graphml is intended to be the core file which c...

Global Cleaning Robot Market size was valued at USD 4.19 billion in 2022 and is poised to grow from USD 4.97 billion in 2023 to USD 12.81 billion by 2031, growing at a CAGR of 22.9% in the forecast period (2024-2031).

Dataset, GDB, and Online Map created by Renee Haley, NMCDC, May 2023 DATA ACQUISITION PROCESS

Scope and purpose of project: New Mexico is struggling to maintain its healthcare workforce, particularly in Rural areas. This project was undertaken with the intent of looking at flows of healthcare workers into and out of New Mexico at the most granular geographic level possible. This dataset, in combination with others (such as housing cost and availability data) may help us understand where our healthcare workforce is relocating and why.

The most relevant and detailed data on workforce indicators in the United States is housed by the Census Bureau's Longitudinal Employer-Household Dynamics, LEHD, System. Information on this system is available here:

https://lehd.ces.census.gov/

The Job-to-Job flows explorer within this system was used to download the data. Information on the J2J explorer can ve found here:

https://j2jexplorer.ces.census.gov/explore.html#1432012

The dataset was built from data queried with the LED Extraction Tool, which allows for the query of more intersectional and detailed data than the explorer. This is a link to the LED extraction tool:

https://ledextract.ces.census.gov/

The geographies used are US Metro areas as determined by the Census, (N=389). The shapefile is named lehd_shp_gb.zip, and can be downloaded under this section of the following webpage: 5.5. Job-to-Job Flow Geographies, 5.5.1. Metropolitan (Complete). A link to the download site is available below:

https://lehd.ces.census.gov/data/schema/j2j_latest/lehd_shapefiles.html

DATA CLEANING PROCESS

This dataset was built from 8 non intersectional datasets downloaded from the LED Extraction Tool.

Separate datasets were downloaded in order to obtain detailed information on the race, ethnicity, and educational attainment levels of healthcare workers and where they are migrating.

Datasets included information for the four separate quarters of 2021. It was not possible to download annual data, only quarterly. Quarterly data was summed in a later step to derive annual totals for 2021.

4 datasets for healthcare workers moving OUT OF New Mexico, with details on race, ethnicity, and educational attainment, were downloaded. 1 contained information on educational attainment, 2 contained information on 7 racial categories identifying as non- Hispanic, 3 contained information on those same 7 categories also identifying as Hispanic, and 4 contained information for workers identifying as white and Hispanic.

4 datasets for healthcare worker moving INTO New Mexico, with details on race, ethnicity, and educational attainment, were downloaded with the same details outlined above.

Each dataset was cleaned according to Data Template which kept key attributes and discarded excess information. Within each dataset, the J2J Indicators reflecting 6 different types of job migration were totaled in order to simplify analysis, as this information was not needed in detail.

After cleaning, each set of 4 datasets for workers moving INTO New Mexico were joined. The process was repeated for workers moving OUT OF New Mexico. This resulted 2 main datasets.

These 2 main datasets still listed all of the variables by each quarter of 2021. Because of this the data was split in JMP, so that attributes of educational attainment, race and ethnicity, of workers migrating by quarter were moved from rows to columns. After this, summary columns for the year of 2021 were derived. This resulted in totals columns for workers identifying as: 6 separate races and all ethnicities, all races and Hispanic, white-Hispanic, and workers of 6 different education levels, reflecting how many workers of each indicator migrated to and from metro areas in New Mexico in 2021.

The data split transposed duplicate rows reflecting differing worker attributes within the same metro area, resulting in one row for each metro area and reflecting the attributes in columns, thus resulting in a mappable dataset.

The 2 datasets were joined (on Metro Area) resulting in one master file containing information on healthcare workers entering and leaving New Mexico.

Rows (N=389) reflect all of the metro areas across the US, and each state. Rows include the 5 metro areas within New Mexico, and New Mexico State.

Columns (N=99) contain information on worker race, ethnicity and educational attainment, specific to each metro area in New Mexico.

78 of these rows reflect workers of specific attributes moving OUT OF the 5 specific Metro Areas in New Mexico and totals for NM State. This level of detail is intended for analyzing who is leaving what area of New Mexico, where they are going to, and why.

13 Columns reflect each worker attribute for healthcare workers moving INTO New Mexico by race, ethnicity and education level. Because all 5 metro areas and New Mexico state are contained in the rows, this information for incoming workers is available by metro area and at the state level - there is less possability for mapping these attributes since it was not realistic or possible to create a dataset reflecting all of these variables for every healthcare worker from every metro area in the US also coming into New Mexico (that dataset would have over 1,000 columns and be unmappable). Therefore this dataset is easier to utilize in looking at why workers are leaving the state but also includes detailed information on who is coming in.

The remaining 8 columns contain geographic information.

GIS AND MAPPING PROCESS

The master file was opened in Arc GIS Pro and the Shapefile of US Metro Areas was also imported

The excel file was joined to the shapefile by Metro Area Name as they matched exactly

The resulting layer was exported as a GDB in order to retain null values which would turn to zeros if exported as a shapefile.

This GDB was uploaded to Arc GIS Online, Aliases were inserted as column header names, and the layer was visualized as desired.

SYSTEMS USED

MS Excel was used for data cleaning, summing NM state totals, and summing quarterly to annual data.

JMP was used to transpose, join, and split data.

ARC GIS Desktop was used to create the shapefile uploaded to NMCDC's online platform.

VARIABLE AND RECODING NOTES

Summary of variables selected for datasets downloaded focused on educational attainment:

J2J Flows by Educational Attainment

Summary of variables selected for datasets downloaded focused on race and ethnicity:

J2J Flows by Race and Ethnicity

Note: Variables in Datasets 1 through 4 downloaded twice, once for workers coming into New Mexico and once for those leaving NM. VARIABLE: LEHD VARIABLE DEFINITION LEHD VARIABLE NOTES DETAILS OR URL FOR RAW DATA DOWNLOAD

Geography Type - State Origin and Destination State

Data downloaded for worker migration into and out of all US States

Geography Type - Metropolitan Areas Origin and Dest Metro Area

Data downloaded for worker migration into and out of all US Metro Areas

NAICS sectors North American Industry Classification System Under Firm Characteristics Only downloaded for Healthcare and Social Assistance Sectors

Other Firm Characteristics No Firm Age / Size Detail Under Firm Characteristics Downloaded data on all firm ages, sizes, and other details.

Worker Characteristics Education, Race, Ethnicity

Non Intersectional data aside from Race / Ethnicity data.

Sex Gender

0 - All Sexes Selected

Age Age

A00 All Ages (14-99)

Education Education Level E0, E1, E2, E3, 34, E5 E0 - All Education Categories, E1 - Less than high school, E2 - High school or equivalent, no college, E3 - Some college or Associate’s degree, E4 - Bachelor's degree or advanced degree, E5 - Educational attainment not available (workers aged 24 or younger)

Dataset 1 All Education Levels, E1, E2, E3, E4, and E5

RACE

A0, A1, A2, A3, A4, A5 OPTIONS: A0 All Races, A1 White Alone, A2 Black or African American Alone, A3 American Indian or Alaska Native Alone, A4 Asian Alone, A5 Native Hawaiian or Other Pacific Islander Alone, SDA7 Two or More Race Groups

ETHNICITY

A0, A1, A2 OPTIONS: A0 All Ethnicities, A1 Not Hispanic or Latino, A2 Hispanic or Latino

Dataset 2 All Races (A0) and All Ethnicities (A0)

Dataset 3 6 Races (A1 through A5) and All Ethnicities (A0)

Dataset 4 White (A1) and Hispanic or Latino (A1)

Quarter Quarter and Year

Data from all quarters of 2021 to sum into annual numbers; yearly data was not available

Employer type Sector: Private or Governmental

Query included all healthcare sector workflows from all employer types and firm sizes from every quarter of 2021

J2J indicator categories Detailed types of job migration

All options were selected for all datasets and totaled: AQHire, AQHireS, EE, EES, J2J, J2JS. Counts were selected vs. earnings, and data was not seasonally adjusted (unavailable).

NOTES AND RESOURCES

The following resources and documentation were used to navigate the LEHD and J2J Worker Flows system and to answer questions about variables:

https://lehd.ces.census.gov/data/schema/j2j_latest/lehd_public_use_schema.html

https://www.census.gov/history/www/programs/geography/metropolitan_areas.html

https://lehd.ces.census.gov/data/schema/j2j_latest/lehd_csv_naming.html

Statewide (New

This data set supports the journal paper "Manipulating the consequences of tests: How Shanghai teens react to different consequences", published in Educational Research and Evaluation, v26 (n5-6), pp.221-251.The data were obtained to test the impact of different levels of consequence for taking a test on student test-taking effort. The data are part of the PhD project of Anran Zhao, supervised by Brown & Meissel.The data set is in MS Excel format. Sheet 1 provides an anonymous wide-format data set post-cleaning and missing value analysis of the data.Sheet 2 provides a description of each variable.

Abstract

The General Household Survey-Panel (GHS-Panel) is implemented in collaboration with the World Bank Living Standards Measurement Study (LSMS) team as part of the Integrated Surveys on Agriculture (ISA) program. The objectives of the GHS-Panel include the development of an innovative model for collecting agricultural data, interinstitutional collaboration, and comprehensive analysis of welfare indicators and socio-economic characteristics. The GHS-Panel is a nationally representative survey of approximately 5,000 households, which are also representative of the six geopolitical zones. The 2023/24 GHS-Panel is the fifth round of the survey with prior rounds conducted in 2010/11, 2012/13, 2015/16 and 2018/19. The GHS-Panel households were visited twice: during post-planting period (July - September 2023) and during post-harvest period (January - March 2024).

Geographic coverage

National

Analysis unit

• Households • Individuals • Agricultural plots • Communities

Universe

The survey covered all de jure households excluding prisons, hospitals, military barracks, and school dormitories.

Kind of data

Sample survey data [ssd]

Sampling procedure

The original GHS‑Panel sample was fully integrated with the 2010 GHS sample. The GHS sample consisted of 60 Primary Sampling Units (PSUs) or Enumeration Areas (EAs), chosen from each of the 37 states in Nigeria. This resulted in a total of 2,220 EAs nationally. Each EA contributed 10 households to the GHS sample, resulting in a sample size of 22,200 households. Out of these 22,200 households, 5,000 households from 500 EAs were selected for the panel component, and 4,916 households completed their interviews in the first wave.

After nearly a decade of visiting the same households, a partial refresh of the GHS‑Panel sample was implemented in Wave 4 and maintained for Wave 5. The refresh was conducted to maintain the integrity and representativeness of the sample. The refresh EAs were selected from the same sampling frame as the original GHS‑Panel sample in 2010. A listing of households was conducted in the 360 EAs, and 10 households were randomly selected in each EA, resulting in a total refresh sample of approximately 3,600 households.

In addition to these 3,600 refresh households, a subsample of the original 5,000 GHS‑Panel households from 2010 were selected to be included in the new sample. This “long panel” sample of 1,590 households was designed to be nationally representative to enable continued longitudinal analysis for the sample going back to 2010. The long panel sample consisted of 159 EAs systematically selected across Nigeria’s six geopolitical zones.

The combined sample of refresh and long panel EAs in Wave 5 that were eligible for inclusion consisted of 518 EAs based on the EAs selected in Wave 4. The combined sample generally maintains both the national and zonal representativeness of the original GHS‑Panel sample.

Sampling deviation

Although 518 EAs were identified for the post-planting visit, conflict events prevented interviewers from visiting eight EAs in the North West zone of the country. The EAs were located in the states of Zamfara, Katsina, Kebbi and Sokoto. Therefore, the final number of EAs visited both post-planting and post-harvest comprised 157 long panel EAs and 354 refresh EAs. The combined sample is also roughly equally distributed across the six geopolitical zones.

Mode of data collection

Computer Assisted Personal Interview [capi]

Research instrument

The GHS-Panel Wave 5 consisted of three questionnaires for each of the two visits. The Household Questionnaire was administered to all households in the sample. The Agriculture Questionnaire was administered to all households engaged in agricultural activities such as crop farming, livestock rearing, and other agricultural and related activities. The Community Questionnaire was administered to the community to collect information on the socio-economic indicators of the enumeration areas where the sample households reside.

GHS-Panel Household Questionnaire: The Household Questionnaire provided information on demographics; education; health; labour; childcare; early child development; food and non-food expenditure; household nonfarm enterprises; food security and shocks; safety nets; housing conditions; assets; information and communication technology; economic shocks; and other sources of household income. Household location was geo-referenced in order to be able to later link the GHS-Panel data to other available geographic data sets (forthcoming).

GHS-Panel Agriculture Questionnaire: The Agriculture Questionnaire solicited information on land ownership and use; farm labour; inputs use; GPS land area measurement and coordinates of household plots; agricultural capital; irrigation; crop harvest and utilization; animal holdings and costs; household fishing activities; and digital farming information. Some information is collected at the crop level to allow for detailed analysis for individual crops.

GHS-Panel Community Questionnaire: The Community Questionnaire solicited information on access to infrastructure and transportation; community organizations; resource management; changes in the community; key events; community needs, actions, and achievements; social norms; and local retail price information.

The Household Questionnaire was slightly different for the two visits. Some information was collected only in the post-planting visit, some only in the post-harvest visit, and some in both visits.

The Agriculture Questionnaire collected different information during each visit, but for the same plots and crops.

The Community Questionnaire collected prices during both visits, and different community level information during the two visits.

Cleaning operations

CAPI: Wave five exercise was conducted using Computer Assisted Person Interview (CAPI) techniques. All the questionnaires (household, agriculture, and community questionnaires) were implemented in both the post-planting and post-harvest visits of Wave 5 using the CAPI software, Survey Solutions. The Survey Solutions software was developed and maintained by the Living Standards Measurement Unit within the Development Economics Data Group (DECDG) at the World Bank. Each enumerator was given a tablet which they used to conduct the interviews. Overall, implementation of survey using Survey Solutions CAPI was highly successful, as it allowed for timely availability of the data from completed interviews.

DATA COMMUNICATION SYSTEM: The data communication system used in Wave 5 was highly automated. Each field team was given a mobile modem which allowed for internet connectivity and daily synchronization of their tablets. This ensured that head office in Abuja had access to the data in real-time. Once the interview was completed and uploaded to the server, the data was first reviewed by the Data Editors. The data was also downloaded from the server, and Stata dofile was run on the downloaded data to check for additional errors that were not captured by the Survey Solutions application. An excel error file was generated following the running of the Stata dofile on the raw dataset. Information contained in the excel error files were then communicated back to respective field interviewers for their action. This monitoring activity was done on a daily basis throughout the duration of the survey, both in the post-planting and post-harvest.

DATA CLEANING: The data cleaning process was done in three main stages. The first stage was to ensure proper quality control during the fieldwork. This was achieved in part by incorporating validation and consistency checks into the Survey Solutions application used for the data collection and designed to highlight many of the errors that occurred during the fieldwork.

The second stage cleaning involved the use of Data Editors and Data Assistants (Headquarters in Survey Solutions). As indicated above, once the interview is completed and uploaded to the server, the Data Editors review completed interview for inconsistencies and extreme values. Depending on the outcome, they can either approve or reject the case. If rejected, the case goes back to the respective interviewer’s tablet upon synchronization. Special care was taken to see that the households included in the data matched with the selected sample and where there were differences, these were properly assessed and documented. The agriculture data were also checked to ensure that the plots identified in the main sections merged with the plot information identified in the other sections. Additional errors observed were compiled into error reports that were regularly sent to the teams. These errors were then corrected based on re-visits to the household on the instruction of the supervisor. The data that had gone through this first stage of cleaning was then approved by the Data Editor. After the Data Editor’s approval of the interview on Survey Solutions server, the Headquarters also reviews and depending on the outcome, can either reject or approve.

The third stage of cleaning involved a comprehensive review of the final raw data following the first and second stage cleaning. Every variable was examined individually for (1) consistency with other sections and variables, (2) out of range responses, and (3) outliers. However, special care was taken to avoid making strong assumptions when resolving potential errors. Some minor errors remain in the data where the diagnosis and/or solution were unclear to the data cleaning team.

Response

Data and R code used for the analysis of data for the publication: Coumoundouros et al., Cognitive behavioural therapy self-help intervention preferences among informal caregivers of adults with chronic kidney disease: an online cross-sectional survey. BMC Nephrology

Summary of study

An online cross-sectional survey for informal caregivers (e.g. family and friends) of people living with chronic kidney disease in the United Kingdom. Study aimed to examine informal caregivers' cognitive behavioural therapy self-help intervention preferences, and describe the caregiving situation (e.g. types of care activities) and informal caregiver's mental health (depression, anxiety and stress symptoms).

Participants were eligible to participate if they were at least 18 years old, lived in the United Kingdom, and provided unpaid care to someone living with chronic kidney disease who was at least 18 years old.

The online survey included questions regarding (1) informal caregiver's characteristics; (2) care recipient's characteristics; (3) intervention preferences (e.g. content, delivery format); and (4) informal caregiver's mental health. Informal caregiver's mental health was assessed using the 21 item Depression, Anxiety, and Stress Scale (DASS-21), which is composed of three subscales measuring depression, anxiety, and stress, respectively.

Sixty-five individuals participated in the survey.

See the published article for full study details.

Description of uploaded files

1. ENTWINE_ESR14_Kidney Carer Survey Data_FULL_2022-08-30: Excel file with the complete, raw survey data. Note: the first half of participant's postal codes was collected, however this data was removed from the uploaded dataset to ensure participant anonymity.

2. ENTWINE_ESR14_Kidney Carer Survey Data_Clean DASS-21 Data_2022-08-30: Excel file with cleaned data for the DASS-21 scale. Data cleaning involved imputation of missing data if participants were missing data for one item within a subscale of the DASS-21. Missing values were imputed by finding the mean of all other items within the relevant subscale.

3. ENTWINE_ESR14_Kidney Carer Survey_KEY_2022-08-30: Excel file with key linking item labels in uploaded datasets with the corresponding survey question.

4. R Code for Kidney Carer Survey_2022-08-30: R file of R code used to analyse survey data.

5. R code for Kidney Carer Survey_PDF_2022-08-30: PDF file of R code used to analyse survey data.

This dataset contains the results of a survey about the use of open government data applied to public agents working in public institutions in Brazil. It has two sets, one with questionnaire responses and metadata and the second with a coding table with interview extracts: 1) In the first dataset, each row holds a response to a questionnaire about the public agent's perceptions of the use and reuse of open government data in Brazilian public institutions. Columns store the questionnaire questions. Data were collected between 8 June and 13 July 2021, and this sample is composed of responses from 40 federal, state, and municipal public administrators. Thus, this dataset contains 40 rows and 158 columns. Data were collected on the LimeSurvey platform, where it was screened for missing values and incomplete responses. After cleaning, data were exported to Excel in tabular format. Questionnaire responses are provided in two files ResultsSurvey_OGDUseBRPubInstitutions_DataSet_PT and ResultsSurvey_OGDUseBRPubInstitutions_DataSet_EN. They contain the same information in Portuguese and English. 2) The second dataset records the code table of the interviews about the benefits, barriers, enablers, and drivers of open government data (OGD) use in Brazilian public institutions. A questionnaire applied to public agents working in Brazilian public institutions was followed up by interviews to broaden an understanding of the use of OGD. Nine interviews were conducted between May 17-31, 2022. This dataset represents the perspective of these public agents. The dataset contains 97 lines and six columns. Each row of the dataset lists the factor code used in the questionnaire, the factor descriptions in Portuguese and English, the interviewee code, the transcription extract of an interviewee narration collected in Portuguese, and the English translation. After collection in Portuguese, interviews were automatically transcribed using the NVivo Transcription software. Then, they were anonymized, and a human reviewed the transcriptions. Interviews were coded using NVivo and used the questionnaire factors to guide coding. Coded extracts were translated to English using Google and Microsoft translators. Then, translated extracts were revised by a human and were used for reporting. The coding table was exported to Excel. Interviews extracts are provided in one file, InterviewsExtracts_OGDUseBR_PublicInstitutions_Dataset.

The National Health and Nutrition Examination Survey (NHANES) provides data on the health and environmental exposure of the non-institutionalized US population. Such data have considerable potential to understand how the environment and behaviors impact human health. These data are also currently leveraged to answer public health questions such as prevalence of disease. However, these data need to first be processed before new insights can be derived through large-scale analyses. NHANES data are stored across hundreds of files with multiple inconsistencies. Correcting such inconsistencies takes systematic cross examination and considerable efforts but is required for accurately and reproducibly characterizing the associations between the exposome and diseases (e.g., cancer mortality outcomes). Thus, we developed a set of curated and unified datasets and accompanied code by merging 614 separate files and harmonizing unrestricted data across NHANES III (1988-1994) and Continuous (1999-2018), totaling 134,310 participants and 4,740 variables. The variables convey 1) demographic information, 2) dietary consumption, 3) physical examination results, 4) occupation, 5) questionnaire items (e.g., physical activity, general health status, medical conditions), 6) medications, 7) mortality status linked from the National Death Index, 8) survey weights, 9) environmental exposure biomarker measurements, and 10) chemical comments that indicate which measurements are below or above the lower limit of detection. We also provide a data dictionary listing the variables and their descriptions to help researchers browse the data. We also provide R markdown files to show example codes on calculating summary statistics and running regression models to help accelerate high-throughput analysis of the exposome and secular trends on cancer mortality. csv Data Record: The curated NHANES datasets and the data dictionaries includes 13 .csv files and 1 excel file. The curated NHANES datasets involves 10 .csv formatted files, one for each module and labeled as the following: 1) mortality, 2) dietary, 3) demographics, 4) response, 5) medications, 6) questionnaire, 7) chemicals, 8) occupation, 9) weights, and 10) comments. The eleventh file is a dictionary that lists the variable name, description, module, category, units, CAS Number, comment use, chemical family, chemical family shortened, number of measurements, and cycles available for all 4,740 variables in NHANES ("dictionary_nhanes.csv"). The 12th csv file contains the harmonized categories for the categorical variables ("dictionary_harmonized_categories.csv"). The 13th file contains the dictionary for descriptors on the drugs codes (“dictionary_drug_codes.csv”). The 14th file is an excel file that contains the cleaning documentation, which records all the inconsistencies for all affected variables to help curate each of the NHANES datasets (“nhanes_inconsistencies_documentation.xlsx”). R Data Record: For researchers who want to conduct their analysis in the R programming language, the curated NHANES datasets and the data dictionaries can be downloaded as a .zip file which include an .RData file and an .R file. We provided an .RData file that contains all the aforementioned datasets as R data objects (“w - nhanes_1988_2018.RData”). Also in this .RData file, we make available all R scripts on customized functions that were written to curate the data. We also provide an .R file that shows how we used the customized functions (i.e. our pipeline) to curate the data (“m - nhanes_1988_2018.R”).

Abstract

The main objectives of the survey were: - To obtain weights for the revision of the Consumer Price Index (CPI) for Funafuti; - To provide information on the nature and distribution of household income, expenditure and food consumption patterns; - To provide data on the household sector's contribution to the National Accounts - To provide information on economic activity of men and women to study gender issues - To undertake some poverty analysis

Geographic coverage

National, including Funafuti and Outer islands

Analysis unit

• Household
• individual

Universe

All the private household are included in the sampling frame. In each household selected, the current resident are surveyed, and people who are usual resident but are currently away (work, health, holydays reasons, or border student for example. If the household had been residing in Tuvalu for less than one year: - but intend to reside more than 12 months => The household is included - do not intend to reside more than 12 months => out of scope

Kind of data

Sample survey data [ssd]

Sampling procedure

It was decided that 33% (one third) sample was sufficient to achieve suitable levels of accuracy for key estimates in the survey. So the sample selection was spread proportionally across all the island except Niulakita as it was considered too small. For selection purposes, each island was treated as a separate stratum and independent samples were selected from each. The strategy used was to list each dwelling on the island by their geographical position and run a systematic skip through the list to achieve the 33% sample. This approach assured that the sample would be spread out across each island as much as possible and thus more representative.

For details please refer to Table 1.1 of the Report.

Sampling deviation

Only the island of Niulakita was not included in the sampling frame, considered too small.

Mode of data collection

Face-to-face [f2f]

Research instrument

There were three main survey forms used to collect data for the survey. Each question are writen in English and translated in Tuvaluan on the same version of the questionnaire. The questionnaires were designed based on the 2004 survey questionnaire.

HOUSEHOLD FORM - composition of the household and demographic profile of each members - dwelling information - dwelling expenditure - transport expenditure - education expenditure - health expenditure - land and property expenditure - household furnishing - home appliances - cultural and social payments - holydays/travel costs - Loans and saving - clothing - other major expenditure items

INDIVIDUAL FORM - health and education - labor force (individu aged 15 and above) - employment activity and income (individu aged 15 and above): wages and salaries, working own business, agriculture and livestock, fishing, income from handicraft, income from gambling, small scale activies, jobs in the last 12 months, other income, childreen income, tobacco and alcohol use, other activities, and seafarer

DIARY (one diary per week, on a 2 weeks period, 2 diaries per household were required) - All kind of expenses - Home production - food and drink (eaten by the household, given away, sold) - Goods taken from own business (consumed, given away) - Monetary gift (given away, received, winning from gambling) - Non monetary gift (given away, received, winning from gambling)

Questionnaire Design Flaws Questionnaire design flaws address any problems with the way questions were worded which will result in an incorrect answer provided by the respondent. Despite every effort to minimize this problem during the design of the respective survey questionnaires and the diaries, problems were still identified during the analysis of the data. Some examples are provided below:

Gifts, Remittances & Donations Collecting information on the following: - the receipt and provision of gifts - the receipt and provision of remittances - the provision of donations to the church, other communities and family occasions is a very difficult task in a HIES. The extent of these activities in Tuvalu is very high, so every effort should be made to address these activities as best as possible. A key problem lies in identifying the best form (questionnaire or diary) for covering such activities. A general rule of thumb for a HIES is that if the activity occurs on a regular basis, and involves the exchange of small monetary amounts or in-kind gifts, the diary is more appropriate. On the other hand, if the activity is less infrequent, and involves larger sums of money, the questionnaire with a recall approach is preferred. It is not always easy to distinguish between the two for the different activities, and as such, both the diary and questionnaire were used to collect this information. Unfortunately it probably wasn?t made clear enough as to what types of transactions were being collected from the different sources, and as such some transactions might have been missed, and others counted twice. The effects of these problems are hopefully minimal overall.

Defining Remittances Because people have different interpretations of what constitutes remittances, the questionnaire needs to be very clear as to how this concept is defined in the survey. Unfortunately this wasn?t explained clearly enough so it was difficult to distinguish between a remittance, which should be of a more regular nature, and a one-off monetary gift which was transferred between two households.

Business Expenses Still Recorded The aim of the survey is to measure "household" expenditure, and as such, any expenditure made by a household for an item or service which was primarily used for a business activity should be excluded. It was not always clear in the questionnaire that this was the case, and as such some business expenses were included. Efforts were made during data cleaning to remove any such business expenses which would impact significantly on survey results.

Purchased goods given away as a gift When a household makes a gift donation of an item it has purchased, this is recorded in section 5 of the diary. Unfortunately it was difficult to know how to treat these items as it was not clear as to whether this item had been recorded already in section 1 of the diary which covers purchases. The decision was made to exclude all information of gifts given which were considered to be purchases, as these items were assumed to have already been recorded already in section 1. Ideally these items should be treated as a purchased gift given away, which in turn is not household consumption expenditure, but this was not possible.

Some key items missed in the Questionnaire Although not a big issue, some key expenditure items were omitted from the questionnaire when it would have been best to collect them via this schedule. A key example being electric fans which many households in Tuvalu own.

Cleaning operations

Consistency of the data: - each questionnaire was checked by the supervisor during and after the collection - before data entry, all the questionnaire were coded - the CSPRo data entry system included inconsistency checks which allow the NSO staff to point some errors and to correct them with imputation estimation from their own knowledge (no time for double entry), 4 data entry operators. - after data entry, outliers were identified in order to check their consistency.

All data entry, including editing, edit checks and queries, was done using CSPro (Census Survey Processing System) with additional data editing and cleaning taking place in Excel.

The staff from the CSD was responsible for undertaking the coding and data entry, with assistance from an additional four temporary staff to help produce results in a more timely manner.

Although enumeration didn't get completed until mid June, the coding and data entry commenced as soon as forms where available from Funafuti, which was towards the end of March. The coding and data entry was then completed around the middle of July.

A visit from an SPC consultant then took place to undertake initial cleaning of the data, primarily addressing missing data items and missing schedules. Once the initial data cleaning was undertaken in CSPro, data was transferred to Excel where it was closely scrutinized to check that all responses were sensible. In the cases where unusual values were identified, original forms were consulted for these households and modifications made to the data if required.

Despite the best efforts being made to clean the data file in preparation for the analysis, no doubt errors will still exist in the data, due to its size and complexity. Having said this, they are not expected to have significant impacts on the survey results.

Under-Reporting and Incorrect Reporting as a result of Poor Field Work Procedures The most crucial stage of any survey activity, whether it be a population census or a survey such as a HIES is the fieldwork. It is crucial for intense checking to take place in the field before survey forms are returned to the office for data processing. Unfortunately, it became evident during the cleaning of the data that fieldwork wasn?t checked as thoroughly as required, and as such some unexpected values appeared in the questionnaires, as well as unusual results appearing in the diaries. Efforts were made to indentify the main issues which would have the greatest impact on final results, and this information was modified using local knowledge, to a more reasonable answer, when required.

Data Entry Errors Data entry errors are always expected, but can be kept to a minimum with

Transportation-disadvantaged populations often face significant challenges in meeting their basic travel needs. Microtransit, a technology-enabled transit mobility solution, can potentially address these issues by providing on-demand, affordable, and flexible services. However, the extent to which microtransit serves underserved populations and the factors influencing their adoption remain unclear. This research focuses on SmaRT Ride, a microtransit pilot program operated by the Sacramento Regional Transit (SacRT) in the Sacramento area. From early February to the end of May 2024, online and intercept surveys were conducted among underserved populations to understand their travel behavior. After data cleaning, 180 valid responses were collected. Descriptive analysis of the data shows that SmaRT Ride has significantly improved transportation access for these communities. Furthermore, logistic regressions were employed to explore factors influencing the willingness to adopt microtransit a..., Reaching underserved communities, especially for surveys, is challenging due to socioeconomic and language barriers. To improve our sample and gather more responses to our survey, we used contacts from existing datasets, worked with local food banks, and identified transit stops and other intercept survey site recommendations from stakeholders such as SacRT. Additionally, multiple methods such as online, in-person, and telephone/text message survey recruitment methods were used to accommodate different preferences and access levels of underserved individuals., , Title: Dataset of underserved microtransit users in the Sacrament Area, California

Access this dataset on Dryad: https://doi.org/10.5061/dryad.r7sqv9smh

Dataset contents: Variables/Features: 273 variables. Detailed descriptions of these variables are provided in the attached "Variable description" Excel file. Number of Entries: 180 cases Time Frame: Data was collected From early February to the end of May 2024 Format: SPSS (.sav)

Description: This dataset contains underserved populations' opinions, daily travel pattern, and use of SmaRT Ride, a microtransit pilot program operated by the Sacmento Regional Transit (SacRT) in the Sacramento area. Sampling methods used to reach underserved communities included obtaining email addresses from existing datasets for online surveys, conducting intercept surveys at food distribution sites associated with food banks, busy transit stops, and other locations recommended by stakeholders such as SacRT. Rea...

Data for manuscript “Functional morphology and efficiency of the antenna cleaner in Camponotus rufifemur ants"Excel file includes 3 data sheets. One sheet for each experiment. The corresponding figures from the manuscript are mentioned above the actual data.Manuscript data.xlsx

Annex_III_occurrence data.xlsx

This Annex is an excel file presenting tables on occurrence data on nitrate and nitrite in feed.

Nitrate Nitrite_OCC_ZENODO.CSV

Contains the raw (no data cleaning applied to it) occurrence dataset on nitrate and nitrite as extracted from EFSA DWH on 03 December 2019 in feed samples presented in the opinion as described in its section 3.2.2. The data is provided in csv format. This dataset is compliant with EFSA SSD model and contains two additional columns documenting issues identified in the cleaning process (column: issue) and the action taken (column: action) to address the issue (e.g. delete record or update values in specific fields).

The link to the catalogues of controlled terminologies can be found under "Related identifiers”.

Abstract

A Knowledge, Attitudes and Practices (KAP) survey was conducted in Ajuong Thok and Pamir Refugee Camps in October 2019 to determine the current Water, Sanitation and Hygiene (WASH) conditions as well as hygiene attitudes and practices within the households (HHs) surveyed. The assessment utilized a systematic random sampling method, and a total of 1,474 HHs (735 HHs in Ajuong Thok and 739 HHs in Pamir) were surveyed using mobile data collection (MDC) within a period of 21 days. Data was cleaned and analyzed in Excel. The summary of the results is presented in this report.

The findings show that the overall average number of liters of water per person per day was 23.4, in both Ajuong Thok and Pamir Camps, which was slightly higher than the recommended United Nations High Commissioner for Refugees (UNHCR) minimum standard of at least 20 liters of water available per person per day. This is a slight improvement from the 21 liters reported the previous year. The average HH size was six people. Women comprised 83% of the surveyed respondents and males 17%. Almost all the respondents were refugees, constituting 99.5% (n=1,466). The refugees were aware of the key health and hygiene practices, possibly as a result of routine health and hygiene messages delivered to them by Samaritan´s Purse (SP) and other health partners. Most refugees had knowledge about keeping the water containers clean, washing hands during critical times, safe excreta disposal and disease prevention.

Geographic coverage

Ajuong Thok and Pamir Refugee Camps

Analysis unit

Households

Universe

All households in Ajuong Thok and Pamir Refugee Camps

Kind of data

Sample survey data [ssd]

Sampling procedure

Households were selected using systematic random sampling. Enumerators systematically walked through the camp block by block, row by row, in such a way as to pass each HH. Within blocks, enumerators started at one corner, then systematically used the sampling interval as they walked up and down each of the rows throughout the block, covering every block in Ajuong Thok and Pamir.

In each location, the first HH sampled in a block was generated using an Excel tool customized by UNHCR which generated a Random Start and Sampling Interval.

Mode of data collection

Face-to-face [f2f]

Research instrument

The survey questionnaire used to collect the data consists of the following sections: - Demographics - Water collection and storage - Drinking water hygiene - Hygiene - Sanitation - Messaging - Distribution (NFI) - Diarrhea prevalence, knowledge and health seeking behaviour - Menstrual hygiene

Cleaning operations

The data collected was uploaded to a server at the end of each day. IFormBuilder generated a Microsoft (MS) Excel spreadsheet dataset which was then cleaned and analyzed using MS Excel.

Given that SP is currently implementing a WASH program in Ajuong Thok and Pamir, the assessment data collected in these camps will not only serve as the endline for UNHCR 2018 programming but also as the baseline for 2019 programming.

Data was anonymized through decoding and local suppression.

Abstract

This dataset was generated from a set of Excel spreadsheets from an Information and Communication Technology Services (ICTS) administrative database on student applications to the University of Cape Town (UCT). This database contains information on applications to UCT between the January 2006 and December 2014. In the original form received by DataFirst the data were ill suited to research purposes. This dataset represents an attempt at cleaning and organizing these data into a more tractable format. To ensure data confidentiality direct identifiers have been removed from the data and the data is only made available to accredited researchers through DataFirst's Secure Data Service.

The dataset was separated into the following data files:

1. Application level information: the "finest" unit of analysis. Individuals may have multiple applications. Uniquely identified by an application ID variable. There are a total of 1,714,669 applications on record.
2. Individual level information: individuals may have multiple applications. Each individual is uniquely identified by an individual ID variable. Each individual is associated with information on "key subjects" from a separate data file also contained in the database. These key subjects are all separate variables in the individual level data file. There are a total of 285,005 individuals on record.
3. Secondary Education Information: individuals can also be associated with row entries for each subject. This data file does not have a unique identifier. Instead, each row entry represents a specific secondary school subject for a specific individual. These subjects are quite specific and the data allows the user to distinguish between, for example, higher grade accounting and standard grade accounting. It also allows the user to identify the educational authority issuing the qualification e.g. Cambridge Internal Examinations (CIE) versus National Senior Certificate (NSC).
4. Tertiary Education Information: the smallest of the four data files. There are multiple entries for each individual in this dataset. Each row entry contains information on the year, institution and transcript information and can be associated with individuals.

Analysis unit

Applications, individuals

Kind of data

Administrative records [adm]

Mode of data collection

Other [oth]

Cleaning operations

The data files were made available to DataFirst as a group of Excel spreadsheet documents from an SQL database managed by the University of Cape Town's Information and Communication Technology Services . The process of combining these original data files to create a research-ready dataset is summarised in a document entitled "Notes on preparing the UCT Student Application Data 2006-2014" accompanying the data.

For any questions about this data please email me at jacob@crimedatatool.com. If you use this data, please cite it.Version 4 release notes:Adds data for 2018Version 3 release notes:Adds data in the following formats: Excel.Changes project name to avoid confusing this data for the ones done by NACJD.Version 2 release notes:Adds data for 2017.Adds a "number_of_months_reported" variable which says how many months of the year the agency reported data.Property Stolen and Recovered is a Uniform Crime Reporting (UCR) Program data set with information on the number of offenses (crimes included are murder, rape, robbery, burglary, theft/larceny, and motor vehicle theft), the value of the offense, and subcategories of the offense (e.g. for robbery it is broken down into subcategories including highway robbery, bank robbery, gas station robbery). The majority of the data relates to theft. Theft is divided into subcategories of theft such as shoplifting, theft of bicycle, theft from building, and purse snatching. For a number of items stolen (e.g. money, jewelry and previous metals, guns), the value of property stolen and and the value for property recovered is provided. This data set is also referred to as the Supplement to Return A (Offenses Known and Reported). All the data was received directly from the FBI as text or .DTA files. I created a setup file based on the documentation provided by the FBI and read the data into R using the package asciiSetupReader. All work to clean the data and save it in various file formats was also done in R. For the R code used to clean this data, see here: https://github.com/jacobkap/crime_data. The Word document file available for download is the guidebook the FBI provided with the raw data which I used to create the setup file to read in data.There may be inaccuracies in the data, particularly in the group of columns starting with "auto." To reduce (but certainly not eliminate) data errors, I replaced the following values with NA for the group of columns beginning with "offenses" or "auto" as they are common data entry error values (e.g. are larger than the agency's population, are much larger than other crimes or months in same agency): 1000, 2000, 3000, 4000, 5000, 6000, 7000, 8000, 9000, 10000, 20000, 30000, 40000, 50000, 60000, 70000, 80000, 90000, 100000, 99942. This cleaning was NOT done on the columns starting with "value."For every numeric column I replaced negative indicator values (e.g. "j" for -1) with the negative number they are supposed to be. These negative number indicators are not included in the FBI's codebook for this data but are present in the data. I used the values in the FBI's codebook for the Offenses Known and Clearances by Arrest data.To make it easier to merge with other data, I merged this data with the Law Enforcement Agency Identifiers Crosswalk (LEAIC) data. The data from the LEAIC add FIPS (state, county, and place) and agency type/subtype. If an agency has used a different FIPS code in the past, check to make sure the FIPS code is the same as in this data.