A Gold Standard Corpus for Activity Information

Dataset Title: A Gold Standard Corpus for Activity Information (GoSCAI)

Dataset Curators: The Epidemiology & Biostatistics Section of the NIH Clinical Center Rehabilitation Medicine Department

Dataset Version: 1.0 (May 16, 2025)

Dataset Citation and DOI: NIH CC RMD Epidemiology & Biostatistics Section. (2025). A Gold Standard Corpus for Activity Information (GoSCAI) [Data set]. Zenodo. doi: 10.5281/zenodo.15528545

EXECUTIVE SUMMARY

This data statement is for a gold standard corpus of de-identified clinical notes that have been annotated for human functioning information based on the framework of the WHO's International Classification of Functioning, Disability and Health (ICF). The corpus includes 484 notes from a single institution within the United States written in English in a clinical setting. This dataset was curated for the purpose of training natural language processing models to automatically identify, extract, and classify information on human functioning at the whole-person, or activity, level.

CURATION RATIONALE

This dataset is curated to be a publicly available resource for the development and evaluation of methods for the automatic extraction and classification of activity-level functioning information as defined in the ICF. The goals of data curation are to 1) create a corpus of a size that can be manually deidentified and annotated, 2) maximize the density and diversity of functioning information of interest, and 3) allow public dissemination of the data.

LANGUAGE VARIETIES

Language Region: en-US

Prose Description: English as written by native and bilingual English speakers in a clinical setting

LANGUAGE USER DEMOGRAPHIC

The language users represented in this dataset are medical and clinical professionals who work in a research hospital setting. These individuals hold professional degrees corresponding to their respective specialties. Specific demographic characteristics of the language users such as age, gender, or race/ethnicity were not collected.

ANNOTATOR DEMOGRAPHIC

The annotator group consisted of five people, 33 to 76 years old, including four females and one male. Socioeconomically, they came from the middle and upper-middle income classes. Regarding first language, three annotators had English as their first language, one had Chinese, and one had Spanish. Proficiency in English, the language of the data being annotated, was native for three of the annotators and bilingual for the other two. The annotation team included clinical rehabilitation domain experts with backgrounds in occupational therapy, physical therapy, and individuals with public health and data science expertise. Prior to annotation, all annotators were trained on the specific annotation process using established guidelines for the given domain, and annotators were required to achieve a specified proficiency level prior to annotating notes in this corpus.

LINGUISTIC SITUATION AND TEXT CHARACTERISTICS

The notes in the dataset were written as part of clinical care within a U.S. research hospital between May 2008 and November 2019. These notes were written by health professionals asynchronously following the patient encounter to document the interaction and support continuity of care. The intended audience of these notes were clinicians involved in the patients' care. The included notes come from nine disciplines - neuropsychology, occupational therapy, physical medicine (physiatry), physical therapy, psychiatry, recreational therapy, social work, speech language pathology, and vocational rehabilitation. The notes were curated to support research on natural language processing for functioning information between 2018 and 2024.

PREPROCESSING AND DATA FORMATTING

The final corpus was derived from a set of clinical notes extracted from the hospital electronic medical record (EMR) for the purpose of clinical research. The original data include character-based digital content originally. We work in ASCII 8 or UNICODE encoding, and therefore part of our pre-processing includes running encoding detection and transformation from encodings such as Windows-1252 or ISO-8859 format to our preferred format.

On the larger corpus, we applied sampling to match our curation rationale. Given the resource constraints of manual annotation, we set out to create a dataset of 500 clinical notes, which would exclude notes over 10,000 characters in length.

To promote density and diversity, we used five note characteristics as sampling criteria. We used the text length as expressed in number of characters. Next, we considered the discipline group as derived from note type metadata and describes which discipline a note originated from: occupational and vocational therapy (OT/VOC), physical therapy (PT), recreation therapy (RT), speech and language pathology (SLP), social work (SW), or miscellaneous (MISC, including psychiatry, neurology and physiatry). These disciplines were selected for collecting the larger corpus because their notes are likely to include functioning information. Existing information extraction tools were used to obtain annotation counts in four areas of functioning and provided a note’s annotation count, annotation density (annotation count divided by text length), and domain count (number of domains with at least 1 annotation).

We used stratified sampling across the 6 discipline groups to ensure discipline diversity in the corpus. Because of low availability, 50 notes were sampled from SLP with relaxed criteria, and 90 notes each from the 5 other discipline groups with stricter criteria. Sampled SLP notes were those with the highest annotation density that had an annotation count of at least 5 and a domain count of at least 2. Other notes were sampled by highest annotation count and lowest text length, with a minimum annotation count of 15 and minimum domain count of 3.

The notes in the resulting sample included certain types of PHI and PII. To prepare for public dissemination, all sensitive or potentially identifying information was manually annotated in the notes and replaced with substituted content to ensure readability and enough context needed for machine learning without exposing any sensitive information. This de-identification effort was manually reviewed to ensure no PII or PHI exposure and correct any resulting readability issues. Notes about pediatric patients were excluded. No intent was made to sample multiple notes from the same patient. No metadata is provided to group notes other than by note type, discipline, or discipline group. The dataset is not organized beyond the provided metadata, but publications about models trained on this dataset should include information on the train/test splits used.

All notes were sentence-segmented and tokenized using the spaCy en_core_web_lg model with additional rules for sentence segmentation customized to the dataset. Notes are stored in an XML format readable by the GATE annotation software (https://gate.ac.uk/family/developer.html), which stores annotations separately in annotation sets.

CAPTURE QUALITY

As the clinical notes were extracted directly from the EMR in text format, the capture quality was determined to be high. The clinical notes did not have to be converted from other data formats, which means this dataset is free from noise introduced by conversion processes such as optical character recognition.

LIMITATIONS

Because of the effort required to manually deidentify and annotate notes, this corpus is limited in terms of size and representation. The curation decisions skewed note selection towards specific disciplines and note types to increase the likelihood of encountering information on functioning. Some subtypes of functioning occur infrequently in the data, or not at all. The deidentification of notes was done in a manner to preserve natural language as it would occur in the notes, but some information is lost, e.g. on rare diseases.

METADATA

Information on the manual annotation process is provided in the annotation guidelines for each of the four domains:

- Communication & Cognition (https://zenodo.org/records/13910167)

- Mobility (https://zenodo.org/records/11074838)

- Self-Care & Domestic Life (SCDL) (https://zenodo.org/records/11210183)

- Interpersonal Interactions & Relationships (IPIR) (https://zenodo.org/records/13774684)

Inter-annotator agreement was established on development datasets described in the annotation guidelines prior to the annotation of this gold standard corpus.

The gold standard corpus consists of 484 documents, which include 35,147 sentences in total. The distribution of annotated information is provided in the table below.

BackgroundMedicare Advantage (MA) and Medicare fee-for-service (FFS) plans have different financial incentives. Medicare pays predetermined rates per beneficiary to MA plans for providing care throughout the year, while providers serving FFS patients are reimbursed per utilization event. It is unknown how these incentives affect post-acute care in skilled nursing facilities (SNFs). The objective of this study was to examine differences in rehabilitation service use, length of stay, and outcomes for patients following hip fracture between FFS and MA enrollees.Methods and findingsThis was a retrospective cohort study to examine differences in health service utilization and outcomes between FFS and MA patients in SNFs following hip fracture hospitalization during the period January 1, 2011, to June 30, 2015, and followed up until December 31, 2015. We linked the Master Beneficiary Summary File, Medicare Provider and Analysis Review data, Healthcare Effectiveness Data and Information Set data, the Minimum Data Set, and the American Community Survey. The 6 primary outcomes of interest in this study included 2 process measures and 4 patient-centered outcomes. Process measures included length of stay in the SNF and average rehabilitation therapy minutes (physical and occupational therapy) received per day. Patient-centered outcomes included 30-day hospital readmission, changes in functional status as measured by the 28-point late loss MDS-ADL scale, likelihood of becoming a long-term resident, and successful discharge to the community. Successful discharge from a SNF was defined as being discharged to the community within 100 days of SNF admission and remaining alive in the community without being institutionalized in any acute or post-acute setting for at least 30 days. We analyzed 211,296 FFS and 75,554 MA patients with hip fracture admitted directly to a SNF following an index hospitalization who had not been in a nursing facility or hospital in the preceding year. We used inverse probability of treatment weighting (IPTW) and nursing facility fixed effects regression models to compare treatments and outcomes between MA and FFS patients. MA patients were younger and less cognitively impaired upon SNF admission than FFS patients. After applying IPTW, demographic and clinical characteristics of MA patients were comparable with those of FFS patients. After adjusting for risk factors using IPTW-weighted fixed effects regression models, MA patients spent 5.1 (95% CI -5.4 to -4.8) fewer days in the SNF and received 463 (95% CI to -483.2 to -442.4) fewer minutes of total rehabilitation therapy during the first 40 days following SNF admission, i.e., 12.1 (95% CI -12.7 to -11.4) fewer minutes of rehabilitation therapy per day compared to FFS patients. In addition, MA patients had a 1.2 percentage point (95% CI -1.5 to -1.1) lower 30-day readmission rate, 0.6 percentage point (95% CI -0.8 to -0.3) lower rate of becoming a long-stay resident, and a 3.2 percentage point (95% CI 2.7 to 3.7) higher rate of successful discharge to the community compared to FFS patients. The major limitation of this study was that we only adjusted for observed differences to address selection bias between FFS and MA patients with hip fracture. Therefore, results may not be generalizable to other conditions requiring extensive rehabilitation.ConclusionsCompared to FFS patients, MA patients had a shorter course of rehabilitation but were more likely to be discharged to the community successfully and were less likely to experience a 30-day hospital readmission. Longer lengths of stay may not translate into better outcomes in the case of hip fracture patients in SNFs.