Shared here are data and code to support the manuscript: "Despite Lacey Act regulation, ongoing amphibian trade into the United States threatens salamanders with disease." A project overview is provided in the GitHub repository, but, in brief, this work required the compilation and cleaning of the United States Fish and Wildlife Service's Law Enforcement Management Information System (LEMIS) data in order to generate a complete time course of amphibian imports to the United States from 1999 to 2021. Others pursuing data reuse will most likely be interested in the full, cleaned amphibian import dataset (in CSV format) contained within these project files, which is named harmonized_amphibian_LEMIS_1999_to_2021.csv. Information regarding the LEMIS data, including detailed description of the data fields, can be found in Eskew et al. 2020, "United States wildlife and wildlife product imports from 2000–2014".

About this course Do you have messy data from multiple inconsistent sources, or open-responses to questionnaires? Do you want to improve the quality of your data by refining it and using the power of the internet? Open Refine is the perfect partner to Excel. It is a powerful, free tool for exploring, normalising and cleaning datasets, and extending data by accessing the internet through APIs. In this course we’ll work through the various features of Refine, including importing data, faceting, clustering, and calling remote APIs, by working on a fictional but plausible humanities research project. Learning Outcomes Download, install and run Open Refine Import data from csv, text or online sources and create projects Navigate data using the Open Refine interface Explore data by using facets Clean data using clustering Parse data using GREL syntax Extend data using Application Programming Interfaces (APIs) Export project for use in other applications Prerequisites The course has no prerequisites. Licence Copyright © 2021 Intersect Australia Ltd. All rights reserved.

This dataset was hosted on IBM Cloud object

You can find the "Automobile Dataset" from the following link: https://archive.ics.uci.edu/ml/machine-learning-databases/autos/imports-85.data.

I cleaned the data myself, you can check notebook "Used Car Pricing Data Wrangling" for details.

Objective: To develop a clinical informatics pipeline designed to capture large-scale structured EHR data for a national patient registry.

Materials and Methods: The EHR-R-REDCap pipeline is implemented using R-statistical software to remap and import structured EHR data into the REDCap-based multi-institutional Merkel Cell Carcinoma (MCC) Patient Registry using an adaptable data dictionary.

Results: Clinical laboratory data were extracted from EPIC Clarity across several participating institutions. Labs were transformed, remapped and imported into the MCC registry using the EHR labs abstraction (eLAB) pipeline. Forty-nine clinical tests encompassing 482,450 results were imported into the registry for 1,109 enrolled MCC patients. Data-quality assessment revealed highly accurate, valid labs. Univariate modeling was performed for labs at baseline on overall survival (N=176) using this clinical informatics pipeline.

Conclusion: We demonstrate feasibility of the facile eLAB workflow. EHR data is successfully transformed, and bulk-loaded/imported into a REDCap-based national registry to execute real-world data analysis and interoperability.

Methods eLAB Development and Source Code (R statistical software):

eLAB is written in R (version 4.0.3), and utilizes the following packages for processing: DescTools, REDCapR, reshape2, splitstackshape, readxl, survival, survminer, and tidyverse. Source code for eLAB can be downloaded directly (https://github.com/TheMillerLab/eLAB).

eLAB reformats EHR data abstracted for an identified population of patients (e.g. medical record numbers (MRN)/name list) under an Institutional Review Board (IRB)-approved protocol. The MCCPR does not host MRNs/names and eLAB converts these to MCCPR assigned record identification numbers (record_id) before import for de-identification.

Functions were written to remap EHR bulk lab data pulls/queries from several sources including Clarity/Crystal reports or institutional EDW including Research Patient Data Registry (RPDR) at MGB. The input, a csv/delimited file of labs for user-defined patients, may vary. Thus, users may need to adapt the initial data wrangling script based on the data input format. However, the downstream transformation, code-lab lookup tables, outcomes analysis, and LOINC remapping are standard for use with the provided REDCap Data Dictionary, DataDictionary_eLAB.csv. The available R-markdown ((https://github.com/TheMillerLab/eLAB) provides suggestions and instructions on where or when upfront script modifications may be necessary to accommodate input variability.

The eLAB pipeline takes several inputs. For example, the input for use with the ‘ehr_format(dt)’ single-line command is non-tabular data assigned as R object ‘dt’ with 4 columns: 1) Patient Name (MRN), 2) Collection Date, 3) Collection Time, and 4) Lab Results wherein several lab panels are in one data frame cell. A mock dataset in this ‘untidy-format’ is provided for demonstration purposes (https://github.com/TheMillerLab/eLAB).

Bulk lab data pulls often result in subtypes of the same lab. For example, potassium labs are reported as “Potassium,” “Potassium-External,” “Potassium(POC),” “Potassium,whole-bld,” “Potassium-Level-External,” “Potassium,venous,” and “Potassium-whole-bld/plasma.” eLAB utilizes a key-value lookup table with ~300 lab subtypes for remapping labs to the Data Dictionary (DD) code. eLAB reformats/accepts only those lab units pre-defined by the registry DD. The lab lookup table is provided for direct use or may be re-configured/updated to meet end-user specifications. eLAB is designed to remap, transform, and filter/adjust value units of semi-structured/structured bulk laboratory values data pulls from the EHR to align with the pre-defined code of the DD.

Data Dictionary (DD)

EHR clinical laboratory data is captured in REDCap using the ‘Labs’ repeating instrument (Supplemental Figures 1-2). The DD is provided for use by researchers at REDCap-participating institutions and is optimized to accommodate the same lab-type captured more than once on the same day for the same patient. The instrument captures 35 clinical lab types. The DD serves several major purposes in the eLAB pipeline. First, it defines every lab type of interest and associated lab unit of interest with a set field/variable name. It also restricts/defines the type of data allowed for entry for each data field, such as a string or numerics. The DD is uploaded into REDCap by every participating site/collaborator and ensures each site collects and codes the data the same way. Automation pipelines, such as eLAB, are designed to remap/clean and reformat data/units utilizing key-value look-up tables that filter and select only the labs/units of interest. eLAB ensures the data pulled from the EHR contains the correct unit and format pre-configured by the DD. The use of the same DD at every participating site ensures that the data field code, format, and relationships in the database are uniform across each site to allow for the simple aggregation of the multi-site data. For example, since every site in the MCCPR uses the same DD, aggregation is efficient and different site csv files are simply combined.

Study Cohort

This study was approved by the MGB IRB. Search of the EHR was performed to identify patients diagnosed with MCC between 1975-2021 (N=1,109) for inclusion in the MCCPR. Subjects diagnosed with primary cutaneous MCC between 2016-2019 (N= 176) were included in the test cohort for exploratory studies of lab result associations with overall survival (OS) using eLAB.

Statistical Analysis

OS is defined as the time from date of MCC diagnosis to date of death. Data was censored at the date of the last follow-up visit if no death event occurred. Univariable Cox proportional hazard modeling was performed among all lab predictors. Due to the hypothesis-generating nature of the work, p-values were exploratory and Bonferroni corrections were not applied.

Overview
This repository contains ready-to-use frequency time series as well as the corresponding pre-processing scripts in python. The data covers three synchronous areas of the European power grid:

• Continental Europe
• Great Britain
• Nordic

This work is part of the paper "Predictability of Power Grid Frequency"[1]. Please cite this paper, when using the data and the code. For a detailed documentation of the pre-processing procedure we refer to the supplementary material of the paper.

Data sources
We downloaded the frequency recordings from publically available repositories of three different Transmission System Operators (TSOs).

• Continental Europe [2]: We downloaded the data from the German TSO TransnetBW GmbH, which retains the Copyright on the data, but allows to re-publish it upon request [3].
• Great Britain [4]: The download was supported by National Grid ESO Open Data, which belongs to the British TSO National Grid. They publish the frequency recordings under the NGESO Open License [5].
• Nordic [6]: We obtained the data from the Finish TSO Fingrid, which provides the data under the open license CC-BY 4.0 [7].

Content of the repository

A) Scripts

1. In the "Download_scripts" folder you will find three scripts to automatically download frequency data from the TSO's websites.
2. In "convert_data_format.py" we save the data with corrected timestamp formats. Missing data is marked as NaN (processing step (1) in the supplementary material of [1]).
3. In "clean_corrupted_data.py" we load the converted data and identify corrupted recordings. We mark them as NaN and clean some of the resulting data holes (processing step (2) in the supplementary material of [1]).

The python scripts run with Python 3.7 and with the packages found in "requirements.txt".

B) Data_converted and Data_cleansed
The folder "Data_converted" contains the output of "convert_data_format.py" and "Data_cleansed" contains the output of "clean_corrupted_data.py".

• File type: The files are zipped csv-files, where each file comprises one year.
• Data format: The files contain two columns. The first one represents the time stamps in the format Year-Month-Day Hour-Minute-Second, which is given as naive local time. The second column contains the frequency values in Hz.
• NaN representation: We mark corrupted and missing data as "NaN" in the csv-files.

Use cases
We point out that this repository can be used in two different was:

• Use pre-processed data: You can directly use the converted or the cleansed data. Note however that both data sets include segments of NaN-values due to missing and corrupted recordings. Only a very small part of the NaN-values were eliminated in the cleansed data to not manipulate the data too much. If your application cannot deal with NaNs, you could build upon the following commands to select the longest interval of valid data from the cleansed data:

from helper_functions import *
import pandas as pd

cleansed_data = pd.read_csv('/Path_to_cleansed_data/data.zip',
            index_col=0, header=None, squeeze=True,
            parse_dates=[0])
valid_bounds, valid_sizes = true_intervals(~cleansed_data.isnull())
start,end= valid_bounds[ np.argmax(valid_sizes) ]
data_without_nan = cleansed_data.iloc[start:end]

• Produce your own cleansed data: Depending on your application, you might want to cleanse the data in a custom way. You can easily add your custom cleansing procedure in "clean_corrupted_data.py" and then produce cleansed data from the raw data in "Data_converted".

License
We release the code in the folder "Scripts" under the MIT license [8]. In the case of Nationalgrid and Fingrid, we further release the pre-processed data in the folder "Data_converted" and "Data_cleansed" under the CC-BY 4.0 license [7]. TransnetBW originally did not publish their data under an open license. We have explicitly received the permission to publish the pre-processed version from TransnetBW. However, we cannot publish our pre-processed version under an open license due to the missing license of the original TransnetBW data.

Datasets (all) in .csv format for direct import into R. The data collection consists of the following datasets:

CH4.data.csv

This is the dataset used for the biocontrol analyses (all mixed effects random intercept models) using Lysmata vittata to reduce the reinfection pressure of Neobenedenia girellae on Epinephelus lanceolatus.

CH4WQ.csv

This is all the water quality data recorded and used the in the water quality analysis (linear regression).

Cyclistic: Google Data Analytics Capstone Project

Cyclistic - Google Data Analytics Certification Capstone Project Moirangthem Arup Singh How Does a Bike-Share Navigate Speedy Success? Background: This project is for the Google Data Analytics Certification capstone project. I am wearing the hat of a junior data analyst working in the marketing analyst team at Cyclistic, a bike-share company in Chicago. Cyclistic is a bike-share program that features more than 5,800 bicycles and 600 docking stations. Cyclistic sets itself apart by also offering reclining bikes, hand tricycles, and cargo bikes, making bike-share more inclusive to people with disabilities and riders who can’t use a standard two-wheeled bike. The majority of riders opt for traditional bikes; about 8% of riders use the assistive options. Cyclistic users are more likely to ride for leisure, but about 30% use them to commute to work each day. Customers who purchase single-ride or full-day passes are referred to as casual riders. Customers who purchase annual memberships are Cyclistic members. The director of marketing believes the company’s future success depends on maximizing the number of annual memberships. Therefore,my team wants to understand how casual riders and annual members use Cyclistic bikes differently. From these insights, my team will design a new marketing strategy to convert casual riders into annual members. But first, Cyclistic executives must approve the recommendations, so they must be backed up with compelling data insights and professional data visualizations. This project will be completed by using the 6 Data Analytics stages: Ask: Identify the business task and determine the key stakeholders. Prepare: Collect the data, identify how it’s organized, determine the credibility of the data. Process: Select the tool for data cleaning, check for errors and document the cleaning process. Analyze: Organize and format the data, aggregate the data so that it’s useful, perform calculations and identify trends and relationships. Share: Use design thinking principles and data-driven storytelling approach, present the findings with effective visualization. Ensure the analysis has answered the business task. Act: Share the final conclusion and the recommendations. Ask: Business Task: Recommend marketing strategies aimed at converting casual riders into annual members by better understanding how annual members and casual riders use Cyclistic bikes differently. Stakeholders: Lily Moreno: The director of marketing and my manager. Cyclistic executive team: A detail-oriented executive team who will decide whether to approve the recommended marketing program. Cyclistic marketing analytics team: A team of data analysts responsible for collecting, analyzing, and reporting data that helps guide Cyclistic’s marketing strategy. Prepare: For this project, I will use the public data of Cyclistic’s historical trip data to analyze and identify trends. The data has been made available by Motivate International Inc. under the license. I downloaded the ZIP files containing the csv files from the above link but while uploading the files in kaggle (as I am using kaggle notebook), it gave me a warning that the dataset is already available in kaggle. So I will be using the dataset cyclictic-bike-share dataset from kaggle. The dataset has 13 csv files from April 2020 to April 2021. For the purpose of my analysis I will use the csv files from April 2020 to March 2021. The source csv files are in Kaggle so I can rely on it's integrity. I am using Microsoft Excel to get a glimpse of the data. There is one csv file for each month and has information about the bike ride which contain details of the ride id, rideable type, start and end time, start and end station, latitude and longitude of the start and end stations. Process: I will use R as language in kaggle to import the dataset to check how it’s organized, whether all the columns have appropriate data type, find outliers and if any of these data have sampling bias. I will be using below R libraries

Load the tidyverse, lubridate, ggplot2, sqldf and psych libraries

library(tidyverse) library(lubridate) library(ggplot2) library(plotrix) ── Attaching packages ─────────────────────────────────────── tidyverse 1.3.1 ──

✔ ggplot2 3.3.5 ✔ purrr 0.3.4 ✔ tibble 3.1.4 ✔ dplyr 1.0.7 ✔ tidyr 1.1.3 ✔ stringr 1.4.0 ✔ readr 2.0.1 ✔ forcats 0.5.1

── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ── ✖ dplyr::filter() masks stats::filter() ✖ dplyr::lag() masks stats::lag()

Attaching package: ‘lubridate’

The following objects are masked from ‘package:base’:

date, intersect, setdiff, union

Set the working directory

setwd("/kaggle/input/cyclistic-bike-share")

Import the csv files

r_202004 <- read.csv("202004-divvy-tripdata.csv") r_202005 <- read.csv("20...

Many organizations are interested in increasing the diversity of their workforce and spend millions of dollars on diversity training. Yet there is little empirical evidence that such training increases diversity in organizations. We implemented a large-scale field experiment in a global telecommunications and engineering firm (n = 10,433) testing whether behaviorally designed training increases the diversity of who is hired. In particular, the diversity training was timely (delivered immediately before hiring managers shortlisted candidates), tailored to the hiring decision, delivered by senior members of the organization, and made diversity salient. Results show that behaviorally designed diversity training can positively influence the hiring of women and non-national applicants relative to business as usual. Our findings suggest that behaviorally designed diversity training can work to change the diversity of hires but that its success relies on carefully considered design choices and..., , , # Behaviorally designed training leads to more diverse hiring

Overview

The program (Stata do-files) for “Behaviorally designed training leads to diversity hiring†by Arslan, Chang, Chilazi, Bohnet, and Hauser, published in Science (2025).

The program files run all the code to import raw data files (xlsx, csv), clean and generate the data (in dta format), prepare the data for the analysis, run the regression analyses, export output, and thus generate tables presented in the paper. The replicator should expect the code to run for up to 15 minutes.

Data Availability and Sharing

The organizational data used in this manuscript is of a proprietary nature. We, the authors of the manuscript, have legitimate access to and permission to use the data, but we are unable to make the data publicly available due to a strict data use agreement with our field partner (global telecommunications and engineering company).

Interested researchers are encouraged to contact MoreThanNow to have...

E-commerce Sales Prediction Dataset

This repository contains a comprehensive and clean dataset for predicting e-commerce sales, tailored for data scientists, machine learning enthusiasts, and researchers. The dataset is crafted to analyze sales trends, optimize pricing strategies, and develop predictive models for sales forecasting.

📂 Dataset Overview

The dataset includes 1,000 records across the following features:

📊 Data Summary

General Properties

Date: - Range: 01-01-2023 to 12-31-2023. - Contains 1,000 unique values without missing data.

Product_Category: - Categories: Electronics (21%), Sports (21%), Other (58%). - Most common category: Electronics (21%).

Price: - Range: From 244 to 999. - Mean: 505, Standard Deviation: 290. - Most common price range: 14.59 - 113.07.

Discount: - Range: From 0.01% to 49.92%. - Mean: 24.9%, Standard Deviation: 14.4%. - Most common discount range: 0.01 - 5.00%.

Customer_Segment: - Segments: Regular (35%), Occasional (34%), Other (31%). - Most common segment: Regular.

Marketing_Spend: - Range: From 2.41k to 10k. - Mean: 4.91k, Standard Deviation: 2.84k.

Units_Sold: - Range: From 5 to 57. - Mean: 29.6, Standard Deviation: 7.26. - Most common range: 24 - 34 units sold.

📈 Data Visualizations

The dataset is suitable for creating the following visualizations: - 1. Price Distribution: Histogram to show the spread of prices. - 2. Discount Distribution: Histogram to analyze promotional offers. - 3. Marketing Spend Distribution: Histogram to understand marketing investment patterns. - 4. Customer Segment Distribution: Bar plot of customer segments. - 5. Price vs Units Sold: Scatter plot to show pricing effects on sales. - 6. Discount vs Units Sold: Scatter plot to explore the impact of discounts. - 7. Marketing Spend vs Units Sold: Scatter plot for marketing effectiveness. - 8. Correlation Heatmap: Identify relationships between features. - 9. Pairplot: Visualize pairwise feature interactions.

💡 How the Data Was Created

The dataset is synthetically generated to mimic realistic e-commerce sales trends. Below are the steps taken for data generation:

1. Feature Engineering:

   • Identified key attributes such as product category, price, discount, and marketing spend, typically observed in e-commerce data.
   • Generated dependent features like units sold based on logical relationships.

2. Data Simulation:

   • Python Libraries: Used NumPy and Pandas to generate and distribute values.
   • Statistical Modeling: Ensured feature distributions aligned with real-world sales data patterns.

3. Validation:

   • Verified data consistency with no missing or invalid values.
   • Ensured logical correlations (e.g., higher discounts → increased units sold).

Note: The dataset is synthetic and not sourced from any real-world e-commerce platform.

🛠 Example Usage: Sales Prediction Model

Here’s an example of building a predictive model using Linear Regression:

Written in python

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score

# Load the dataset
df = pd.read_csv('ecommerce_sales.csv')

# Feature selection
X = df[['Price', 'Discount', 'Marketing_Spend']]
y = df['Units_Sold']

# Train-test split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Model training
model = LinearRegression()
model.fit(X_train, y_train)

# Predictions
y_pred = model.predict(X_test)

# Evaluation
mse = mean_squared_error(y_test, y_pred)
r2 = r2_score(y_test, y_pred)

print(f'Mean Squared Error: {mse:.2f}')
print(f'R-squared: {r2:.2f}')

OMOP2OBO Mappings - N3C OMOP to OBO Working group

This repository stores OMOP2OBO mappings which have been processed for use within the National COVID Cohort Collaborative (N3C) Enclave. The version of the mappings stored in this repository have been specifically formatted for use within the N3C Enclave.

N3C OMOP to OBO Working Group: https://covid.cd2h.org/ontology

Accessing the N3C-Formatted Mappings

You can access the three OMOP2OBO HPO mapping files in the Enclave from the Knowledge store using the following link: https://unite.nih.gov/workspace/compass/view/ri.compass.main.folder.1719efcf-9a87-484f-9a67-be6a29598567.

The mapping set includes three files, but you only need to merge the following two files with existing data in the Enclave in order to be able to create the concept sets:

OMOP2OBO_v2.0.0_N3C_Enclave_CSV_concept_set_expression_items.csv

OMOP2OBO_v2.0.0_N3C_Enclave_CSV_concept_set_version.csv

The first file OMOP2OBO_v2.0.0_N3C_Enclave_CSV_concept_set_expression_items.csv, contains columns for the OMOP concept ids and codes as well as specifies information like whether or not the OMOP concept’s descendants should be included when deriving the concept sets (defaults to FALSE). The other file OMOP2OBO_v2.0.0_N3C_Enclave_CSV_concept_set_version.csv, contains details on the mapping’s label (i.e., the HPO curie and label in the concept_set_id field) and its provenance/evidence (the specific column to access for this information is called intention).

Creating Concept Sets

Merge these files together on the column named codeset_id and then join them with existing Enclave tables like concept and condition_occurrence to populate the actual concept sets. The name of the concept set can be obtained from the OMOP2OBO_v2.0.0_N3C_Enclave_CSV_concept_set_version.csv file and is stored as a string in the column called concept_set_id. Although not ideal (but is the best way to approach this currently given what fields are available in the Enclave), to get the HPO CURIE and label will require applying a regex to this column.

An example mapping is shown below (highlighting some of the most useful columns):

codeset_id: 900000000 concept_set_id: [OMOP2OBO] hp_0002031-abnormal_esophagus_morphology concept: 23868 code: 69771008 codeSystem: SNOMED includeDescendants: False intention:

Mixed - This mapping was created using the OMOP2OBO mapping algorithm (https://github.com/callahantiff/OMOP2OBO).

The Mapping Category and Evidence supporting the mappings are provided below, by OMOP concept:

23868

Mapping Category: Automatic Exact - Concept

Mapping Provenance

OBO_DbXref-OMOP_ANCESTOR_SOURCE_CODE:snomed_69771008 | OBO_DbXref-OMOP_CONCEPT_SOURCE_CODE:snomed_69771008 | CONCEPT_SIMILARITY:HP_0002031_0.713

Release Notes - v2.0.0

Preparation

In order to import data into the Enclave, the following items are needed:

Obtain API Token, which will be included in the authorization header (stored as GitHub Secret)

Obtain username hash from the Enclave

OMOP2OBO Mappings (v1.5.0)

Data

Concept Set Container (concept_set_container): CreateNewConceptSet

Concept Set Version (code_sets): CreateNewDraftOMOPConceptSetVersion

Concept Set Expression Items (concept_set_version_item): addCodeAsVersionExpression

Script

n3c_mapping_conversion.py

Generated Output

Need to have the codeset_id filled from self-generation (ideally, from a conserved range) prior to beginning any of the API steps. The current list of assigned identifiers is stored in the file named omop2obo_enclave_codeset_id_dict_v2.0.0.json. Note that in order to accommodate the 1:Many mappings the codeset ids were re-generated and rather than being ampped to HPO concepts, they are mapped to SNOMED-CT concepts. This creates a cleaner mapping and will easily scale to future mapping builds.

To be consistent with OMOP tools, specifically Atlas, we have also created Atlas-formatted json files for each mapping, which are stored in the zipped directory named atlas_json_files_v2.0.0.zip. Note that as mentioned above, to enable the representation of 1:Many mappings the filenames are no longer named after HPO concepts they are now named with the OMOP concept_id and label and additional fields have been added within the JSON files that includes the HPO ids, labels, mapping category, mapping logic, and mapping evidence.

File 1: concept_set_container

Generated Data: OMOP2OBO_v2.0.0_N3C_Enclave_CSV_concept_set_container.csv

Columns:

concept_set_id

concept_set_name

intention

assigned_informatician

assigned_sme

project_id

status

stage

n3c_reviewer

alias

archived

created_by

created_at

File 2: concept_set_expression_items

Generated Data: OMOP2OBO_v2.0.0_N3C_Enclave_CSV_concept_set_expression_items.csv

Columns:

codeset_id

concept_id

code

codeSystem

ontology_id

ontology_label

mapping_category

mapping_logic

mapping_evidence

isExcluded

includeDescendants

includeMapped

item_id

annotation

created_by

created_at

File 3: concept_set_version

Generated Data: OMOP2OBO_v2.0.0_N3C_Enclave_CSV_concept_set_version.csv

Columns:

codeset_id

concept_set_id

concept_set_version_title

project

source_application

source_application_version

created_at

atlas_json

most_recent_version

comments

intention

limitations

issues

update_message

status

has_review

reviewed_by

created_by

provenance

atlas_json_resource_url

parent_version_id

is_draft

Generated Output:

OMOP2OBO_v2.0.0_N3C_Enclave_CSV_concept_set_container.csv

OMOP2OBO_v2.0.0_N3C_Enclave_CSV_concept_set_expression_items.csv

OMOP2OBO_v2.0.0_N3C_Enclave_CSV_concept_set_version.csv

atlas_json_files_v2.0.0.zip

omop2obo_enclave_codeset_id_dict_v2.0.0.json

Below is a draft DMP–style description of your credit‐card fraud detection experiment, modeled on the antiquities example:

1. Dataset Description

Research Domain
This work resides in the domain of financial fraud detection and applied machine learning. We focus on detecting anomalous credit‐card transactions in real time to reduce financial losses and improve trust in digital payment systems.

Purpose
The goal is to train and evaluate a binary classification model that flags potentially fraudulent transactions. By publishing both the code and data splits via FAIR repositories, we enable reproducible benchmarking of fraud‐detection algorithms and support future research on anomaly detection in transaction data.

Data Sources
We used the publicly available credit‐card transaction dataset from Kaggle (original source: https://www.kaggle.com/mlg-ulb/creditcardfraud), which contains anonymized transactions made by European cardholders over two days in September 2013. The dataset includes 284 807 transactions, of which 492 are fraudulent.

Method of Dataset Preparation

1. Schema validation: Renamed columns to snake_case (e.g. transaction_amount, is_declined) so they conform to DBRepo’s requirements.

2. Data import: Uploaded the full CSV into DBRepo, assigned persistent identifiers (PIDs).

3. Splitting: Programmatically derived three subsets—training (70%), validation (15%), test (15%)—using range‐based filters on the primary key actionnr. Each subset was materialized in DBRepo and assigned its own PID for precise citation.

4. Cleaning: Converted the categorical flags (is_declined, isforeigntransaction, ishighriskcountry, isfradulent) from “Y”/“N” to 1/0 and dropped non‐feature identifiers (actionnr, merchant_id).

5. Modeling: Trained a RandomForest classifier on the training split, tuned on validation, and evaluated on the held‐out test set.

2. Technical Details

Dataset Structure

• The raw data is a single CSV with columns:

  • actionnr (integer transaction ID)

  • merchant_id (string)

  • average_amount_transaction_day (float)

  • transaction_amount (float)

  • is_declined, isforeigntransaction, ishighriskcountry, isfradulent (binary flags)

  • total_number_of_declines_day, daily_chargeback_avg_amt, sixmonth_avg_chbk_amt, sixmonth_chbk_freq (numeric features)

Naming Conventions

• All columns use lowercase snake_case.

• Subsets are named creditcard_training, creditcard_validation, creditcard_test in DBRepo.

• Files in the code repo follow a clear structure:

  ├── data/         # local copies only; raw data lives in DBRepo 
  ├── notebooks/Task.ipynb 
  ├── models/rf_model_v1.joblib 
  ├── outputs/        # confusion_matrix.png, roc_curve.png, predictions.csv 
  ├── README.md 
  ├── requirements.txt 
  └── codemeta.json 
  

Required Software

• Python 3.9+

• pandas, numpy (data handling)

• scikit-learn (modeling, metrics)

• matplotlib (visualizations)

• dbrepo‐client.py (DBRepo API)

• requests (TU WRD API)

Additional Resources

• Original dataset: https://www.kaggle.com/mlg-ulb/creditcardfraud

• Scikit-learn docs: https://scikit-learn.org/stable

• DBRepo API guide: via the starter notebook’s dbrepo_client.py template

• TU WRD REST API spec: https://test.researchdata.tuwien.ac.at/api/docs

3. Further Details

Data Limitations

• Highly imbalanced: only ~0.17% of transactions are fraudulent.

• Anonymized PCA features (V1–V28) hidden; we extended with domain features but cannot reverse engineer raw variables.

• Time‐bounded: only covers two days of transactions, may not capture seasonal patterns.

Licensing and Attribution

• Raw data: CC-0 (per Kaggle terms)

• Code & notebooks: MIT License

• Model artifacts & outputs: CC-BY 4.0

• DUWRD records include ORCID identifiers for the author.

Recommended Uses

• Benchmarking new fraud‐detection algorithms on a standard imbalanced dataset.

• Educational purposes: demonstrating model‐training pipelines, FAIR data practices.

• Extension: adding time‐series or deep‐learning models.

Known Issues

• Possible temporal leakage if date/time features not handled correctly.

• Model performance may degrade on live data due to concept drift.

• Binary flags may oversimplify nuanced transaction outcomes.

Below is a draft DMP–style description of your credit‐card fraud detection experiment, modeled on the antiquities example:

1. Dataset Description

Research Domain
This work resides in the domain of financial fraud detection and applied machine learning. We focus on detecting anomalous credit‐card transactions in real time to reduce financial losses and improve trust in digital payment systems.

Purpose
The goal is to train and evaluate a binary classification model that flags potentially fraudulent transactions. By publishing both the code and data splits via FAIR repositories, we enable reproducible benchmarking of fraud‐detection algorithms and support future research on anomaly detection in transaction data.

Data Sources
We used the publicly available credit‐card transaction dataset from Kaggle (original source: https://www.kaggle.com/mlg-ulb/creditcardfraud), which contains anonymized transactions made by European cardholders over two days in September 2013. The dataset includes 284 807 transactions, of which 492 are fraudulent.

Method of Dataset Preparation

1. Schema validation: Renamed columns to snake_case (e.g. transaction_amount, is_declined) so they conform to DBRepo’s requirements.

2. Data import: Uploaded the full CSV into DBRepo, assigned persistent identifiers (PIDs).

3. Splitting: Programmatically derived three subsets—training (70%), validation (15%), test (15%)—using range‐based filters on the primary key actionnr. Each subset was materialized in DBRepo and assigned its own PID for precise citation.

4. Cleaning: Converted the categorical flags (is_declined, isforeigntransaction, ishighriskcountry, isfradulent) from “Y”/“N” to 1/0 and dropped non‐feature identifiers (actionnr, merchant_id).

5. Modeling: Trained a RandomForest classifier on the training split, tuned on validation, and evaluated on the held‐out test set.

2. Technical Details

Dataset Structure

• The raw data is a single CSV with columns:

  • actionnr (integer transaction ID)

  • merchant_id (string)

  • average_amount_transaction_day (float)

  • transaction_amount (float)

  • is_declined, isforeigntransaction, ishighriskcountry, isfradulent (binary flags)

  • total_number_of_declines_day, daily_chargeback_avg_amt, sixmonth_avg_chbk_amt, sixmonth_chbk_freq (numeric features)

Naming Conventions

• All columns use lowercase snake_case.

• Subsets are named creditcard_training, creditcard_validation, creditcard_test in DBRepo.

• Files in the code repo follow a clear structure:

  ├── data/         # local copies only; raw data lives in DBRepo 
  ├── notebooks/Task.ipynb 
  ├── models/rf_model_v1.joblib 
  ├── outputs/        # confusion_matrix.png, roc_curve.png, predictions.csv 
  ├── README.md 
  ├── requirements.txt 
  └── codemeta.json 
  

Required Software

• Python 3.9+

• pandas, numpy (data handling)

• scikit-learn (modeling, metrics)

• matplotlib (visualizations)

• dbrepo‐client.py (DBRepo API)

• requests (TU WRD API)

Additional Resources

• Original dataset: https://www.kaggle.com/mlg-ulb/creditcardfraud

• Scikit-learn docs: https://scikit-learn.org/stable

• DBRepo API guide: via the starter notebook’s dbrepo_client.py template

• TU WRD REST API spec: https://test.researchdata.tuwien.ac.at/api/docs

3. Further Details

Data Limitations

• Highly imbalanced: only ~0.17% of transactions are fraudulent.

• Anonymized PCA features (V1–V28) hidden; we extended with domain features but cannot reverse engineer raw variables.

• Time‐bounded: only covers two days of transactions, may not capture seasonal patterns.

Licensing and Attribution

• Raw data: CC-0 (per Kaggle terms)

• Code & notebooks: MIT License

• Model artifacts & outputs: CC-BY 4.0

• DUWRD records include ORCID identifiers for the author.

Recommended Uses

• Benchmarking new fraud‐detection algorithms on a standard imbalanced dataset.

• Educational purposes: demonstrating model‐training pipelines, FAIR data practices.

• Extension: adding time‐series or deep‐learning models.

Known Issues

• Possible temporal leakage if date/time features not handled correctly.

• Model performance may degrade on live data due to concept drift.

• Binary flags may oversimplify nuanced transaction outcomes.

This is an Australian extract of Speedtest Open data available at Amazon WS (link below - opendata.aws).AWS data licence is "CC BY-NC-SA 4.0", so use of this data must be:- non-commercial (NC)- reuse must be share-alike (SA)(add same licence).This restricts the standard CC-BY Figshare licence.A world speedtest open data was dowloaded (>400Mb, 7M lines of data). An extract of Australia's location (lat, long) revealed 88,000 lines of data (attached as csv).A Jupyter notebook of extract process is attached.See Binder version at Github - https://github.com/areff2000/speedtestAU.+> Install: 173 packages | Downgrade: 1 packages | Total download: 432MBBuild container time: approx - load time 25secs.=> Error: Timesout - BUT UNABLE TO LOAD GLOBAL DATA FILE (6.6M lines).=> Error: Overflows 8GB RAM container provided with global data file (3GB)=> On local JupyterLab M2 MBP; loads in 6 mins.Added Binder from ARDC service: https://binderhub.rc.nectar.org.auDocs: https://ardc.edu.au/resource/fair-for-jupyter-notebooks-a-practical-guide/A link to Twitter thread of outputs provided.A link to Data tutorial provided (GitHub), including Jupyter Notebook to analyse World Speedtest data, selecting one US State.Data Shows: (Q220)- 3.1M speedtests- 762,000 devices- 88,000 grid locations (600m * 600m), summarised as a point- average speed 33.7Mbps (down), 12.4M (up)- Max speed 724Mbps- data is for 600m * 600m grids, showing average speed up/down, number of tests, and number of users (IP). Added centroid, and now lat/long.See tweet of image of centroids also attached.NB: Discrepancy Q2-21, Speedtest Global shows June AU average speedtest at 80Mbps, whereas Q2 mean is 52Mbps (v17; Q1 45Mbps; v14). Dec 20 Speedtest Global has AU at 59Mbps. Could be possible timing difference. Or spatial anonymising masking shaping highest speeds. Else potentially data inconsistent between national average and geospatial detail. Check in upcoming quarters.NextSteps:Histogram - compare Q220, Q121, Q122. per v1.4.ipynb.Versions:v40: Added AUS Q125 (93k lines avg d/l 116.6 Mbps u/l 21.35 Mbps). Imported using v2 Jupyter notebook (MBP 16Gb). Mean tests: 16.9. Mean devices: 5.13. Download, extract and publish: 14 mins.v39: Added AUS Q424 (95k lines avg d/l 110.9 Mbps u/l 21.02 Mbps). Imported using v2 Jupyter notebook (MBP 16Gb). Mean tests: 17.2. Mean devices: 5.24. Download, extract and publish: 14 mins.v38: Added AUS Q324 (92k lines avg d/l 107.0 Mbps u/l 20.79 Mbps). Imported using v2 Jupyter notebook (iMac 32Gb). Mean tests: 17.7. Mean devices: 5.33.Added github speedtest-workflow-importv2vis.ipynb Jupyter added datavis code to colour code national map. (per Binder on Github; link below).v37: Added AUS Q224 (91k lines avg d/l 97.40 Mbps u/l 19.88 Mbps). Imported using speedtest-workflow-importv2 jupyter notebook. Mean tests:18.1. Mean devices: 5.4.v36 Load UK data, Q1-23 and compare to AUS and NZ Q123 data. Add compare image (au-nz-ukQ123.png), calc PlayNZUK.ipynb, data load import-UK.ipynb. UK data bit rough and ready as uses rectangle to mark out UK, but includes some EIRE and FR. Indicative only and to be definitively needs geo-clean to exclude neighbouring countries.v35 Load Melb geo-maps of speed quartiles (0-25, 25-50, 50-75, 75-100, 100-). Avg in 2020; 41Mbps. Avg in 2023; 86Mbps. MelbQ323.png, MelbQ320.png. Calc with Speedtest-incHist.ipynb code. Needed to install conda mapclassify. ax=melb.plot(column=...dict(bins[25,50,75,100]))v34 Added AUS Q124 (93k lines avg d/l 87.00 Mbps u/l 18.86 Mbps). Imported using speedtest-workflow-importv2 jupyter notebook. Mean tests:18.3. Mean devices: 5.5.v33 Added AUS Q423 (92k lines avg d/l 82.62 Mbps). Imported using speedtest-workflow-importv2 jupyter notebook. Mean tests:18.0. Mean devices: 5.6. Added link to Github.v32 Recalc Au vs NZ for upload performance; added image. using PlayNZ Jupyter. NZ approx 40% locations at or above 100Mbps. Aus

This data report contains the annotated figures, statistics and visualisations of the project "Die twitternde Zunft. Historikertage auf Twitter (2012-2018)" by Mareike König and Paul Ramisch. In addition, the methodological approach to corpus creation, data cleaning, coding, network and text analysis as well as the legal and ethical considerations of the project are described.

The datasheets contain the dehydrated and annotated tweet ids that were used for our study. With the Twitter API this can be used to hydrate and restore the whole corpus, apart from deleted tweets. There are two versions of the CSV file, one with clean id values, the other where the id values are prepended with an “x”. This prevents certain tools from using scientific notation for the ids and breaking them, with the R library rtweet function read_twitter_csv() this is automatically resolved on import.

The files contain the following data:

  status_id: The Twitter status id of the tweet

  corpus_user_id: A corpus specific id for each user within the corpus (not the Twitter user id)

  hauptkategorie_1: Primary category

  hauptkategorie_2: Primary category 2

  Gender: Gender of the user

  Nebenkategorie: Secondary category

Furthermore, the following boolean variables describe what sub corpus each tweet is in, the main corpus per year that contains of both data sources (TAGS and API) and the yearly sub corpora divided by their data source (TAGS: orig_, API: api_):

You can find the code on R on GitHub: https://github.com/dhiparis/historikertag-twitter.

WikiHow is a new large-scale dataset using the online WikiHow (http://www.wikihow.com/) knowledge base.

There are two features: - text: wikihow answers texts. - headline: bold lines as summary.

There are two separate versions: - all: consisting of the concatenation of all paragraphs as the articles and the bold lines as the reference summaries. - sep: consisting of each paragraph and its summary.

Download "wikihowAll.csv" and "wikihowSep.csv" from https://github.com/mahnazkoupaee/WikiHow-Dataset and place them in manual folder https://www.tensorflow.org/datasets/api_docs/python/tfds/download/DownloadConfig. Train/validation/test splits are provided by the authors. Preprocessing is applied to remove short articles (abstract length < 0.75 article length) and clean up extra commas.

To use this dataset:

import tensorflow_datasets as tfds

ds = tfds.load('wikihow', split='train')
for ex in ds.take(4):
 print(ex)

See the guide for more informations on tensorflow_datasets.

Title: Natural Gas Price Determinants: A Comprehensive Dataset

Description: This dataset provides a rich and detailed exploration of various factors influencing natural gas prices, offering valuable insights for researchers, analysts, and policymakers. The data is sourced from AEMO (Australian Energy Market Operator), ensuring the highest level of accuracy and reliability.

Key Features: Comprehensive Parameter Coverage: The dataset includes a wide range of variables relevant to natural gas pricing, such as: Supply Factors: Gas production rates, storage levels, and pipeline capacities. * Demand Factors: Consumption patterns, industrial usage, and residential demand. * Economic Indicators: GDP growth, inflation rates, and consumer confidence. * Weather Conditions: Temperature variations, precipitation, and extreme weather events. * Geopolitical Factors: International conflicts, trade policies, and regulatory changes. * Time Series Data: The dataset spans multiple years, allowing for in-depth analysis of price trends, seasonality, and long-term correlations. * Granular Level of Detail: Data is provided at a granular level, enabling detailed examination of price fluctuations across different regions and time periods. * Clean and Standardized Format: The dataset is carefully curated and standardized to ensure data quality and consistency.

Potential Use Cases:

• Price Forecasting: Researchers can develop accurate models to predict future natural gas prices based on historical data and identified trends.
• Risk Assessment: Analysts can assess the impact of various factors on price volatility and identify potential risks for market participants.
• Policy Analysis: Policymakers can evaluate the effectiveness of different regulatory measures and pricing strategies.
• Investment Decisions: Investors can make informed decisions about investments in the natural gas sector by understanding the underlying drivers of price fluctuations.

Dataset Format:

• CSV: The dataset will be provided in a CSV format for easy import and analysis using popular data science tools.

Acknowledgements:

We would like to thank AEMO for providing the data and supporting this research.

Keywords: natural gas, price, determinants, factors, AEMO, dataset, analysis, forecasting, risk, policy, investment.