This is the dataset used for pre-training in "ReasonBERT: Pre-trained to Reason with Distant Supervision", EMNLP'21.

There are two files:

sentence_pairs_for_pretrain_no_tokenization.tar.gz -> Contain only sentences as evidence, Text-only

table_pairs_for_pretrain_no_tokenization.tar.gz -> At least one piece of evidence is a table, Hybrid

The data is chunked into multiple tar files for easy loading. We use WebDataset, a PyTorch Dataset (IterableDataset) implementation providing efficient sequential/streaming data access.

For pre-training code, or if you have any questions, please check our GitHub repo https://github.com/sunlab-osu/ReasonBERT

Below is a sample code snippet to load the data

import webdataset as wds

path to the uncompressed files, should be a directory with a set of tar files

url = './sentence_multi_pairs_for_pretrain_no_tokenization/{000000...000763}.tar' dataset = ( wds.Dataset(url) .shuffle(1000) # cache 1000 samples and shuffle .decode() .to_tuple("json") .batched(20) # group every 20 examples into a batch )

Please see the documentation for WebDataset for more details about how to use it as dataloader for Pytorch

You can also iterate through all examples and dump them with your preferred data format

Below we show how the data is organized with two examples.

Text-only

{'s1_text': 'Sils is a municipality in the comarca of Selva, in Catalonia, Spain.', # query sentence 's1_all_links': { 'Sils,_Girona': [[0, 4]], 'municipality': [[10, 22]], 'Comarques_of_Catalonia': [[30, 37]], 'Selva': [[41, 46]], 'Catalonia': [[51, 60]] }, # list of entities and their mentions in the sentence (start, end location) 'pairs': [ # other sentences that share common entity pair with the query, group by shared entity pairs { 'pair': ['Comarques_of_Catalonia', 'Selva'], # the common entity pair 's1_pair_locs': [[[30, 37]], [[41, 46]]], # mention of the entity pair in the query 's2s': [ # list of other sentences that contain the common entity pair, or evidence { 'md5': '2777e32bddd6ec414f0bc7a0b7fea331', 'text': 'Selva is a coastal comarque (county) in Catalonia, Spain, located between the mountain range known as the Serralada Transversal or Puigsacalm and the Costa Brava (part of the Mediterranean coast). Unusually, it is divided between the provinces of Girona and Barcelona, with Fogars de la Selva being part of Barcelona province and all other municipalities falling inside Girona province. Also unusually, its capital, Santa Coloma de Farners, is no longer among its larger municipalities, with the coastal towns of Blanes and Lloret de Mar having far surpassed it in size.', 's_loc': [0, 27], # in addition to the sentence containing the common entity pair, we also keep its surrounding context. 's_loc' is the start/end location of the actual evidence sentence 'pair_locs': [ # mentions of the entity pair in the evidence [[19, 27]], # mentions of entity 1 [[0, 5], [288, 293]] # mentions of entity 2 ], 'all_links': { 'Selva': [[0, 5], [288, 293]], 'Comarques_of_Catalonia': [[19, 27]], 'Catalonia': [[40, 49]] } } ,...] # there are multiple evidence sentences }, ,...] # there are multiple entity pairs in the query }

Hybrid

{'s1_text': 'The 2006 Major League Baseball All-Star Game was the 77th playing of the midseason exhibition baseball game between the all-stars of the American League (AL) and National League (NL), the two leagues comprising Major League Baseball.', 's1_all_links': {...}, # same as text-only 'sentence_pairs': [{'pair': ..., 's1_pair_locs': ..., 's2s': [...]}], # same as text-only 'table_pairs': [ 'tid': 'Major_League_Baseball-1', 'text':[ ['World Series Records', 'World Series Records', ...], ['Team', 'Number of Series won', ...], ['St. Louis Cardinals (NL)', '11', ...], ...] # table content, list of rows 'index':[ [[0, 0], [0, 1], ...], [[1, 0], [1, 1], ...], ...] # index of each cell [row_id, col_id]. we keep only a table snippet, but the index here is from the original table. 'value_ranks':[ [0, 0, ...], [0, 0, ...], [0, 10, ...], ...] # if the cell contain numeric value/date, this is its rank ordered from small to large, follow TAPAS 'value_inv_ranks': [], # inverse rank 'all_links':{ 'St._Louis_Cardinals': { '2': [ [[2, 0], [0, 19]], # [[row_id, col_id], [start, end]] ] # list of mentions in the second row, the key is row_id }, 'CARDINAL:11': {'2': [[[2, 1], [0, 2]]], '8': [[[8, 3], [0, 2]]]}, } 'name': '', # table name, if exists 'pairs': { 'pair': ['American_League', 'National_League'], 's1_pair_locs': [[[137, 152]], [[162, 177]]], # mention in the query 'table_pair_locs': { '17': [ # mention of entity pair in row 17 [ [[17, 0], [3, 18]], [[17, 1], [3, 18]], [[17, 2], [3, 18]], [[17, 3], [3, 18]] ], # mention of the first entity [ [[17, 0], [21, 36]], [[17, 1], [21, 36]], ] # mention of the second entity ] } } ] }

Context

This is a dataset for the TecoGan Pytorch model. The Github repo can be found here.

Content

There are 400 scenes from the UCF101 dataset. Each video was split into photos with a maximum length of 120 photos. The photos were put into this dataset in the format that the TecoGan dataloader takes.

Acknowledgements

The original UCF101 dataset can be found here. And you can find the original TecoGan repo here.

Inspiration

Let's see how good your super resolution images can look. How close can you get to the original?

On the Generalization of WiFi-based Person-centric Sensing in Through-Wall Scenarios

This repository contains the 3DO dataset proposed in [1].

PyTroch Dataloader

A minimal PyTorch dataloader for the 3DO dataset is provided at: https://github.com/StrohmayerJ/3DO

Dataset Description

The 3DO dataset comprises 42 five-minute recordings (~1.25M WiFi packets) of three human activities performed by a single person, captured in a WiFi through-wall sensing scenario over three consecutive days. Each WiFi packet is annotated with a 3D trajectory label and a class label for the activities: no person/background (0), walking (1), sitting (2), and lying (3). (Note: The labels returned in our dataloader example are walking (0), sitting (1), and lying (2), because background sequences are not used.)

The directories 3DO/d1/, 3DO/d2/, and 3DO/d3/ contain the sequences from days 1, 2, and 3, respectively. Furthermore, each sequence directory (e.g., 3DO/d1/w1/) contains a csiposreg.csv file storing the raw WiFi packet time series and a csiposreg_complex.npy cache file, which stores the complex Channel State Information (CSI) of the WiFi packet time series. (If missing, csiposreg_complex.npy is automatically generated by the provided dataloader.)

Dataset Structure:

/3DO

├── d1 <-- day 1 subdirectory

  └── w1 <-- sequence subdirectory

     └── csiposreg.csv <-- raw WiFi packet time series

     └── csiposreg_complex.npy <-- CSI time series cache

├── d2 <-- day 2 subdirectory

├── d3 <-- day 3 subdirectory

In [1], we use the following training, validation, and test split:

Subset Day Sequences

Train 1 w1, w2, w3, s1, s2, s3, l1, l2, l3

Val 1 w4, s4, l4

Test 1 w5 , s5, l5

Test 2 w1, w2, w3, w4, w5, s1, s2, s3, s4, s5, l1, l2, l3, l4, l5

Test 3 w1, w2, w4, w5, s1, s2, s3, s4, s5, l1, l2, l4

w = walking, s = sitting and l= lying

Note: On each day, we additionally recorded three ten-minute background sequences (b1, b2, b3), which are provided as well.

Download and UseThis data may be used for non-commercial research purposes only. If you publish material based on this data, we request that you include a reference to our paper [1].

[1] Strohmayer, J., Kampel, M. (2025). On the Generalization of WiFi-Based Person-Centric Sensing in Through-Wall Scenarios. In: Pattern Recognition. ICPR 2024. Lecture Notes in Computer Science, vol 15315. Springer, Cham. https://doi.org/10.1007/978-3-031-78354-8_13

BibTeX citation:

@inproceedings{strohmayerOn2025, author="Strohmayer, Julian and Kampel, Martin", title="On the Generalization of WiFi-Based Person-Centric Sensing in Through-Wall Scenarios", booktitle="Pattern Recognition", year="2025", publisher="Springer Nature Switzerland", address="Cham", pages="194--211", isbn="978-3-031-78354-8" }

Feral Cat Segmentation Dataset

Overview

This dataset provides image segmentation data for feral cats, designed for computer vision and machine learning tasks. It builds upon the original public domain dataset by Paul Cashman from Roboflow, with additional preprocessing and multiple data formats for easier consumption.

Dataset Source

• Original Author: Paul Cashman
• Original Source: Roboflow Universe
• Extended by: Lu Hou Yang
• GitHub: https://github.com/luhouyang/open_circles
• License: Public Domain

Dataset Contents

The dataset is organized into three standard splits: - Train set - Validation set - Test set

Each split contains data in multiple formats: 1. Original JPG images 2. Segmentation mask JPG images 3. Parquet files containing flattened image and mask data 4. Pickle files containing serialized image and mask data

Data Formats

1. Image Files

• Format: JPG
• Resolution: 224×224 pixels
• Directory Structure:
  • train/: Original training images
  • valid/: Original validation images
  • test/: Original test images
  • train_mask/: Corresponding segmentation masks for training
  • valid_mask/: Corresponding segmentation masks for validation
  • test_mask/: Corresponding segmentation masks for testing

2. Parquet Files

• Files: train_dataset.parquet, valid_dataset.parquet, test_dataset.parquet
• Content: Flattened image data and corresponding masks combined in a single table
• Structure: Each row contains the flattened pixel values of an image followed by the flattened pixel values of its mask
• Data Division: Image and mask data are split at index split_at = image_size[0] * image_size[1] * image_channels
  • Data before this index: image pixel values (reshaped to [-1, 224, 224, 3])
  • Data after this index: mask pixel values (reshaped to [-1, 224, 224, 1])
• Benefits: Efficient storage and faster loading compared to individual image files

3. Pickle Files

• Files: train_dataset.pkl, valid_dataset.pkl, test_dataset.pkl
• Content: Serialized Python objects containing images and their corresponding masks
• Structure: List of [image, mask] pairs, where each image and mask is serialized using Python's pickle
• Data Access: Similar to parquet files, when loaded through the provided dataset class, data is split at the same index: split_at = image_size[0] * image_size[1] * image_channels
• Benefits: Preserves original data structure and enables quick loading in Python

4. CSV Files

• Files: train_dataset.csv, valid_dataset.csv, test_dataset.csv
• Content: Same data as parquet files but in CSV format
• Structure: No headers, raw flattened pixel values
• Data Division: Same split point as parquet files

Image Preprocessing

All images were preprocessed with the following operations: - Resized to 224×224 pixels using bilinear interpolation - Segmentation masks were also resized to match the images using nearest neighbor interpolation - Original RLE (Run-Length Encoding) segmentation data converted to binary masks

Data Normalization

When used with the provided PyTorch dataset class, images are normalized with: - Mean: [0.48235, 0.45882, 0.40784] - Standard Deviation: [0.00392156862745098, 0.00392156862745098, 0.00392156862745098]

PyTorch Integration

A custom CatDataset class is included for easy integration with PyTorch:

from cat_dataset import CatDataset

# Load from parquet format
dataset = CatDataset(
  root="path/to/dataset",
  split="train", # Options: "train", "valid", "test"
  format="parquet", # Options: "parquet", "pkl"
  image_size=[224, 224],
  image_channels=3,
  mask_channels=1
)

# Use with PyTorch DataLoader
from torch.utils.data import DataLoader
dataloader = DataLoader(dataset, batch_size=32, shuffle=True)

Performance Comparison

Loading time benchmarks from the original implementation: - Parquet format: ~1.29 seconds per iteration - Pickle format: ~0.71 seconds per iteration

The pickle format provides the fastest loading times and is recommended for most use cases.

Citation

If you use this dataset in your research or projects, please cite:

@misc{feral-cat-segmentation_dataset,
 title = {feral-cat-segmentation Dataset},
 type = {Open Source Dataset},
 author = {Paul Cashman},
 howpublished = {\url{https://universe.roboflow.com/paul-cashman-mxgwb/feral-cat-segmentation}},
 url = {https://universe.roboflow.com/paul-cashman-mxgwb/feral-cat-segmentation},
 journal = {Roboflow Universe},
 publisher = {Roboflow},
 year = {2025},
 month = {mar},
 note = {visited on 2025-03-19},
}

Sample Usage Code

Basic Dataset Loading

from ca...

Official Repository for the Real Bokeh Dataset

  This dataset was presented as part of our paper Bokehlicious: Photorealistic Bokeh Rendering with Controllable Apertures at ICCV 2025 (HF Papers)

You can find the code to our Bokeh Rendering solution and a PyTorch Dataloader for this dataset in the official code repository! If you find our dataset useful for your research work please cite: @inproceedings{seizinger2025bokehlicious, author = {Seizinger, Tim and Vasluianu… See the full description on the dataset page: https://huggingface.co/datasets/timseizinger/RealBokeh_3MP.

The University of Turin (UniTO) released the open-access dataset UniTOBrain collected for the homonymous Use Case 3 in the DeepHealth project (https://deephealth-project.eu/). UniToBrain is a dataset of Computed Tomography (CT) perfusion images (CTP). The dataset includes 100 training subjects and 15 testing subjects used in a submitted publication for the training and the testing of a Convolutional Neural Network (CNN, see for details: https://arxiv.org/abs/2101.05992, https://paperswithcode.com/paper/neural-network-derived-perfusion-maps-a-model, https://www.medrxiv.org/content/10.1101/2021.01.13.21249757v1). At this stage, the UniTO team released this dataset privately, but soon it will be public. This is a subsample of a greater dataset of 258 subjects that will be soon available for download at https://ieee-dataport.org/. CTP data from 258 consecutive patients were retrospectively obtained from the hospital PACS of Città della Salute e della Scienza di Torino (Molinette). CTP acquisition parameters were as follows: Scanner GE, 64 slices, 80 kV, 150 mAs, 44.5 sec duration, 89 volumes (40 mm axial coverage), injection of 40 ml of Iodine contrast agent (300 mg/ml) at 4 ml/s speed. Along with the dataset, we provide some utility files. dicomtonpy.py: It converts the dicom files in the dataset to numpy arrays. These are 3D arrays, where CT slices at the same height are piled-up over the temporal acquisition. dataloader_pytorch.py: Dataloader for the pytorch deep learning framework. It converts the numpy arrays in normalized tensors, which can be provided as input to standard deep learning models. dataloader_pyeddl.py: Dataloader for the pyeddl deep learning framework. It converts the numpy arrays in normalized tensors, which can be provided as input to standard deep learning models using the european library EDDL. Visit https://github.com/EIDOSlab/UC3-UNITOBrain to have a full companion code where a U-Net model is trained over the dataset.

WiFi CSI-based Long-Range Person Localization Using Directional Antennas

This repository contains the HAllway LOCalization (HALOC) dataset and WiFi system CAD files as proposed in [1].

PyTroch Dataloader

A minimal PyTorch dataloader for the HALOC dataset is provided at: https://github.com/StrohmayerJ/HALOC

Dataset Description

The HALOC dataset comprises six sequences (in .csv format) of synchronized WiFi Channel State Information (CSI) and 3D position labels. Each row in a given .csv file represents a single WiFi packet captured via ESP-IDF, with CSI and 3D coordinates stored in the "data" and ("x", "y", "z") fields, respectively.

The sequences are divided into training, validation, and test subsets as follows:

Subset Sequences

Training 0.csv, 1.csv, 2.csv and 3.csv

Validation 4.csv

Test 5.csv

WiFi System CAD files

We provide CAD files for the 3D printable parts of the proposed WiFi system consisting of the main housing (housing.stl), the lid (lid.stl), and the carrier board (carrier.stl) featuring mounting points for the Nvidia Jetson Orin Nano and the ESP32-S3-DevKitC-1 module.

Download and UseThis data may be used for non-commercial research purposes only. If you publish material based on this data, we request that you include a reference to our paper [1].

[1] Strohmayer, J., and Kampel, M. (2024). “WiFi CSI-based Long-Range Person Localization Using Directional Antennas”, The Second Tiny Papers Track at ICLR 2024, May 2024, Vienna, Austria. https://openreview.net/forum?id=AOJFcEh5Eb

BibTeX citation:

@inproceedings{strohmayer2024wifi,title={WiFi {CSI}-based Long-Range Person Localization Using Directional Antennas},author={Julian Strohmayer and Martin Kampel},booktitle={The Second Tiny Papers Track at ICLR 2024},year={2024},url={https://openreview.net/forum?id=AOJFcEh5Eb}}

Inside the SynthSOD-data folder, there is a folder for every one of the songs of the dataset and inside them, there is a folder called Tree with the signals synthesized for the Decca Tree (which provide a reasonable stereo mix with the original reverberation of the synthesizer) and a folder called Close Mic with the signals synthesized for the close mics of the instruments (which are the driest signals generated by the synthesizer and can be used as source signals if wanting to add custom reverberation). Inside these folders are the FLAC files of the instruments present in the mix, which should be at least two of the followings: Violin_1.flac, Violin_2.flac, Viola.flac, Cello.flac, Bass.flac, Flute.flac, Piccolo.flac, Clarinet.flac, Oboe.flac, coranglais.flac, Bassoon.flac, Horn.flac, Trumpet.flac, Trombone.flac, Tuba.flac, Harp.flac, Timpani.flac, and untunedpercussion.flac. The file SynthSOD_metadata_all.json contains information about the instruments present in the dataset and the activity time of every one of them and their combinations for the whole dataset and for every one of the songs as well as the ID of every song in the SOD. The files SynthSOD_metadata_train.json, SynthSOD_metadata_evaluation.json, and SynthSOD_metadata_test.json contain the same information but only for the songs in the official train, evaluation, and test partitions of the dataset. Note that the folder SynthSOD-data contains the songs for all the partitions without any splits, so the information about the partitions is only in the JSON files. You can find an example of a PyTorch dataloader for the dataset in the repository of the baseline model. The compressed file SynthSOD-sample.zip is just a subset of the full dataset with 10 pieces that can be downloaded to take a look/listen to the data before downloading the full dataset.

This directory contains the training data and code for training and testing a ResMLP with experience replay for creating a machine-learning physics parameterization for the Community Atmospheric Model.

The directory is structured as follows:

1. Download training and testing data: https://portal.nersc.gov/archive/home/z/zhangtao/www/hybird_GCM_ML

2. Unzip nncam_training.zip

nncam_training

- models

model definition of ResMLP and other models for comparison purposes

- dataloader

utility scripts to load data into pytorch dataset

- training_scripts

scripts to train ResMLP model with/without experience replay

- offline_test

scripts to perform offline test (Table 2, Figure 2)

3. Unzip nncam_coupling.zip

nncam_srcmods

- SourceMods

SourceMods to be used with CAM modules for coupling with neural network

- otherfiles

additional configuration files to setup and run SPCAM with neural network

- pythonfiles

python scripts to run neural network and couple with CAM

- ClimAnalysis

- paper_plots.ipynb

scripts to produce online evaluation figures (Figure 1, Figure 3-10)

The MELD Preprocessed Dataset is a multi-modal dataset designed for research on emotion recognition from audio, video, and textual data. The dataset builds upon the original MELD dataset and applies extensive preprocessing steps to extract features from different modalities. Each sample is saved as a .pt file containing a dictionary of preprocessed features, making it easy for developers to load and integrate into PyTorch-based workflows.

Data Sources

• Audio: Waveforms extracted from the original video files.
• Video: Video files are processed to sample frames at a target frame rate (default: 2 fps) and to detect faces using a Haar Cascade classifier.
• Text: Utterances from the dialogue, which are cleaned using custom encoding functions to fix potential byte encoding issues.
• Emotion Labels: Each sample is associated with an emotion label.

Preprocessing Pipeline

The preprocessing script performs several key steps:

1. Text Cleaning:

   • fix_encoding_with_bytes(text): Decodes text from bytes using UTF-8, Latin-1, or cp1252, ensuring correct encoding.
   • replace_double_encoding(text): Fixes issues related to double-encoded characters (e.g., replacing "Â’" with the proper apostrophe).

2. Audio Processing:

   • Extracts raw audio waveform from each sample.
   • Computes a Mel-spectrogram using torchaudio.transforms.MelSpectrogram with 64 mel bins (VGGish format).
   • Converts the spectrogram to a logarithmic scale for numerical stability.

3. Video Processing:

   • Reads video frames at a specified target FPS (default: 2 fps) using OpenCV.
   • For each video, samples frames evenly based on the original video's FPS.
   • Applies Haar Cascade face detection on the frames to extract the first detected face.
   • Resizes the detected face to 224x224 and converts it to RGB. If no face is detected, a default black image (224x224x3) is returned.

4. Saving Processed Samples:

   • Each sample is saved as a .pt file in a directory structure split by data type (train, dev, and test).
   • The filename is derived from the original video filename (e.g., dia0_utt1.mp4 becomes dia0_utt1.pt).

Data Format

Each preprocessed sample is stored in a .pt file and contains a dictionary with the following keys:

• utterance (str): The cleaned textual utterance.
• emotion (str/int): The corresponding emotion label.
• video_path (str): Original path to the video file from which the sample was extracted.
• audio (Tensor): Raw audio waveform tensor of shape [channels, time].
• audio_sample_rate (int): The sampling rate of the audio waveform.
• audio_mel (Tensor): The computed log-scaled Mel-spectrogram with shape [channels, n_mels, time].
• face (NumPy array): The extracted face image (RGB format) of shape (224, 224, 3). If no face was detected, a default black image is provided.

Directory Structure

The preprocessed files are organized into splits: preprocessed_data/ ├── train/ │ ├── dia0_utt0.pt │ ├── dia1_utt1.pt │ └── ... ├── dev/ │ ├── dia0_utt0.pt │ ├── dia1_utt1.pt │ └── ... └── test/ │ ├── dia0_utt0.pt │ ├── dia1_utt1.pt └── ...

Loading and Using the Dataset

A custom PyTorch dataset and DataLoader are provided to facilitate easy integration:

Dataset Class

from torch.utils.data import Dataset
import os
import torch

class PreprocessedMELDDataset(Dataset):
  def _init_(self, data_dir):
    """
    Args:
      data_dir (str): Directory where preprocessed .pt files are stored.
    """
    self.data_dir = data_dir
    self.files = [os.path.join(data_dir, f) for f in os.listdir(data_dir) if f.endswith('.pt')]
    
  def _len_(self):
    return len(self.files)
  
  def _getitem_(self, idx):
    sample_path = self.files[idx]
    sample = torch.load(sample_path)
    return sample

Custom Collate Function

def preprocessed_collate_fn(batch):
  """
  Collates a list of sample dictionaries into a single dictionary with keys mapping to lists.
  Modify this function to pad or stack tensor data if needed.
  """
  collated = {}
  collated['utterance'] = [sample['utterance'] for sample in batch]
  collated['emotion'] = [sample['emotion'] for sample in batch]
  collated['video_path'] = [sample['video_path'] for sample in batch]
  collated['audio'] = [sample['audio'] for sample in batch]
  collated['audio_sample_rate'] = batch[0]['audio_sample_rate']
  collated['audio_mel'] = [sample['audio_mel'] for sample in batch]
  collated['face'] = [sample['face'] for sample in batch]
  return collated

Creating DataLoaders

from torch.utils.data import DataLoader

# Define paths for each split
train_data_dir = "preprocessed_data/train"
dev_data_dir = "preproces...

SpecTf: Transformers Enable Data-Driven Imaging Spectroscopy Cloud Detection

Summary

Manuscript in review. Preprint: https://arxiv.org/abs/2501.04916

This repository contains the dataset used to train and evaluate the Spectroscopic Transformer model for EMIT cloud screening.

• spectf_cloud_labelbox.hdf5
  • 1,841,641 Labeled spectra from 221 EMIT Scenes.
• spectf_cloud_mmgis.hdf5
  • 1,733,801 Labeled spectra from 313 EMIT Scenes.
  • These scenes were speciffically labeled to correct false detections by an earlier version of the model.
• train_fids.csv
  • 465 EMIT scenes comprising the training set.
• test_fids.csv
  • 69 EMIT scenes comprising the held-out validation set.

v2 adds validation_scenes.pdf, a PDF displaying the 69 validation scenes in RGB and Falsecolor, their existing baseline cloud masks, as well as their cloud masks produced by the ANN and GBT reference models and the SpecTf model.

Data Description

221 EMIT Scenes were initially selected for labeling with diversity in mind. After sparse segmentation labeling of confident regions in Labelbox, up to 10,000 spectra were selected per-class per-scene to form the spectf_cloud_labelbox dataset. We deployed a preliminary model trained on these spectra on all EMIT scenes observed in March 2024, then labeled another 313 EMIT Scenes using MMGIS's polygonal labeling tool to correct false positive and false negative detections. After similarly sampling spectra from these scenes, A total of 3,575,442 spectra were labeled and sampled.

The train/test split was randomly determined by scene FID to prevent the same EMIT scene from contributing spectra to both the training and validation datasets.

Please refer to Section 4.2 in the paper for a complete description, and to our code repository for example usage and a Pytorch dataloader.

Each hdf5 file contains the following arrays:

• 'spectra'
  • Top-of-Atmosphere reflectance calculated from the EMIT L1B Radiance product
  • Float64 of shape (n, 268)
• 'fids'
  • The FID from which each spectrum was sampled
  • Binary string of shape (n,)
• 'indices'
  • The (col, row) index from which each spectrum was sampled
  • Int64 of shape (n, 2)
• 'labels'
  • Annotation label of each spectrum
    • 0 - "Clear"
    • 1 - "Cloud"
    • 2 - "Cloud Shadow" (Only for the Labelbox dataset, and this class was combined with the clear class for this work. See paper for details.)
      • label[label==2] = 0
  • Int64 of shape (n,2)

Each hdf5 file contains the following attribute:

• 'bands'
  • The band center wavelengths (nm) of the spectrum
  • Float64 of shape (268,)

Acknowledgements

The EMIT online mapping tool was developed by the JPL MMGIS team. The High Performance Computing resources used in this investigation were provided by funding from the JPL Information and Technology Solutions Directorate.

This research was carried out at the Jet Propulsion Laboratory, California Institute of Technology, under a contract with the National Aeronautics and Space Administration (80NM0018D0004).

© 2024 California Institute of Technology. Government sponsorship acknowledged.

Image dataset of mediterranen fruit fly. Dataset consists of: - 169 training images - 39 test images You can find labels for each file in train_labels.jons and test_labels.json. There is a notebook in the code section which converts the labels to TFRecordsDataset. Notebook for PyTorch Dataloader will be uploaded soon.

Duck Hunt Object Detection Dataset

This dataset contains 1,004 labeled images from the classic NES game "Duck Hunt" (1984), specifically prepared for YOLO (You Only Look Once) object detection training. The dataset includes sprites of the iconic hunting dog and ducks in various states, augmented to provide a balanced and comprehensive training set for computer vision models.

Perfect for: - Object detection model training - Computer vision research - Retro gaming AI projects - YOLO algorithm benchmarking - Educational purposes

🎯 Dataset Statistics

Class Distribution

📁 Dataset Structure

yolo_dataset_augmented/
├── images/
│  ├── train/      # 711 training images
│  └── val/       # 260 validation images
├── labels/
│  ├── train/      # 711 YOLO annotation files
│  └── val/       # 260 YOLO annotation files
├── classes.txt     # Class names mapping
├── dataset.yaml     # YOLO configuration file
└── augmented_dataset_stats.json # Detailed statistics

🔧 Data Augmentation Details

The original 47 images were enhanced using advanced data augmentation techniques to create a balanced dataset:

Augmentation Techniques Applied:

• Geometric Transformations: Rotation (±15°), horizontal/vertical flipping, scaling (0.8-1.2x), translation
• Color Adjustments: Brightness (0.7-1.3x), contrast (0.8-1.2x), saturation (0.8-1.2x)
• Quality Variations: Gaussian noise, slight blur for robustness
• Advanced Techniques: Mosaic augmentation (YOLO-style 4-image combination)

Augmentation Parameters:

{
  'rotation_range': (-15, 15),    # Small rotations for game sprites
  'brightness_range': (0.7, 1.3),  # Brightness variations
  'contrast_range': (0.8, 1.2),   # Contrast adjustments
  'saturation_range': (0.8, 1.2),  # Color saturation
  'noise_intensity': 0.02,      # Gaussian noise
  'horizontal_flip_prob': 0.5,    # 50% chance horizontal flip
  'scaling_range': (0.8, 1.2),    # Scale variations
}

🚀 Usage Examples

Loading with YOLOv8 (Ultralytics)

from ultralytics import YOLO

# Load and train
model = YOLO('yolov8n.pt') # Load pretrained model
results = model.train(data='dataset.yaml', epochs=100, imgsz=640)

# Validate
metrics = model.val()

# Predict
results = model('path/to/test/image.png')

Loading with PyTorch

import torch
from torch.utils.data import Dataset, DataLoader
from PIL import Image
import os

class DuckHuntDataset(Dataset):
  def _init_(self, images_dir, labels_dir, transform=None):
    self.images_dir = images_dir
    self.labels_dir = labels_dir
    self.transform = transform
    self.images = os.listdir(images_dir)
  
  def _len_(self):
    return len(self.images)
  
  def _getitem_(self, idx):
    img_path = os.path.join(self.images_dir, self.images[idx])
    label_path = os.path.join(self.labels_dir, 
                 self.images[idx].replace('.png', '.txt'))
    
    image = Image.open(img_path)
    # Load YOLO annotations
    with open(label_path, 'r') as f:
      labels = f.readlines()
    
    if self.transform:
      image = self.transform(image)
      
    return image, labels

# Usage
dataset = DuckHuntDataset('images/train', 'labels/train')
dataloader = DataLoader(dataset, batch_size=32, shuffle=True)

YOLO Annotation Format

Each .txt file contains one line per object: class_id center_x center_y width height

Example annotation: 0 0.492 0.403 0.212 0.315 Where values are normalized (0-1) relative to image dimensions.

📊 Technical Specifications

• Image Dimensions: Variable (original sprite sizes preserved)
• Color Channels: RGB (3 channels)
• Annotation Precision: Float32 (normalized coordinates)
• File Naming: Descriptive names indicating class and augmentation type
• Quality: High-resolution pixel art sprites

🎮 Dataset Context

This dataset is based on sprites from the iconic 1984 NES game "Duck Hunt," one of the most recognizable video games in history. The game featured:

• The Dog: Your hunting companion who retrieves ducks and ...