Coast Train is a library of images of coastal environments, annotations, and corresponding thematic label masks (or ‘label images’) collated for the purposes of training and evaluating machine learning (ML), deep learning, and other models for image segmentation. It includes image sets from both geospatial satellite, aerial, and UAV imagery and orthomosaics, as well as non-geospatial oblique and nadir imagery. Images include a diverse range of coastal environments from the U.S. Pacific, Gulf of Mexico, Atlantic, and Great Lakes coastlines, consisting of time-series of high-resolution (≤1m) orthomosaics and satellite image tiles (10–30m). Each image, image annotation, and labelled image is available as a single NPZ zipped file. NPZ files follow the following naming convention: {datasource}{numberofclasses}{threedigitdatasetversion}.zip, where {datasource} is the source of the original images (for example, NAIP, Landsat 8, Sentinel 2), {numberofclasses} is the number of classes used to annotate the images, and {threedigitdatasetversion} is the three-digit code corresponding to the dataset version (in other words, 001 is version 1). Each zipped folder contains a collection of NPZ format files, each of which corresponds to an individual image. An individual NPZ file is named after the image that it represents and contains (1) a CSV file with detail information for every image in the zip folder and (2) a collection of the following NPY files: orig_image.npy (original input image unedited), image.npy (original input image after color balancing and normalization), classes.npy (list of classes annotated and present in the labelled image), doodles.npy (integer image of all image annotations), color_doodles.npy (color image of doodles.npy), label.npy (labelled image created from the classes present in the annotations), and settings.npy (annotation and machine learning settings used to generate the labelled image from annotations). All NPZ files can be extracted using the utilities available in Doodler (Buscombe, 2022). A merged CSV file containing detail information on the complete imagery collection is available at the top level of this data release, details of which are available in the Entity and Attribute section of this metadata file.

Cape Hatteras Landsat8 RGB Images and Labels for Image Segmentation using the program, Segmentation Gym

Overview

• Test datasets and files for testing the segmentation gym program for image segmentation
• Data set made by Daniel Buscombe, Marda Science LLC.
• Dataset consists of a time-series of Landsat-8 images of Cape Hatteras National Seashore, courtesy of the U.S. Geological Survey.
• Imagery spans the period February 2015 to September 2021.
• Labels were created by Daniel Buscombe, Marda Science, using the labeling program Doodler.

Download this file and unzip to somewhere on your machine (although not inside the segmentation_gym folder), then see the relevant page on the segmentation gym wiki for further explanation.

This dataset and associated models were made by Dr Daniel Buscombe, Marda Science LLC, for the purposes of demonstrating the functionality of Segmentation Gym. The labels were created using Doodler.

Previous versions:

1.0. https://zenodo.org/record/5895128#.Y1G5s3bMIuU original release, Oct 2021, conforming to Segmentation Gym functionality on Oct 2021

2.0 https://zenodo.org/record/7036025#.Y1G57XbMIuU, Jan 23 2022, conforming to Segmentation Gym functionality on Jan 23 2022

This is version 4.0, created 2/25/23, and has been tested with Segmentation Gym using doodleverse-utils 0.0.26 https://pypi.org/project/doodleverse-utils/0.0.26/

file structure

/Users/Someone/my_segmentation_zoo_datasets
          │  ├── config
          │  |  └── *.json
          │  ├── capehatteras_data
          |  |  ├── fromDoodler
          |  |  |   ├──images
          │  |  |   └──labels
          |  |  ├──npzForModel
          │  |  └──toPredict
          │  └── modelOut
          │    └── *.png
          │  └── weights
          │    └── *.h5

config

There are 4 config files: 1. /config/hatteras_l8_resunet.json 2. /config/hatteras_l8_vanilla_unet.json 3. /config/hatteras_l8_resunet_model2.json

1. /config/hatteras_l8_segformer.json

The first two are for res-unet and unet models respectively. The third one differs from the first only with specification of kernel size. It is provided as an example of how to conduct model training experiments, modifying one hyperparameter at a time in the effort to create an optimal model. The last one is based on the new Segformer model architecture.

They all contain the same essential information and differ as indicated below

{
 "TARGET_SIZE": [768,768], # the size of the imagery you wish the model to train on. This may not be the original size
 "MODEL": "resunet", # model name. Otherwise, "unet" or "segformer"
 "NCLASSES": 4, # number of classes
 "KERNEL":9, # horizontal size of convolution kernel in pixels
 "STRIDE":2, # stride in convolution kernel
 "BATCH_SIZE": 7, # number of images/labels per batch
 "FILTERS":6, # number of filters
 "N_DATA_BANDS": 3, # number of image bands
 "DROPOUT":0.1, # amount of dropout
 "DROPOUT_CHANGE_PER_LAYER":0.0, # change in dropout per layer
 "DROPOUT_TYPE":"standard", # type of dropout. Otherwise "spatial"
 "USE_DROPOUT_ON_UPSAMPLING":false, # if true, dropout is used on upsampling as well as downsampling
 "DO_TRAIN": false, # if false, the model will not train, but you will select this config file, data directory, and the program will load the model weights and test the model on the validation subset
 if true, the model will train from scratch (warning! this will overwrite the existing weights file in h5 format)
 "LOSS":"dice", # model training loss function, otherwise "cat" for categorical cross-entropy
 "PATIENCE": 10, # number of epochs of no model improvement before training is aborted
 "MAX_EPOCHS": 100, # maximum number of training epochs
 "VALIDATION_SPLIT": 0.6, #proportion to use for validation
 "RAMPUP_EPOCHS": 20, # [LR-scheduler] rampup to maximim
 "SUSTAIN_EPOCHS": 0.0, # [LR-scheduler] sustain at maximum
 "EXP_DECAY": 0.9, # [LR-scheduler] decay rate
 "START_LR": 1e-7, # [LR-scheduler] start lr
 "MIN_LR": 1e-7, # [LR-scheduler] min lr
 "MAX_LR": 1e-4, # [LR-scheduler] max lr
 "FILTER_VALUE": 0, #if >0, the size of a median filter to apply on outputs (not recommended unless you have noisy outputs)
 "DOPLOT": true, #make plots
 "ROOT_STRING": "hatteras_l8_aug_768", #data file (npz) prefix string
 "USEMASK": false, # use the convention 'mask' in label image file names, instead of the preferred 'label'
 "AUG_ROT": 5, # [augmentation] amount of rotation in degrees
 "AUG_ZOOM": 0.05, # [augmentation] amount of zoom as a proportion
 "AUG_WIDTHSHIFT": 0.05, # [augmentation] amount of random width shift as a proportion
 "AUG_HEIGHTSHIFT": 0.05,# [augmentation] amount of random width shift as a proportion
 "AUG_HFLIP": true, # [augmentation] if true, randomly apply horizontal flips
 "AUG_VFLIP": false, # [augmentation] if true, randomly apply vertical flips
 "AUG_LOOPS": 10, #[augmentation] number of portions to split the data into (recommended > 2 to save memory)
 "AUG_COPIES": 5 #[augmentation] number iof augmented copies to make
 "SET_GPU": "0" #which GPU to use. If multiple, list separated by a comma, e.g. '0,1,2'. If CPU is requested, use "-1"
 "WRITE_MODELMETADATA": false, #if true, the prompts `seg_images_in_folder.py` to write detailed metadata for each sample file
 "DO_CRF": true #if true, apply CRF post-processing to outputs


 "LOSS_WEIGHTS": false, #if true, apply per-class weights to loss function


 "MODE": "all", #'all' means use both non-augmented and augmented files, "noaug" means use non-augmented only, "aug" uses augmented only


 "SET_PCI_BUS_ID": true, #if true, make keras aware of the PCI BUS ID (advanced or nonstandard GPU usage)


 "TESTTIMEAUG": true, #if true, apply test-time augmentation when model in inference mode


 "WRITE_MODELMETADATA": true,# if true, write model metadata per image when model in inference mode


 "OTSU_THRESHOLD": true# if true, and NCLASSES=2 only, use per-image Otsu threshold rather than decision boundary of 0.5 on softmax scores


}

capehatteras_data

Folder containing all the model input data

          │  ├── capehatteras_data: folder containing all the model input data
          |  |  ├── fromDoodler: folder containing images and labels exported from Doodler using [this program](https://github.com/dbuscombe-usgs/dash_doodler/blob/main/utils/gen_images_and_labels_4_zoo.py)
          |  |  |   ├──images: jpg format files, one per label image
          │  |  |   └──labels: jpg format files, one per image
          |  |  ├──npzForModel: npz format files for model training using [this program](https://github.com/dbuscombe-usgs/segmentation_zoo/blob/main/train_model.py) that have been created following the workflow [documented here](https://github.com/dbuscombe-usgs/segmentation_zoo/wiki/Create-a-model-ready-dataset) using [this program](https://github.com/dbuscombe-usgs/segmentation_zoo/blob/main/make_nd_dataset.py)
          │  |  └──toPredict: a folder of images to test model prediction using [this program](https://github.com/dbuscombe-usgs/segmentation_zoo/blob/main/seg_images_in_folder.py)

modelOut

PNG format files containing example model outputs from the train ('_train_' in filename) and validation ('_val_' in filename) subsets as well as an image showing training loss and accuracy curves with trainhist in the filename. There are two sets of these files, those associated with the residual unet trained with dice loss contain resunet in their name, and those from the UNet are named with vanilla_unet.

weights

There are model weights files associated with each config files.

Overview

The Stereo Instances on Surfaces Dataset (STIOS) is created for evaluation of instance-based algorithms. It is a representative dataset to achieve uniform comparability for instance detection and segmentation with different input modalities (RGB, RGB-D, stereo RGB). STIOS is mainly intended for robotic applications (e.g. object manipulation), which is why the dataset refers to horizontal surfaces.

Sensors

STIOS contains recordings from two different sensors: a rc_visard 65 color and a Stereolabs ZED camera. Aside stereo RGB (left and right RGB image), the internally generated depth maps are also saved for both sensors. In addition, the ZED sensor provides normal images and point cloud data which are also provided in STIOS. Since some objects / surfaces have little texture and this would have a negative impact on the quality of the depth map, an additional LED projector with a random point pattern is used when recording the depth images (only used for rc_visard 65 color). Consequently, for the rc_visard 65 color STIOS includes RGB images and the resulting depth maps with and without a projected pattern.

The large number of different input modalities should enable evaluation of a wide variety of methods. As you can see in the picture, the ZED sensor was mounted above the rc_visard 65 lenses to get a similar viewing angle. This enables an evaluation between the sensors, whereby comparisons can be made about the generalization of a method with regard to sensors or the quality of the input modality.

Objects

The dataset contains the following objects from the YCB video dataset and thus covers several application areas such as unknown instance segmentation, instance detection and segmentation (detection + classification):

003_cracker_box, 005_tomato_soup_can, 006_mustard_bottle, 007_tuna_fish_can, 008_pudding_box, 010_potted_meat_can, 011_banana, 019_pitcher_base, 021_bleach_cleanser, 024_bowl, 025_mug, 035_power_drill, 037_scissors, 052_extra_large_clamp, 061_foam_brick.

Due to the widespread use of these objects in robotic applications there are 3D models for each of the objects which can be used to generate synthetic training data for e.g. instance detection based on RGB-D. In order to guarantee an evenly distributed occurrence of the 15 objects, 4-6 random objects are selected by machine for each sample. The alignment of the objects is either easy (objects do not touch) or difficult (objects may touch or lie on top of each other).

Surroundings

The data set contains 8 different environments in order to cover the variation of environmental parameters such as lighting, background or scene surfaces. Scenes for the data set were recorded in the following environments: office carpet, workbench, white table, wooden table, conveyor belt, lab floor, wooden plank und tool cabinet.

The scenes where chosen carefully to ensure that they contain surfaces that are both friendly as well as challenging to stereo sensors. STIOS therefore contains low-texture surfaces (e.g. white table, conveyor belt) and texture-rich surfaces (e.g. lab floor, wooden plank). The above-mentioned variations of the surfaces and environments allows to evaluate methods in terms of robustness against and generalization to various environmental parameters.

For each scene surface, 3 easy and 3 difficult samples are generated from 4 manually set camera angles (approx. 0.3-1m distance). As the illustration shows, even with light object alignment the objects can occlude each other in some camera angles. The 6 samples per camera setting result in 24 samples per environment for each sensor, which results in a total of 192 samples per sensor.

Annotations

For each of these samples (192x2) all object instances in the left camera image were annotated manually (instance mask + object class). The annotations are available in the form of 8-bit grayscale images, which represent the semantic classes in the image. Since each object appears only once in the image, object instance masks can also be obtained from this format at the same time.

The dataset is structured as follows:

STIOS
|--rc_visard
| |--conveyor_belt
| | |--left_rgb
| | |--right_rgb
| | |--gt
| | |--depth
| | |--left_rgb_pattern
| | |--right_rgb_pattern
| | |--depth_pattern
| |--lab_floor
| |-- ...
|--zed
| |-- conveyor_belt
| | |--left_rgb
| | |--right_rgb
| | |--gt
| | |--depth
| | |--normals
| | |--pcd
| |--lab_floor
| |--...

We also provide code utilities which allow visualization of images and annotations of STIOS and contain various utility functions to e.g. generate bounding box annotations from the semantic grayscale images. Please find them here: https://github.com/DLR-RM/stios-utils.

Citation

If STIOS is useful for your research please cite

@misc{durner2021unknown,
  title={Unknown Object Segmentation from Stereo Images}, 
  author={Maximilian Durner and Wout Boerdijk and Martin Sundermeyer and Werner Friedl and Zoltan-Csaba Marton and Rudolph Triebel},
  year={2021},
  eprint={2103.06796},
  archivePrefix={arXiv},
  primaryClass={cs.CV}
}

STIOS in projects

Unknown Object Segmentation from Stereo Images
M. Durner, W. Boerdijk, M. Sundermeyer, W. Friedl, Z.-C. Marton, and R. Triebel. "Unknown Object Segmentation from Stereo Images", arXiv preprint arXiv:2103.06796 (2021).

This method enables the segmentation of unknown object instances that are located on horizontal surfaces (e.g. tables, floors, etc.). Due to the often incomplete depth data in robotic applications, stereo RGB images are used here. On the one hand, STIOS is employed to show the functionality of stereo images for unknown instance segmentation, and on the other hand, to make a comparison with existing work, which for the most part directly access depth data.

"What's This?" - Learning to Segment Unknown Objects from Manipulation Sequences
W. Boerdijk, M. Sundermeyer, M. Durner, and R. Triebel. "'What's This?' - Learning to Segment Unknown Objects from Manipulation Sequences", International Conference on Robotics and Automation (ICRA), 2021 (to appear).

This work deals with the segmentation of objects that have been grasped by a robotic arm. With the help of this method it is possible to generate object-specific image data in an automated process. This data can then be used for training object detectors or segmentation approaches. In order to show the usability of the generated data, STIOS is used as an evaluation data set for instance segmentation on RGB images.

Doodleverse/Segmentation Zoo Res-UNet models for Aerial/NOAA ERI/2-class (water, nowater) segmentation of RGB 1024x768 high-res. images

Residual-UNet models are trained on 1,179 pairs of human-generated segmentation labels and images from Emergency Response Imagery (ERI) collected by US National Oceanic and Atmospheric Administration (NOAA) after Hurricane Barry, Delta, Dorian, Florence, Ida, Laura, Michael, Sally, Zeta, and Tropical Storm Gordon.

The dataset is available here**: https://doi.org/10.5281/zenodo.7268082

Models have been created using Segmentation Gym*:

Code - https://github.com/Doodleverse/segmentation_gym

Paper - https://doi.org/10.1029/2022EA002332

The model takes input images that are 512 x 512 x 3 pixels, and the output is 512 x 512 x 2, corresponding to 2 classes:

1. water
2. other

Included here are 6 files with the same root name:

'.json' config file: this is the file that was used by Segmentation Gym to create the weights file. It contains instructions for how to make the model and the data it used, as well as instructions for how to use the model for prediction.
'.h5' weights file: this is the file that was created by the Segmentation Gym function `train_model.py`. It contains the trained model's parameter weights. It can called by the Segmentation Gym function `seg_images_in_folder.py`.
'_model_history.npz' model training history file: this numpy archive file contains numpy arrays describing the training and validation losses and metrics. It is created by the Segmentation Gym function `train_model.py`
'.png' model training loss and mean IoU plot: this png file contains plots of training and validation losses and mean IoU scores during model training. A subset of data inside the .npz file. It is created by the Segmentation Gym function `train_model.py`
'.zip' of the model in the Tensorflow ‘saved model’ format. It is created by the Segmentation Gym function `utils/gen_saved_model.py`
'_modelcard.json' model card file: this is a json file containing fields that collectively describe the model origins, training choices, and dataset that the model is based upon. There is some redundancy between this file and the `config` file (described above) that contains the instructions for the model training and implementation. The model card file is not used by the program but is important metadata so it is important to keep with the other files that collectively make the model and is such is considered part of the model

Additionally, BEST_MODEL.txt contains the name of the model with the best validation loss and mean IoU

References

*Segmentation Gym: Buscombe, D., & Goldstein, E. B. (2022). A reproducible and reusable pipeline for segmentation of geoscientific imagery. Earth and Space Science, 9, e2022EA002332. https://doi.org/10.1029/2022EA002332 See: https://github.com/Doodleverse/segmentation_gym

** Goldstein, Evan B., Buscombe, Daniel, Budavi, Priyanka, Favela, Jaycee, Fitzpatrick, Sharon, Gabbula, Sai Ram Ajay Krishna, Ku, Venus, Lazarus, Eli D., McCune, Ryan, Shah, Manish, Sigdel, Rajesh, & Tagner, Steven. (2022). Segmentation Labels for Emergency Response Imagery from Hurricane Barry, Delta, Dorian, Florence, Isaias, Laura, Michael, Sally, Zeta, and Tropical Storm Gordon (Version v1) [Data set]. Zenodo. https://doi.org/10.5281/zenodo.7268083

The PENGWIN segmentation challenge is designed to advance the development of automated pelvic fracture segmentation techniques in both 3D CT scans (Task 1) and 2D X-ray images (Task 2), aiming to enhance their accuarcy and robustness. The full 3D dataset comprises CT scans from 150 patients scheduled for pelvic reduction surgery, collected from multiple institutions using a variety of scanning devices. This dataset represents a diverse range of patient cohorts and fracture types. Ground-truth segmentations for sacrum and hipbone fragments have been semi-automatically annotated and subsequently validated by medical experts, and are available here. From this 3D data, we have generated high-quality, realistic X-ray images and corresponding 2D labels from the CT data using DeepDRR, incorporating a range of virtual C-arm camera positions and surgical tools. This dataset contains the training set for fragment segmentation in synthetic X-ray (task 2).

The training set is derived from 100 CTs, with 500 images each, for a total of 50,000 training images and segmentations. The C-arm geometry is randomly sampled for each CT within reasonable parameters for a full-size C-arm. The virtual patient is assumed to be in a head-first supine position. Imaging centers are randomly sampled within 50 mm of a fragment, ensuring good visibility. Viewing directions are sampled uniformly on the sphere within 45 degrees of vertical. Half of the images (IDs XXX_0250 - XXX_0500) contain up to 10 simulated K-wires and/or orthopaedic screws oriented randomly in the field of view.

The input images are raw intensity images without any windowing or normalization applied. It is standard practice to first apply the negative log transformation and then window each image appropriately for feeding into a model. See the included augmentation pipeline in pengwin_utils.py for one approach. For viewing raw images, the FIJI image viewer is a viable option, but it is recommended to use the included visualization functions in pengwin_utilities.py to first apply CLAHE normalization and save to a universally readable PNG (see example usage below).

Because X-ray images feature overlapping segmentation maks, the segmentations have been encoded as multi-label uint32 images, where each pixel should be treated as a binary vector with bits 1 - 10 for SA fragments, 11 - 20 for LI, and 21 - 30 for RI. Thus, the raw segmentation files are not viewable with standard image viewing software. pengwin_utilities.py includes functions for converting to and from this format and for visualizing masks overlaid onto the original image (see below).

To use the utilities, first install dependencies with pip install -r requirement.txt. Then, to visualize an image with its segmentation, you can do the following (assuming the training set has been downloaded and unzipped in the same folder):

import pengwin_utils from PIL import Image

image_path = "train/input/images/x-ray/001_0000.tif" seg_path = "train/output/images/x-ray/001_0000.tif"

load image and masks

image = pengwin_utils.load_image(image_path) # raw intensity image masks, category_ids, fragment_ids = pengwin_utils.load_masks(seg_path)

save visualization of image and masks

applies CLAHE normalization to the raw intensity image before overlaying segmentations.

vis_image = pengwin_utils.visualize_sample(image, masks, category_ids, fragment_ids) vis_path = "vis_image.png" Image.fromarray(vis_image).save(vis_path) print(f"Wrote visualization to {vis_path}")

Obtain predicted masks, category_ids, and fragment_ids

Category IDs are

Fragment IDs are the integer labels from label_{category}.nii.gz, with 1 corresponding to the main fragment.

pred_masks, pred_category_ids, pred_fragment_ids = masks, category_ids, fragment_ids # replace with your model

save the predicted masks for upload to the challenge

Note: cv2 does not work with uint32 images. It is recommended to use PIL or imageio.v3

pred_seg = pengwin_utils.masks_to_seg(pred_masks, pred_category_ids, pred_fragment_ids) pred_seg_path = "pred/train/output/images/x-ray/001_0000.tif" # ensure dir exists! Image.fromarray(pred_seg).save(pred_seg_path) print(f"Wrote segmentation to {pred_seg_path}")

The pengwin_utils.Dataset class is provided as an example of a Pytorch dataset, with strong domain randomization included to facilitate sim-to-real performance, but it is recommended to write your own as needed.

This record contains code and data for segmentation using a three-dimensional level-set method, written by Amy Tabb in C++. The record also contains two datasets of root systems in media imaged with X-Ray CT, and the results of running the code on those datasets. The code will also perform a pre-processing task in three-dimensional image sets, and a dataset for that purpose is included as well. This work is a companion to the paper : "Segmenting root systems in X-ray computed tomography images using level sets" (WACV 2018) by the authors or this record, and and open-access version of the paper is here -- https://arxiv.org/abs/1809.06398 . The code is also available from Github: https://github.com/amy-tabb/tabb-level-set-segmentation , using a DOI and stable releases https://doi.org/10.5281/zenodo.3344906.

Format of the data:

Three input datasets are provided; two for the segmentation functionality of the code, and one to test the pre-processing functionality. The two segmentation sets are the same as were used in the paper, and are CassavaDataset, and SoybeanDataset. The pre-processing set is CassavaSlices. The output set for Soybean is SoybeanResultsJul11. The Cassava result set is large, so I broke it into three compressed folders, CassavaResultsJul12_A, _B, _C. _B is the largest, and only contains the results overwritten on the original X-Ray images. Unless your connection to Zenodo is extremely fast, it will be faster to compute the result than to download it.

The goal of this project is to provide all the materials to the community to resolve the problem of echocardiographic image segmentation and volume estimation from 2D ultrasound sequences (both two and four-chamber views). To this aim, the following solutions were set-up introduction of the largest publicly-available and fully-annotated dataset for 2D echocardiographic assessment (to our knowledge). The CAMUS dataset, containing 2D apical four-chamber and two-chamber view sequences acquired from 500 patients, is made available for download.

Dataset Properties

The overall CAMUS dataset consists of clinical exams from 500 patients, acquired at the University Hospital of St Etienne (France) and included in this study within the regulation set by the local ethical committee of the hospital after full anonymization. The acquisitions were optimized to perform left ventricle ejection fraction measurements. In order to enforce clinical realism, neither prerequisite nor data selection have been performed. Consequently,

• some cases were difficult to trace;

• the dataset involves a wide variability of acquisition settings;

• for some patients, parts of the wall were not visible in the images;

for some cases, the probe orientation recommendation to acquire a rigorous four-chambers view was simply impossible to follow and a five-chambers view was acquired instead. This produced a highly heterogeneous dataset, both in terms of image quality and pathological cases, which is typical of daily clinical practice data.

The dataset has been made available to the community HERE. The dataset comprises : i) a training set of 450 patients along with the corresponding manual references based on the analysis of one clinical expert; ii) a testing set composed of 50 new patients. The raw input images are provided through the raw/mhd file format.

Study population

Half of the dataset population has a left ventricle ejection fraction lower than 45%, thus being considered at pathological risk (beyond the uncertainty of the measurement). Also, 19% of the images have a poor quality (based on the opinion of one expert), indicating that for this subgroup the localization of the left ventricle endocarium and left ventricle epicardium as well as the estimation of clinical indices are not considered clinically accurate and workable. In classical analysis, poor quality images are usually removed from the dataset because of their clinical uselessness. Therefore, those data were not involved in this project during the computation of the different metrics but were used to study their influence as part of the training and validation sets for deep learning techniques.

Involved systems

The full dataset was acquired from GE Vivid E95 ultrasound scanners (GE Vingmed Ultrasound, Horten Norway), with a GE M5S probe (GE Healthcare, US). No additional protocol than the one used in clinical routine was put in place. For each patient, 2D apical four-chamber and two-chamber view sequences were exported from EchoPAC analysis software (GE Vingmed Ultrasound, Horten, Norway). These standard cardiac views were chosen for this study to enable the estimation of left ventricle ejection fraction values based on the Simpson’s biplane method of discs. Each exported sequence corresponds to a set of B-mode images expressed in polar coordinates. The same interpolation procedure was used to express all sequences in Cartesian coordinates with a unique grid resolution, i.e. λ/2 = 0.3 mm along the x-axis (axis parallel to the probe) and λ/4 = 0.15 mm along the z-axis (axis perpendicular to the probe), where λ corresponds to the wavelength of the ultrasound probe. At least one full cardiac cycle was acquired for each patient in each view, allowing manual annotation of cardiac structures at ED and ES.

****This work has published to IEEE TMI journal. You must cite this paper for any use of the CAMUS database.**** - S. Leclerc, E. Smistad, J. Pedrosa, A. Ostvik, et al. "Deep Learning for Segmentation using an Open Large-Scale Dataset in 2D Echocardiography" in IEEE Transactions on Medical Imaging, vol. 38, no. 9, pp. 2198-2210, Sept. 2019. doi: 10.1109/TMI.2019.2900516

Please note: This is a large data product with 2.7 million polygon features (1.2GB file in ESRI File Geodatabase format). It is not possible to download in Shapefile format. Please access the data using the APIs or select another download format.This is the spatial framework around which the Living England Phase II habitat classification is based. The segmentation was created in the Trimble eCognition software using Sentinel-2 Analysis Ready Data (ARD) image mosaics for winter (February 2019) and summer (June 2019).

Sentinel-2 Analysis Ready Data (ARD) produced by the Earth Observation Data Service (JNCC / DEFRA) were used as the input for the segmentation. The Sentinel-2 ARD is available under an Open Government License (OGL). It is not intended that the 2019 segmentation will be revised, however, as Living England progresses and up-to-date image mosaics are created new habitat segmentation datasets will be developed from the up-to-date imagery.Full metadata can be viewed on data.gov.uk.

Introduction

The Surgical Planning Laboratory (SPL) and the National Center for Image Guided Therapy (NCIGT) are making this dataset available as a resource to aid in the development of algorithms and tools for deformable registration, segmentation and analysis of prostate magnetic resonance imaging (MRI) and ultrasound (US) images.

Description

This dataset contains anonymized images of the human prostate (N=3 patients) collected during two sessions for each patient:

1. MRI examination of the prostate for the purposes of disease staging.
2. US examination of the prostate for the purposes of volumetric examination in preparation to the brachytherapy implant.

These are three-dimensional (multi-slice) scalar images.

Image files are stored using NRRD file format (files with .nrrd extension), see details at http://teem.sourceforge.net/nrrd/format.html. Each image file includes a code for the case number (internal numbering at the research site) and the modality (US or MR).

Image annotations were prepared by Dr. Fedorov (no professional training in radiology) and Dr. Tuncali (10+ in prostate imaging interpretation). Annotations include

1. Manual contouring (segmentation) of the whole prostate gland, performed in 3D Slicer software. These segmentation images are coded in the same fashion as the image files, and saved in NRRD format, with "-label" suffix.
2. Manually placed points (fiducials) corresponding to the location of urethra entry into the prostate at base (coded as UB), verumontanum (VM), urethra entry into the prostate at apex (UA), as well as centroids of cysts and calcifications. UB, UA and VM locations are annotated both in MR and US for all cases, while cysts and calcifications are annotated when applicable. Fiducial points are stored in comma-separated CSV-style format adopted by 3D Slicer software (.fcsv file extension). There is one row per point in these files, encoding the location of the point in RAS coordinate space relative to the image data, and the name of the point.

Viewing the collection

We tested visualization of images, segmentations and fiducials in 3D Slicer software, and thus recommend 3D Slicer as the platform for visualization. 3D Slicer is a free open source platform (see http://slicer.org), with the pre-compiled binaries available for all major operating systems. You can download 3D Slicer at http://download.slicer.org.

Acknowledgments

Preparation of this data collection was made possible thanks to the funding from the National Institutes of Health (NIH) through grants R01 CA111288 and P41 RR019703.

If you use this dataset in a publication, please cite the following manuscript. You can also learn more about this dataset from the publication below.

Fedorov, A., Khallaghi, S., Antonio Sánchez, C., Lasso, A., Fels, S., Tuncali, K., Sugar, E. N., Kapur, T., Zhang, C., Wells, W., Nguyen, P. L., Abolmaesumi, P. & Tempany, C. Open-source image registration for MRI–TRUS fusion-guided prostate interventions. Int J CARS 10, 925–934 (2015). https://pubmed.ncbi.nlm.nih.gov/25847666/

Contact

Andrey Fedorov, fedorov@bwh.harvard.edu

This dataset is a derivative of the original French speaker database, designed for training and testing deep learning models for upper airway segmentation. It includes upper airway MRI scans and their corresponding expert-crafted annotations, specifically created for the upper airway segmentation project. The dataset also features synthetic sentences to ensure adequate coverage of the French phonetic context. Each 3D scan, acquired using a Siemens Prisma 3T scanner with a VIBE sequence (TR = 3.8 ms, TE = 1.55 ms, FOV = 22 × 22 cm², slice thickness = 1.2 mm, and image size = 320 × 290 × 36 slices), has a duration of 7 seconds.The segmentations were manually created in a two-stage process by an annotator team with expertise in voice-science. First, the volumes were distributed to two senior graduate students in voice science who performed the segmentations. Next, the segmentations were further refined by an expert vocologist with more than 20 years experience in voice pedagogy. All processing was done in the Slicer environment. Subsequently, the segmentations and the original MRI volumes underwent conversion into NRRD format. For training, we have used a subset of 45 volumes across 7 subjects from these 53 volumes.For the test set, assessing the performance of an algorithm using annotated label maps from a single human expert as a reference poses challenges in evaluation due to potential bias introduced by the human. To mitigate this challenge, we employed the STAPLE algorithm. This algorithm considers a collection of segmentations from multiple human annotators, and computes a probabilistic estimate of the true segmentation. In creating the test set, we chose 8 volumes from three subjects (non-overlapping with the training set), and collected manual segmentations from three different annotators. The first two annotators were senior graduate students with expertise in image processing and biomedical engineering, and segmentations from a third annotator were those provided by the voice-science team as described above. These segmentations were then input into the STAPLE algorithm to generate an optimal segmentation. The STAPLE outputs are provided in the test set.Dataset StructureTraining:RTrainVolumes: 3D upper airway MRI volumes.RtrainLabels: Corresponding segmentation annotations.Testing:RVolumes: 3D upper airway MRI volumes.RLabels: Corresponding segmentation annotations.CitationIf you use this dataset, please cite the following papers:Subin Erattakulangara, Karthika Kelat, Katie Burnham, Rachel Balbi, Sarah E. Gerard, David Meyer, Sajan Goud Lingala,Open-Source Manually Annotated Vocal Tract Database for Automatic Segmentation from 3D MRI Using Deep Learning: Benchmarking 2D and 3D Convolutional and Transformer Networks, Journal of Voice, 2025, ISSN 0892-1997, https://doi.org/10.1016/j.jvoice.2025.02.026.Isaieva, Karyna, et al. "Multimodal dataset of real-time 2D and static 3D MRI of healthy French speakers." Scientific Data, 8.1 (2021): 258.Erattakulangara, Subin, et al. "Automatic multiple articulator segmentation in dynamic speech MRI using a protocol adaptive stacked transfer learning u-net model." Bioengineering 10.5 (2023): 623.

Overview

• Introduction
• Object Classes
  • Document Macro Structure
  • Digitization Artefact Zone
  • Graphic Zone
  • Graphic Zone - Decoration
  • Graphic Zone - Fig Desc
  • Graphic Zone - Head
  • Main Zone Classes
  • Margin Text Zone Classes
  • Numbering Zone
  • Quire Mark Zone
  • Running Title Zone
  • Stamp Zone
  • Table Zone
  • Title Page Zone

Introduction

The dataset aims to annotate various elements found in digitized historical documents to comprehend document structure and layout. The classes include text zones, graphic elements, numberings, decorative elements, and more, to encapsulate the layout.

Object Classes

Document Macro Structure

Description

Represents the overall layout structure of a document.

Instructions

Annotate the visible edges of the document including all elements that contribute to the structural layout. Do not capture decorative or specific text areas under this class.

Digitization Artefact Zone

Description

Zones indicating artefacts from the digitization process, such as shadows or scanner marks.

Instructions

Mark any unintended markings or shadows that are a result of scanning. Avoid annotating intended text or graphic content.

Graphic Zone

Description

Areas containing non-text graphics, like images or illustrations.

Instructions

Outline the complete area of the image or illustration, including associated captions if directly attached. Do not separate text unless specifically part of the graphic.

Graphic Zone - Decoration

Description

Decorative elements that embellish text or page borders.

Instructions

Enclose artwork such as decorative borders or flourishes that enhance the page layout. Exclude surrounding text.

Graphic Zone - Fig Desc

Description

Descriptions that accompany figures or graphics.

Instructions

Label the area containing text that describes graphics. Ensure not to capture the graphic itself as part of this annotation.

Graphic Zone - Head

Description

Headings associated with graphics, usually found above or beside them.

Instructions

Identify headings directly related to graphics, encircling the text without extending to main text areas.

Main Zone Classes

Description

Text that forms the primary content of the document.

Instructions for each Sub-Class

• Entry: Begin at the entry text and extend across all central main content lines.
• Entry - Continued: Mark text listed as a continuation of an entry.
• Form: Include structured segments formatted uniquely, like forms or tables.
• Head: Denote the main title of sections.
• List: List all enumerated items.
• Other: Any main content not covered by specific sub-classes.
• P: Paragraphs forming the bulk of text.
• P - CatalogueDesc: Catalogue descriptions included in main text.
• P - Continued: Extended portions of paragraph texts.
• Signature: Identify signed areas, usually at the document's end.

Ensure clarity between different sub-zones, and focus on text alignment and indentation cues.

Margin Text Zone Classes

Description

Notes or additional comments located in the margins.

Instructions for each Sub-Class

• Manuscript Addendum: Side notes or hand-written additions in the margin.
• Note: Standalone commentary or remarks.
• Note - Continued: Extensions of margin notes.

Prioritize notable deviations in alignment or format from the main text.

Numbering Zone

Description

Numerical identifiers, such as page numbers.

Instructions

Spot all numerical indicators located in header or footer regions, ensuring no overlap with other annotations.

Quire Mark Zone

Description

Marks showing quire or gathering information.

Instructions

Encircle symbols or shorthand describing document assembly, ignoring main or marginal text.

Running Title Zone

Description

Titles or headings that repeat on multiple pages.

Instructions

Cover repeated headers or titles at the top edge of pages, avoiding interaction with the main text body.

Stamp Zone

Description

Stamped markings on the documents.

Instructions

Identify all institutional or approval stamps, along with philatelic elements. Ensure clear separation from text and images.

Table Zone

Description

Zones containing tabular data.

Instructions

Delineate the boundaries of grids or tables including headings if within the zone. Avoid extending beyond table borders.

Title Page Zone

Description

Front page of the d

Data Citation

Please cite this dataset as follows:

Vasquez, V., Cushman, K., Ramos, P., Williamson, C., Villareal, P., Gomez Correa, L. F., & Muller-Landau, H. (2023). Barro Colorado Island 50-ha plot crown maps: manually segmented and instance segmented. (Version 2). Smithsonian Tropical Research Institute. https://doi.org/10.25573/data.24784053

This data is licensed as CC BY 4.0 and is thus freely available for reuse with proper citation. We ask that data users share any resulting publications, preprints, associated analysis code, and derived data products with us by emailing mullerh@si.edu. We are open to contributing our expert knowledge of the study site and datasets to projects that use these data; please direct queries regarding potential collaboration to Vicente Vasquez, vasquezv@si.edu, and Helene Muller-Landau, mullerh@si.edu.

Note that this dataset is part of a collection of Panama UAV data on Smithsonian Figshare, which can be viewed at https://smithsonian.figshare.com/projects/Panama_Forest_Landscapes_UAV/115572

Additional information about this research can be found at the Muller-Landau lab web site at https://hmullerlandau.com/

All required code is freely available at https://github.com/P-polycephalum/ForestLandscapes/blob/main/LandscapeScripts/segmentation.py and it can be cited as:

Vicente Vasquez. (2023). P-polycephalum/ForestLandscapes: segmentwise (v0.0.2-beta). Zenodo. https://doi.org/10.5281/zenodo.10380517

Data Description

This dataset is part of a larger initiative monitoring forests in Panama using drones (unoccupied aerial vehicles), an initiative led by Dr. Helene Muller- Landau at the Smithsonian Tropical Research Institute. As part of this initiative, we have been collecting repeat imagery of the 50-ha forest dynamics plot on Barro Colorado Island (BCI), Panama, since October 2014 (see Garcia et al. 2021a, b for data products for 2014-2019).

Contained within this dataset are two sets of field-derived crown maps, presented in both their raw and improved versions. The 2021 crown mapping campaign was overseen by KC Cushman, accompanied by field technician Pablo Ramos and Paulino Villarreal. Additionally, Cecilia Williamson and KC Cushman reviewed polygon quality and made necessary corrections. Image data occurred on August 1, 2020, utilizing a DJI Phantom 4 Pro at a resolution of 4cm per pixel. A total of 2454 polygons were manually delineated, encompassing insightful metrics like crown completeness and liana load.

The 2023 crown mapping campaign, led by Vicente Vasquez and field technicians Pablo Ramos, Paulino Villarreal, involved quality revisions and corrections performed by Luisa Fernanda Gomez Correa and Vicente Vasquez. Image data collection occurred on September 29, 2022, utilizing a DJI Phantom 4 Pro drone at a 4cm per pixel resolution. The 2023 campaign integrated model 230103_randresize_full of the detectree2 model garden (Ball, 2023). Tree crown polygons were generated pre-field visit, with those attaining a field validation score of 7 or higher retained as true tree crowns.

The data collection forms are prepared using ArcGIS field maps. The creator of the data forms uses the spatial points from the trees in the ForestGeo 50-ha censuses to facilitate finding the tree tags in the field (Condit et al., 2019). The field technicians confirm that the tree crown is visible from the drone imagery, they proceed to collect variables of interest and delineate the tree crown manually. In the case of the 2023 field campaign, the field technicians were able to skip manual delineation when the polygons generated by 230103_randresize_full were evaluated as true detection.

The improved version of the 2023 and 2021 crown map data collection takes as input the raw crown maps and the globally aligned orthomosaics to refine the edges of the crown. We use the model SAM from segment-anything module developed my Meta AI (Krillov, 2023). We adapted the use of their instance segmentation algorithm to take geospatial imagery in the form of tiles. We inputted multiple bounding boxes in the form of CPU torch tensors for each of the files. Furthermore, we perform several tasks to clean the crowns and remove the polygons overlaps to avoid ambiguity. This results in a very well delineated crown map with no overlapping between tree crowns. Despite our diligent efforts in detecting, delineating, and evaluating all visible tree crowns from drone imagery, this dataset exhibits certain limitations. These include missing tags denoted as -9999, erroneous manual delineations or instance segmentation of tree crown polygons, duplicated tags, and undetected tree crowns. These limitations are primarily attributed to human error, logistical constraints, and the challenge of confirming individual tree crown emergence above the canopy. In numerous instances, particularly within densel... Visit https://dataone.org/datasets/doi%3A10.60635%2FC3PP4J for complete metadata about this dataset.

The goal of the Automated Cardiac Diagnosis Challenge (ACDC) challenge is to:

compare the performance of automatic methods on the segmentation of the left ventricular endocardium and epicardium as the right ventricular endocardium for both end diastolic and end systolic phase instances; compare the performance of automatic methods for the classification of the examinations in five classes (normal case, heart failure with infarction, dilated cardiomyopathy, hypertrophic cardiomyopathy, abnormal right ventricle).

The overall ACDC dataset was created from real clinical exams acquired at the University Hospital of Dijon. Acquired data were fully anonymized and handled within the regulations set by the local ethical committee of the Hospital of Dijon (France). Our dataset covers several well-defined pathologies with enough cases to (1) properly train machine learning methods and (2) clearly assess the variations of the main physiological parameters obtained from cine-MRI (in particular diastolic volume and ejection fraction). The dataset is composed of 150 exams (all from different patients) divided into 5 evenly distributed subgroups (4 pathological plus 1 healthy subject groups) as described below. Furthermore, each patient comes with the following additional information : weight, height, as well as the diastolic and systolic phase instants.

The database is made available to participants through two datasets from the dedicated online evaluation website after a personal registration: i) a training dataset of 100 patients along with the corresponding manual references based on the analysis of one clinical expert; ii) a testing dataset composed of 50 new patients, without manual annotations but with the patient information given above. The raw input images are provided through the Nifti format.

MVTec AD is a dataset for benchmarking anomaly detection methods with a focus on industrial inspection. It contains over 5000 high-resolution images divided into fifteen different object and texture categories. Each category comprises a set of defect-free training images and a test set of images with various kinds of defects as well as images without defects.

There are two common metrics: Detection AUROC and Segmentation (or pixelwise) AUROC

Detection (or, classification) methods output single float (anomaly score) per input test image.

Segmentation methods output anomaly probability for each pixel. "To assess segmentation performance, we evaluate the relative per-region overlap of the segmentation with the ground truth. To get an additional performance measure that is independent of the determined threshold, we compute the area under the receiver operating characteristic curve (ROC AUC). We define the true positive rate as the percentage of pixels that were correctly classified as anomalous" [1] Later segmentation metric was improved to balance regions with small and large area, see PRO-AUC and other in [2]

[1] Paul Bergmann et al, "MVTec AD — A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection" [2] Bergmann, P., Batzner, K., Fauser, M. et al. The MVTec Anomaly Detection Dataset: A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection. Int J Comput Vis (2021). https://doi.org/10.1007/s11263-020-01400-4

Focused ion beam (FIB) tomography is a destructive technique used to collect three-dimensional (3D) structural information at a resolution of a few nanometers. For FIB tomography, a material sample is degraded by layer-wise milling. After each layer, the current surface is imaged by a scanning electron microscope (SEM), providing a consecutive series of cross-sections of the three-dimensional material sample. Especially for nanoporous materials, the reconstruction of the 3D microstructure of the material, from the information collected during FIB tomography, is impaired by the so-called shine-through effect. This effect prevents a unique mapping between voxel intensity values and material phase (e.g., solid or void). It often substantially reduces the accuracy of conventional methods for image segmentation. Here we demonstrate how machine learning can be used to tackle this problem. A bottleneck in doing so is the availability of sufficient training data. To overcome this problem, we present a novel approach to generate synthetic training data in the form of FIB-SEM images generated by Monte Carlo simulations. Based on this approach, we compare the performance of different machine learning architectures for segmenting FIB tomography data of nanoporous materials. We demonstrate that two-dimensional (2D) convolutional neural network (CNN) architectures processing a group of adjacent slices as input data as well as 3D CNN perform best and can enhance the segmentation performance significantly.

Description of the folder content :

1) The macro in .ijm format.
Suited for analysis of 3-channel confocal fluorescence microscopy images of mammalian cells (~200*200µm).
Requires ImageJ v1.4 with Bio-render plugin.
Images should be as .nd2 format but it can easily be changed, simply search & replace all occurences of ".nd2" with your format in the macro code.
Images should be organized with every replicate of a same test-condition in a unique folder. The macro will analyze the whole folder at once and will create a folder in it to save results.

2) A folder named "example_data", it contains 3 representative images that can be used to test the macro.
It also contains a results folder with representative data obtained via the analysis of these representative images with the macro (see Description of the macro for description of the results obtained)

_

Description of the macro :

input : 3-channel image with

C1 = nucleus labeling (e.g. DAPI, Hoechst, etc.)
C2 = signal of interest, the one you want to measure in whole cells & in the region of interest
C3 = region of interest (ROI) (e.g. an antibody directed against a particular organelle, in our case Golgi apparatus)

this macro will :
count the cells according to C1 (user input of threshold values for C1)
create ROI(s) according to C3 (user input of threshold values, or manual setting of each image for C3)
measure signal of C2 (mean min max grey values, integrated density, area) in whole cells (user input of threshold values for C2) measure signal of C2 in ROI(s)
save results as a .csv file

it will also create several .png images for each analyzed one :
C1+nucleusROI (to assess correct cell counting)
C3+ROIC3 (to assess correct creation of ROI(s) from C3 signal)
C2 (glow LUT) + ROIC3 (to assess correct thresholding of C2 signal)
C2+ROIC3
merge C1+C2+C3

This is Part 2/2 of the ActiveHuman dataset! Part 1 can be found here.

Dataset Description

ActiveHuman was generated using Unity's Perception package.

It consists of 175428 RGB images and their semantic segmentation counterparts taken at different environments, lighting conditions, camera distances and angles. In total, the dataset contains images for 8 environments, 33 humans, 4 lighting conditions, 7 camera distances (1m-4m) and 36 camera angles (0-360 at 10-degree intervals).

The dataset does not include images at every single combination of available camera distances and angles, since for some values the camera would collide with another object or go outside the confines of an environment. As a result, some combinations of camera distances and angles do not exist in the dataset.

Alongside each image, 2D Bounding Box, 3D Bounding Box and Keypoint ground truth annotations are also generated via the use of Labelers and are stored as a JSON-based dataset. These Labelers are scripts that are responsible for capturing ground truth annotations for each captured image or frame. Keypoint annotations follow the COCO format defined by the COCO keypoint annotation template offered in the perception package.

Folder configuration

The dataset consists of 3 folders:

• JSON Data: Contains all the generated JSON files.
• RGB Images: Contains the generated RGB images.
• Semantic Segmentation Images: Contains the generated semantic segmentation images.

Essential Terminology

• Annotation: Recorded data describing a single capture.
• Capture: One completed rendering process of a Unity sensor which stored the rendered result to data files (e.g. PNG, JPG, etc.).
• Ego: Object or person on which a collection of sensors is attached to (e.g., if a drone has a camera attached to it, the drone would be the ego and the camera would be the sensor).
• Ego coordinate system: Coordinates with respect to the ego.
• Global coordinate system: Coordinates with respect to the global origin in Unity.
• Sensor: Device that captures the dataset (in this instance the sensor is a camera).
• Sensor coordinate system: Coordinates with respect to the sensor.
• Sequence: Time-ordered series of captures. This is very useful for video capture where the time-order relationship of two captures is vital.
• UIID: Universal Unique Identifier. It is a unique hexadecimal identifier that can represent an individual instance of a capture, ego, sensor, annotation, labeled object or keypoint, or keypoint template.

Dataset Data

The dataset includes 4 types of JSON annotation files files:

• annotation_definitions.json: Contains annotation definitions for all of the active Labelers of the simulation stored in an array. Each entry consists of a collection of key-value pairs which describe a particular type of annotation and contain information about that specific annotation describing how its data should be mapped back to labels or objects in the scene. Each entry contains the following key-value pairs:
  • id: Integer identifier of the annotation's definition.
  • name: Annotation name (e.g., keypoints, bounding box, bounding box 3D, semantic segmentation).
  • description: Description of the annotation's specifications.
  • format: Format of the file containing the annotation specifications (e.g., json, PNG).
  • spec: Format-specific specifications for the annotation values generated by each Labeler.

Most Labelers generate different annotation specifications in the spec key-value pair:

• BoundingBox2DLabeler/BoundingBox3DLabeler:
  • label_id: Integer identifier of a label.
  • label_name: String identifier of a label.
• KeypointLabeler:
  • template_id: Keypoint template UUID.
  • template_name: Name of the keypoint template.
  • key_points: Array containing all the joints defined by the keypoint template. This array includes the key-value pairs:
    • label: Joint label.
    • index: Joint index.
    • color: RGBA values of the keypoint.
    • color_code: Hex color code of the keypoint
  • skeleton: Array containing all the skeleton connections defined by the keypoint template. Each skeleton connection defines a connection between two different joints. This array includes the key-value pairs:
    • label1: Label of the first joint.
    • label2: Label of the second joint.
    • joint1: Index of the first joint.
    • joint2: Index of the second joint.
    • color: RGBA values of the connection.
    • color_code: Hex color code of the connection.
• SemanticSegmentationLabeler:
  • label_name: String identifier of a label.
  • pixel_value: RGBA values of the label.
  • color_code: Hex color code of the label.

• captures_xyz.json: Each of these files contains an array of ground truth annotations generated by each active Labeler for each capture separately, as well as extra metadata that describe the state of each active sensor that is present in the scene. Each array entry in the contains the following key-value pairs:
  • id: UUID of the capture.
  • sequence_id: UUID of the sequence.
  • step: Index of the capture within a sequence.
  • timestamp: Timestamp (in ms) since the beginning of a sequence.
  • sensor: Properties of the sensor. This entry contains a collection with the following key-value pairs:
    • sensor_id: Sensor UUID.
    • ego_id: Ego UUID.
    • modality: Modality of the sensor (e.g., camera, radar).
    • translation: 3D vector that describes the sensor's position (in meters) with respect to the global coordinate system.
    • rotation: Quaternion variable that describes the sensor's orientation with respect to the ego coordinate system.
    • camera_intrinsic: matrix containing (if it exists) the camera's intrinsic calibration.
    • projection: Projection type used by the camera (e.g., orthographic, perspective).
  • ego: Attributes of the ego. This entry contains a collection with the following key-value pairs:
    • ego_id: Ego UUID.
    • translation: 3D vector that describes the ego's position (in meters) with respect to the global coordinate system.
    • rotation: Quaternion variable containing the ego's orientation.
    • velocity: 3D vector containing the ego's velocity (in meters per second).
    • acceleration: 3D vector containing the ego's acceleration (in ).
  • format: Format of the file captured by the sensor (e.g., PNG, JPG).
  • annotations: Key-value pair collections, one for each active Labeler. These key-value pairs are as follows:
    • id: Annotation UUID .
    • annotation_definition: Integer identifier of the annotation's definition.
    • filename: Name of the file generated by the Labeler. This entry is only present for Labelers that generate an image.
    • values: List of key-value pairs containing annotation data for the current Labeler.

Each Labeler generates different annotation specifications in the values key-value pair:

• BoundingBox2DLabeler:
  • label_id: Integer identifier of a label.
  • label_name: String identifier of a label.
  • instance_id: UUID of one instance of an object. Each object with the same label that is visible on the same capture has different instance_id values.
  • x: Position of the 2D bounding box on the X axis.
  • y: Position of the 2D bounding box position on the Y axis.
  • width: Width of the 2D bounding box.
  • height: Height of the 2D bounding box.
• BoundingBox3DLabeler:
  • label_id: Integer identifier of a label.
  • label_name: String identifier of a label.
  • instance_id: UUID of one instance of an object. Each object with the same label that is visible on the same capture has different instance_id values.
  • translation: 3D vector containing the location of the center of the 3D bounding box with respect to the sensor coordinate system (in meters).
  • size: 3D

General

For more details and the most up-to-date information please consult our project page: https://kainmueller-lab.github.io/fisbe.

Summary

A new dataset for neuron instance segmentation in 3d multicolor light microscopy data of fruit fly brains

30 completely labeled (segmented) images

71 partly labeled images

altogether comprising ∼600 expert-labeled neuron instances (labeling a single neuron takes between 30-60 min on average, yet a difficult one can take up to 4 hours)

To the best of our knowledge, the first real-world benchmark dataset for instance segmentation of long thin filamentous objects

A set of metrics and a novel ranking score for respective meaningful method benchmarking

An evaluation of three baseline methods in terms of the above metrics and score

Abstract

Instance segmentation of neurons in volumetric light microscopy images of nervous systems enables groundbreaking research in neuroscience by facilitating joint functional and morphological analyses of neural circuits at cellular resolution. Yet said multi-neuron light microscopy data exhibits extremely challenging properties for the task of instance segmentation: Individual neurons have long-ranging, thin filamentous and widely branching morphologies, multiple neurons are tightly inter-weaved, and partial volume effects, uneven illumination and noise inherent to light microscopy severely impede local disentangling as well as long-range tracing of individual neurons. These properties reflect a current key challenge in machine learning research, namely to effectively capture long-range dependencies in the data. While respective methodological research is buzzing, to date methods are typically benchmarked on synthetic datasets. To address this gap, we release the FlyLight Instance Segmentation Benchmark (FISBe) dataset, the first publicly available multi-neuron light microscopy dataset with pixel-wise annotations. In addition, we define a set of instance segmentation metrics for benchmarking that we designed to be meaningful with regard to downstream analyses. Lastly, we provide three baselines to kick off a competition that we envision to both advance the field of machine learning regarding methodology for capturing long-range data dependencies, and facilitate scientific discovery in basic neuroscience.

Dataset documentation:

We provide a detailed documentation of our dataset, following the Datasheet for Datasets questionnaire:

FISBe Datasheet

Our dataset originates from the FlyLight project, where the authors released a large image collection of nervous systems of ~74,000 flies, available for download under CC BY 4.0 license.

Files

fisbe_v1.0_{completely,partly}.zip

contains the image and ground truth segmentation data; there is one zarr file per sample, see below for more information on how to access zarr files.

fisbe_v1.0_mips.zip

maximum intensity projections of all samples, for convenience.

sample_list_per_split.txt

a simple list of all samples and the subset they are in, for convenience.

view_data.py

a simple python script to visualize samples, see below for more information on how to use it.

dim_neurons_val_and_test_sets.json

a list of instance ids per sample that are considered to be of low intensity/dim; can be used for extended evaluation.

Readme.md

general information

How to work with the image files

Each sample consists of a single 3d MCFO image of neurons of the fruit fly.For each image, we provide a pixel-wise instance segmentation for all separable neurons.Each sample is stored as a separate zarr file (zarr is a file storage format for chunked, compressed, N-dimensional arrays based on an open-source specification.").The image data ("raw") and the segmentation ("gt_instances") are stored as two arrays within a single zarr file.The segmentation mask for each neuron is stored in a separate channel.The order of dimensions is CZYX.

We recommend to work in a virtual environment, e.g., by using conda:

conda create -y -n flylight-env -c conda-forge python=3.9conda activate flylight-env

How to open zarr files

Install the python zarr package:

pip install zarr

Opened a zarr file with:

import zarrraw = zarr.open(, mode='r', path="volumes/raw")seg = zarr.open(, mode='r', path="volumes/gt_instances")

optional:import numpy as npraw_np = np.array(raw)

Zarr arrays are read lazily on-demand.Many functions that expect numpy arrays also work with zarr arrays.Optionally, the arrays can also explicitly be converted to numpy arrays.

How to view zarr image files

We recommend to use napari to view the image data.

Install napari:

pip install "napari[all]"

Save the following Python script:

import zarr, sys, napari

raw = zarr.load(sys.argv[1], mode='r', path="volumes/raw")gts = zarr.load(sys.argv[1], mode='r', path="volumes/gt_instances")

viewer = napari.Viewer(ndisplay=3)for idx, gt in enumerate(gts): viewer.add_labels( gt, rendering='translucent', blending='additive', name=f'gt_{idx}')viewer.add_image(raw[0], colormap="red", name='raw_r', blending='additive')viewer.add_image(raw[1], colormap="green", name='raw_g', blending='additive')viewer.add_image(raw[2], colormap="blue", name='raw_b', blending='additive')napari.run()

Execute:

python view_data.py /R9F03-20181030_62_B5.zarr

Metrics

S: Average of avF1 and C

avF1: Average F1 Score

C: Average ground truth coverage

clDice_TP: Average true positives clDice

FS: Number of false splits

FM: Number of false merges

tp: Relative number of true positives

For more information on our selected metrics and formal definitions please see our paper.

Baseline

To showcase the FISBe dataset together with our selection of metrics, we provide evaluation results for three baseline methods, namely PatchPerPix (ppp), Flood Filling Networks (FFN) and a non-learnt application-specific color clustering from Duan et al..For detailed information on the methods and the quantitative results please see our paper.

License

The FlyLight Instance Segmentation Benchmark (FISBe) dataset is licensed under the Creative Commons Attribution 4.0 International (CC BY 4.0) license.

Citation

If you use FISBe in your research, please use the following BibTeX entry:

@misc{mais2024fisbe, title = {FISBe: A real-world benchmark dataset for instance segmentation of long-range thin filamentous structures}, author = {Lisa Mais and Peter Hirsch and Claire Managan and Ramya Kandarpa and Josef Lorenz Rumberger and Annika Reinke and Lena Maier-Hein and Gudrun Ihrke and Dagmar Kainmueller}, year = 2024, eprint = {2404.00130}, archivePrefix ={arXiv}, primaryClass = {cs.CV} }

Acknowledgments

We thank Aljoscha Nern for providing unpublished MCFO images as well as Geoffrey W. Meissner and the entire FlyLight Project Team for valuablediscussions.P.H., L.M. and D.K. were supported by the HHMI Janelia Visiting Scientist Program.This work was co-funded by Helmholtz Imaging.

Changelog

There have been no changes to the dataset so far.All future change will be listed on the changelog page.

Contributing

If you would like to contribute, have encountered any issues or have any suggestions, please open an issue for the FISBe dataset in the accompanying github repository.

All contributions are welcome!

The atypical cadherins Fat and Dachsous (Ds) signal through the Hippo pathway to regulate growth of numerous organs, including the Drosophila wing. Here, we find that Ds-Fat signaling tunes a unique feature of cell proliferation found to control the rate of wing growth. The duration of the cell cycle increases in direct proportion to the size of the wing, leading to linear rather than exponential growth. Ds-Fat signaling enhances the rate at which the cell cycle lengthens with wing size, thus diminishing the linear rate of wing growth. We show that this results in a complex but stereotyped relative scaling of wing growth with body growth in Drosophila. Finally, we examine the dynamics of Fat and Ds protein distribution in the wing, observing graded distributions that change during growth. However, the significance of these dynamics is unclear since perturbations in expression have negligible impact on wing growth. Methods Data accompanying a published eLife paper titled "Scaling between cell cycle duration and wing growth is regulated by Fat-Dachsous signaling in Drosophila."

Imaging data were acquired on confocal micrscope and processed. Please refer to the Methods section of the accompanying eLife paper for detailed methodology of how images were collected. How the data was processed are descripted below. Code is available at https://github.com/andrewliu321/LarvaSeg and https://github.com/andrewliu321/ProteinIntensity.

Image processing

Raw images were processed using a custom-built Matlab pipeline with no prior preprocessing. The pipeline consists of several modules: 1) surface detection, 2) wing disc, pouch, and midline segmentation, 3) volume measurement, 4) fluorescence intensity measurement, 5) cell segmentation, 6) mitotic index measurement.

Surface detection

Fat-GFP and Ds-GFP proteins localize to the apical region of cells in the wing disc proper. These proteins are also localized in cells of the peripodial membrane, which is positioned near the apical surface of the disc proper. In order to get a 2D projection of the signal in the wing disc proper excluding the peripodial membrane signal, we used an open-source software package called ImSAnE – Image Surface Analysis Environment. The detailed parameters we used have been previously described. Briefly, we used the MIPDetector module to find the brightest z-position of every xy pixel followed by tpsFitter to fit a single layer surface through these identified z-positions. Using the onionOpts function in ImSAnE, we output a 9-layer z-stack, 4 layers above and below the computed surface that capture the entire signal from the wing disc proper. However, this operation still sometimes includes fluorescence signals from the peripodial membrane. Therefore, we manually masked the residual peripodial signal using FIJI 1.53t. The resulting z-stack was sum-projected to form a 2D surface projection of the wing disc proper.

Wing disc, pouch, and midline segmentation

Wing discs were counterstained for both Wg and En proteins, which mark the wing pouch dorsal-ventral (DV) midline and anterior-posterior (AP) midline, respectively. Although both Wg and En proteins were stained with the same Alexa 546 fluorescent antibody, the two signals were readily distinguished by their distinct separation in z space. Wg is apically localized in cells of the disc proper and En is nuclear localized more basally in the disc proper. Moreover, the Wg signal was far stronger than En, allowing for detection of its stripe in the posterior compartment even with a max projection.

We built a semi-automated Matlab script that computationally segments the wing disc into discrete objects. i) Wing disc segmentation. Endogenous Fat-GFP or Ds-GFP signal was used to segment the wing disc from surrounding media. ii) Wing pouch segmentation. The 3D morphology of the wing disc creates narrow and deep tissue folds that surround the wing pouch. Using these morphological landmarks that were visualized by the Fat-GFP or Ds-GFP signals, the Matlab script recorded user-derived mouse-clicks that defined the wing pouch boundary. This method was validated to be 95.5% accurate when compared to a wing pouch boundary that was defined by the expression boundary of a reporter for the vestigial quadrant enhancer, 5x-QE-DsRed. iii) DV midline segmentation. The Wg signal was used to segment the DV midline running through the segmented wing pouch. Since the Wg and En signals are distinguishable in z space, the upper third of the z-stack was max projected to segment Wg. An adaptive threshold of 0.6 was used to binarize the image into Wg-positive pixels. The binarized and raw images were used to inform manual input of the DV midline. iv) AP midine segmentation. The En signal was used to segment the AP midline running through the segmented wing pouch. The lower two-thirds of the z-stack was max projected and binarized in En+ pixels. The binarized and raw images were used to inform manual input of the AP midline.

Volume measurement

Areas of each of the segmented objects were calculated by summing the number of pixels in each object and multiplying by the pixel dimensions in xy physical space. Notum-hinge area was calculated by subtracting the segmented pouch area from total segmented wing disc area. The thickness of the wing pouch was measured at the intersection of the AP and DV midlines using the orthogonal views tool in FIJI. The first layer is defined by the initial signal of Fat-GFP or Ds-GFP at this xy position. The last layer is defined by the first appearance of background signal in the composite image. Thickness of the object was calculated by multiplying the sum of z-slices by the z-separation. To calculate the volume of segmented objects, we multiplied the thickness of the object (in µm) by the object’s surface area (in µm2). Conversion from µm3 to nL units was performed.

Fluorescence intensity measurement of Fat-GFP and Ds-GFP

Fluorescence intensity values were averaged across a vector of 50 pixels length that was orthogonal to the segmented boundary of interest and having 25 pixels residing on each side of the segmented line. These values were then averaged in a sliding window of 100 pixels length that moved along the segmented boundary of interest. Physical distance along the boundaries were measured using ImSAnE function Proper_Dist to account for the curvature of the segmented objects. The intersection of the segmented DV and AP midlines was defined as the center (0,0 µm) of the wing pouch, with the anterior/dorsal annotated in units of negative µm and the posterior/ventral annotated in units of positive µm. A minimum of three wing discs of the same age and genotype were aligned by their (0,0) centers and their fluorescent measurements were averaged along the AP and DV midlines.

Older third instar, WPP, and BPP wing discs begin to evert such that the ventral compartment is partially folded underneath the dorsal compartment. However, the ventral compartment can still be accurately segmented to measure area and thickness even under these conditions. For GFP measurements, the folded specimen resulted in dimmer fluorescent signals from the compartment farthest from the objective due to tissue thickness and light scattering. Thus, measurements were limited to the compartment closest to the objective.

Cell boundary segmentation

To count cell numbers and cell sizes in the wing pouch, we analyzed wing discs imaged from E-cadherin-GFP or E-cadherin-mCherry larvae. We used a machine learning pixel-classification model based on a convolutional neural net to segment cell boundaries in the surface projections. This model was trained on a broad range of image data derived from Cadherin-GFP labeled Drosophila imaginal discs. The model is > 99.5% accurate at segmenting cells when compared to ground truth. Cell size (surface area) and number were computed for specific compartments in the wing pouch.

Mitotic index measurement

Phospho-histone H3 (PHH3) has been used to estimate mitotic index previously. Wing discs were immunostained for PHH3, which labels nuclei undergoing mitosis. These nuclei were manually recorded by user-defined mouse clicks at or near the center of each nucleus. Their Euclidean distances relative to the segmented AP and DV midlines were calculated as was the number of PHH3+ cells. To estimate the total cell number in a wing pouch, we used E-cadherin to computationally segment cells as described above. Each imaged wing pouch had a subset of cell boundaries segmented in a subdomain of the pouch. This was then used to calculate cell density: number of segmented cells divided by subdomain area. The density value was multiplied by total wing pouch area to estimate the total number of wing pouch cells for that sample. We then derived an averaged conversion factor to apply to each volume measurement in order to estimate total cell number. This was done by plotting the estimated total cell number versus wing pouch volume for all discs of a given genotype. Linear regression of the data produced an equation to convert pouch volume to cell number.

This dataset includes brain MRI scans of adult brain glioma patients, comprising of 4 structural modalities (i.e., T1, T1c, T2, T2-FLAIR) and associated manually generated ground truth labels for each tumor sub-region (enhancement, necrosis, edema), as well as their MGMT promoter methylation status. These scans are a collection of data from existing TCIA collections, but also cases provided by individual institutions and willing to share with a cc-by license. The BraTS dataset describes a retrospective collection of brain tumor structural mpMRI scans of 2,040 patients (1,480 here), acquired from multiple different institutions under standard clinical conditions, but with different equipment and imaging protocols, resulting in a vastly heterogeneous image quality reflecting diverse clinical practice across different institutions. The 4 structural mpMRI scans included in the BraTS challenge describe a) native (T1) and b) post-contrast T1-weighted (T1Gd (Gadolinium)), c) T2-weighted (T2), and d) T2 Fluid Attenuated Inversion Recovery (T2-FLAIR) volumes, acquired with different protocols and various scanners from multiple institutions. Furthermore, data on the O[6]-methylguanine-DNA methyltransferase (MGMT) promoter methylation status is provided as a binary label. Notably, MGMT is a DNA repair enzyme that the methylation of its promoter in newly diagnosed glioblastoma has been identified as a favorable prognostic factor and a predictor of chemotherapy response. It is curated for computational image analysis of segmentation and prediction of the MGMT promoter methylation status.

A note about available TCIA data which were converted for use in this Challenge: (Training, Validation, Test)