• [2024.07.30] segment-anything-2-main.zip is from https://github.com/facebookresearch/segment-anything-2

Each folder in 'Prompting data.zip' corresponds to a single category (Bird, Cat, Bus etc), and each of these contain folders corresponding to a single participant (st1, st2 etc). Each participant folder should contain 5 subfolders:

• 'masks' contains the binary masks produced for each image, in the format a_b_mask.png, where 'a' corresponds to the image number (0 to 399) and 'b' indexes through timestamps in the prompting process
• 'points' contains the inclusion and exclusion points formatted as a_green.npy and a_red.npy respectively, where 'a' corresponds to the image number. Each of these files is a list of lists corresponding to the prompted points at each timestep. The outer list is of size (t,), where 't' is the number of timesteps for that image, an each inner list is fo size (n,2), where 'n' is the number of points at a given timestep
• 'scores' contains the scores at each timestep for every image (mIoU)
• 'sorts' contains sorted timestamp indexes, going from max to min based on the score
• 'eachround' indicates which timesteps belong to each of the two rounds (if they exist). Each file contains a list of lenght t (number of timestamps) where values of 0 corresponds to timestamps that belong to the first round and values of 1 correspond to timestamps that belong to the second round

Quick usage:

-To get the best (highes score) mask for a given image : masks[sorts[0]]
-To get the best set of prompts for that image : green[sorts[0]] and red[sorts[0]]
-To get which round produced the highest score in that image : eachround[sorts[0]]

The codebase associated with this work can be found at this Github.

Please refer to our lab-wide github for more information regarding the code associated with our other papers.

Introduction This dataset aims to explore the realm of object detection and segmentation with a specific focus on its applications in agriculture. The primary objective is to employ YOLOv8 and SAM techniques to develop robust models for detecting grape bunches.

Dataset Description The dataset comprises four trained models utilizing YOLOv8 architecture. It includes two single-class models, one utilizing object detection and the other employing instance segmentation for grape detection. Additionally, there are two multi-class models capable of predicting and detecting different grape varietals. All models were trained using the large model from the Ultralytics repository (https://github.com/ultralytics/ultralytics).

The dataset encompasses four grape varietals: - Pinot Noir: 102 images and labels - Chardonnay: 39 images and labels from me 47 from thsant - Sauvignon Blanc: 42 images and labels - Pinot Gris: 111 images and labels

Total used for training: 341

Note that the training of the segmentation models used a total of 20 images from each for a total of 100.

Datasets Used for Training To see the dataset (e.g train/test/val folders) used for training the multi class object detection model please see the following zip file and note book:

https://www.kaggle.com/code/nicolaasregnier/buildtraindataset/output?select=train-dataset-multi-class.zip

To build a custom train-dataset please follow the instructions in the notebook: https://www.kaggle.com/code/nicolaasregnier/buildtraindataset/

The labels used for training the multi-class instance segmentation model are under the folder SAMPreds

Data Sources The dataset incorporates two primary data sources. The first source is a collection of images captured using an iPad Air 2 RGB camera. These images possess a resolution of 3226x2449 pixels and an 8-megapixel quality. The second source is contributed by GitHub user thsant, who has created an impressive project available at https://github.com/thsant/wgisd/tree/master.

To label the data, a base model from a previous dataset was utilized, and the annotation process was carried out using LabelImg (https://github.com/heartexlabs/labelImg). It is important to note that some annotations from thsant's dataset required modifications for completeness.

Implementation Steps The data preparation involved the utilization of classes and functions from the "my_SAM" (https://github.com/regs08/my_SAM) and "KaggleUtils" (https://github.com/regs08/KaggleUtils) repositories, facilitating the creation of training sets and the application of SAM techniques.

For model training, the YOLOv8 architecture with default hyperparameters was employed. The object detection models underwent 50 epochs of training, while the instance segmentation models were trained for 75 epochs.

Segment Anything (SAM) from https://segment-anything.com/ was applied to the bbox-labeled data to generate images and corresponding masks for the instance segmentation models. No further editing of the images occurred after applying SAM.

Evaluation and Inference The evaluation metrics utilized were Mean Average Precision (mAP). The following mAP values were obtained:

Single-class object detection: - mAP50: 0.85449 - mAP50-95: 0.56177

Multi-class object detection: - mAP50: 0.85336 - mAP50-95: 0.56316

Single-class instance segmentation: - mAP50: (value not provided) - mAP50-95: (value not provided)

Multi-class instance segmentation: - mAP50: 0.89436 - mAP50-95: 0.62785

For more comprehensive metrics, please refer to the results folder corresponding to the model of interest.

General

For more details and the most up-to-date information please consult our project page: https://kainmueller-lab.github.io/fisbe.

Summary

• A new dataset for neuron instance segmentation in 3d multicolor light microscopy data of fruit fly brains
  • 30 completely labeled (segmented) images
  • 71 partly labeled images
  • altogether comprising ∼600 expert-labeled neuron instances (labeling a single neuron takes between 30-60 min on average, yet a difficult one can take up to 4 hours)
• To the best of our knowledge, the first real-world benchmark dataset for instance segmentation of long thin filamentous objects
• A set of metrics and a novel ranking score for respective meaningful method benchmarking
• An evaluation of three baseline methods in terms of the above metrics and score

Abstract

Instance segmentation of neurons in volumetric light microscopy images of nervous systems enables groundbreaking research in neuroscience by facilitating joint functional and morphological analyses of neural circuits at cellular resolution. Yet said multi-neuron light microscopy data exhibits extremely challenging properties for the task of instance segmentation: Individual neurons have long-ranging, thin filamentous and widely branching morphologies, multiple neurons are tightly inter-weaved, and partial volume effects, uneven illumination and noise inherent to light microscopy severely impede local disentangling as well as long-range tracing of individual neurons. These properties reflect a current key challenge in machine learning research, namely to effectively capture long-range dependencies in the data. While respective methodological research is buzzing, to date methods are typically benchmarked on synthetic datasets. To address this gap, we release the FlyLight Instance Segmentation Benchmark (FISBe) dataset, the first publicly available multi-neuron light microscopy dataset with pixel-wise annotations. In addition, we define a set of instance segmentation metrics for benchmarking that we designed to be meaningful with regard to downstream analyses. Lastly, we provide three baselines to kick off a competition that we envision to both advance the field of machine learning regarding methodology for capturing long-range data dependencies, and facilitate scientific discovery in basic neuroscience.

Dataset documentation:

We provide a detailed documentation of our dataset, following the Datasheet for Datasets questionnaire:

>> FISBe Datasheet

Our dataset originates from the FlyLight project, where the authors released a large image collection of nervous systems of ~74,000 flies, available for download under CC BY 4.0 license.

Files

• fisbe_v1.0_{completely,partly}.zip
  • contains the image and ground truth segmentation data; there is one zarr file per sample, see below for more information on how to access zarr files.
• fisbe_v1.0_mips.zip
  • maximum intensity projections of all samples, for convenience.
• sample_list_per_split.txt
  • a simple list of all samples and the subset they are in, for convenience.
• view_data.py
  • a simple python script to visualize samples, see below for more information on how to use it.
• dim_neurons_val_and_test_sets.json
  • a list of instance ids per sample that are considered to be of low intensity/dim; can be used for extended evaluation.
• Readme.md
  • general information

How to work with the image files

Each sample consists of a single 3d MCFO image of neurons of the fruit fly.
For each image, we provide a pixel-wise instance segmentation for all separable neurons.
Each sample is stored as a separate zarr file (zarr is a file storage format for chunked, compressed, N-dimensional arrays based on an open-source specification.").
The image data ("raw") and the segmentation ("gt_instances") are stored as two arrays within a single zarr file.
The segmentation mask for each neuron is stored in a separate channel.
The order of dimensions is CZYX.

We recommend to work in a virtual environment, e.g., by using conda:

conda create -y -n flylight-env -c conda-forge python=3.9
conda activate flylight-env

How to open zarr files

1. Install the python zarr package:

   pip install zarr

2. Opened a zarr file with:
   

   import zarr
raw = zarr.open(
seg = zarr.open(

   # optional:
import numpy as np
raw_np = np.array(raw)

Zarr arrays are read lazily on-demand.
Many functions that expect numpy arrays also work with zarr arrays.
Optionally, the arrays can also explicitly be converted to numpy arrays.

How to view zarr image files

We recommend to use napari to view the image data.

1. Install napari:

   pip install "napari[all]"

2. Save the following Python script:
   

   import zarr, sys, napari

   raw = zarr.load(sys.argv[1], mode='r', path="volumes/raw")
gts = zarr.load(sys.argv[1], mode='r', path="volumes/gt_instances")

   viewer = napari.Viewer(ndisplay=3)
for idx, gt in enumerate(gts):
viewer.add_labels(
gt, rendering='translucent', blending='additive', name=f'gt_{idx}')
viewer.add_image(raw[0], colormap="red", name='raw_r', blending='additive')
viewer.add_image(raw[1], colormap="green", name='raw_g', blending='additive')
viewer.add_image(raw[2], colormap="blue", name='raw_b', blending='additive')
napari.run()

3. Execute:

   python view_data.py 

Metrics

• S: Average of avF1 and C
• avF1: Average F1 Score
• C: Average ground truth coverage
• clDice_TP: Average true positives clDice
• FS: Number of false splits
• FM: Number of false merges
• tp: Relative number of true positives

For more information on our selected metrics and formal definitions please see our paper.

Baseline

To showcase the FISBe dataset together with our selection of metrics, we provide evaluation results for three baseline methods, namely PatchPerPix (ppp), Flood Filling Networks (FFN) and a non-learnt application-specific color clustering from Duan et al..
For detailed information on the methods and the quantitative results please see our paper.

License

The FlyLight Instance Segmentation Benchmark (FISBe) dataset is licensed under the Creative Commons Attribution 4.0 International (CC BY 4.0) license.

Citation

If you use FISBe in your research, please use the following BibTeX entry:

@misc{mais2024fisbe,
 title =    {FISBe: A real-world benchmark dataset for instance
         segmentation of long-range thin filamentous structures},
 author =    {Lisa Mais and Peter Hirsch and Claire Managan and Ramya
         Kandarpa and Josef Lorenz Rumberger and Annika Reinke and Lena
         Maier-Hein and Gudrun Ihrke and Dagmar Kainmueller},
 year =     2024,
 eprint =    {2404.00130},
 archivePrefix ={arXiv},
 primaryClass = {cs.CV}
}

Acknowledgments

We thank Aljoscha Nern for providing unpublished MCFO images as well as Geoffrey W. Meissner and the entire FlyLight Project Team for valuable
discussions.
P.H., L.M. and D.K. were supported by the HHMI Janelia Visiting Scientist Program.
This work was co-funded by Helmholtz Imaging.

Changelog

There have been no changes to the dataset so far.
All future change will be listed on the changelog page.

Contributing

If you would like to contribute, have encountered any issues or have any suggestions, please open an issue for the FISBe dataset in the accompanying github repository.

All contributions are welcome!

Citation (TBD)

@article{mollineda2025sam,
 title={Sex classification from hand X-ray images in pediatric patients: how zero-shot Segment Anything Model (SAM) can improve medical image analysis},
 author={Ramón A. Mollineda, Becerra Karel, Mederos Boris},
 journal={Computers in Biology and Medicine},
 year={2025 TBD}
}

GitHub (full source code): https://github.com/karelbecerra/sam-x-ray-medical-images-hand-sex-classification

https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F15158801%2Fa0bbbcdf339757ea914b8bbe5b0e2368%2Foutput.png?generation=1734016330629261&alt=media" alt=""> https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F15158801%2Fe34d08df8a86389986a56d21309ab05a%2Foutput.png?generation=1734015411698176&alt=media" alt=""> https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F15158801%2Fbb090716b3b62c28d78a6a1e0ba4ff27%2Foutput.png?generation=1734015350940879&alt=media" alt=""> https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F15158801%2F46e52530c372de21b2b25f517c3087b2%2Foutput.png?generation=1734015445526796&alt=media" alt=""> https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F15158801%2F8e96bbf814d51d25ba43233df2295067%2Foutput.png?generation=1734015266920312&alt=media" alt=""> https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F15158801%2F2fc7354933806283986e88cac8479eb5%2Foutput.png?generation=1734015320114184&alt=media" alt="">

Description

Starting from Bone Age RSNA Challenge this datasets contains 6 subsets in '**npz**' formats (Numpy zips).

Each includes: training, validations and test subsets.

Original

Original RSNA Bone Age Challenge datasets

Hand Regions

Including x-ray information only

Enhanced Hand Regios

Enhanced x-ray only region.

Hand Masks

All x-ray information removed

Background (after subtracting the hand region)

Excluding x-ray information

Background (after subtracting the convex hull)

Excluding x-ray information and any morphological information from hand

Ground-based observations from fixed-mount cameras have the potential to fill an important role in environmental sensing, including direct measurement of water levels and qualitative observation of ecohydrological research sites. All of this is theoretically possible for anyone who can install a trail camera. Easy acquisition of ground-based imagery has resulted in millions of environmental images stored, some of which are public data, and many of which contain information that has yet to be used for scientific purposes. The goal of this project was to develop and document key image processing and machine learning workflows, primarily related to semi-automated image labeling, to increase the use and value of existing and emerging archives of imagery that is relevant to ecohydrological processes.

   This data package includes imagery, annotation files, water segmentation model and model performance plots, and model test results (overlay images and masks) for USGS Monitoring Site East Branch Brandywine Creek below Downingtown, PA. All imagery was acquired from the USGS Hydrologic Imagery Visualization and Information System (HIVIS; see https://apps.usgs.gov/hivis/camera/PA_East_Branch_Brandywine_Creek_below_Downingtown for this specific data set) and/or the National Imagery Management System (NIMS) API.

   Water segmentation models were created by tuning the open-source Segment Anything Model 2 (SAM2, https://github.com/facebookresearch/sam2) using images that were annotated by team members on this project. The models were trained on the "water" annotations, but annotation files may include additional labels, such as "snow", "sky", and "unknown". Image annotation was done in Computer Vision Annotation Tool (CVAT) and exported in COCO format (.json).

   All model training and testing was completed in GaugeCam Remote Image Manager Educational Artificial Intelligence (GRIME AI, https://gaugecam.org/) software (Version: Beta 16). Model performance plots were automatically generated during this process.

   This project was conducted in 2023-2025 by collaborators at the University of Nebraska-Lincoln, University of Nebraska at Kearney, and the U.S. Geological Survey.

   This material is based upon work supported by the U.S. Geological Survey under Grant/Cooperative Agreement No. G23AC00141-00. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the opinions or policies of the U.S. Geological Survey. Mention of trade names or commercial products does not constitute their endorsement by the U.S. Geological Survey. We gratefully acknowledge graduate student support from Daugherty Water for Food Global Institute at the University of Nebraska.

Laboro Tomato Github

## Overview

Laboro Tomato Github is a dataset for instance segmentation tasks - it contains Ripness Segmentation Laboro annotations for 804 images.

## Getting Started

You can download this dataset for use within your own projects, or fork it into a workspace on Roboflow to create your own model.

  ## License

  This dataset is available under the [BY-NC-SA 4.0 license](https://creativecommons.org/licenses/BY-NC-SA 4.0).

IAM: Enhancing RGB-D Instance Segmentation with New Benchmarks

For detailed statistics about our datasets, please refer to the following paper:Preprint: https://arxiv.org/abs/2501.01685 Github pages:https://github.com/AIM-SKKU/NYUDv2-IS https://github.com/AIM-SKKU/SUN-RGBD-IS https://github.com/AIM-SKKU/Box-IS

Doodleverse/Segmentation Zoo/Seg2Map Res-UNet models for OpenEarthMap/9-class segmentation of RGB 512x512 high-res. images

These Residual-UNet model data are based on the OpenEarthMap dataset

Models have been created using Segmentation Gym* using the following dataset**: https://zenodo.org/record/7223446#.Y9gtWHbMIuV

Image size used by model: 512 x 512 x 3 pixels

classes: 1. bareland 2. rangeland 3. development 4. road 5. tree 6. water 7. agricultural 8. building 9. nodata

File descriptions

For each model, there are 5 files with the same root name:

1. '.json' config file: this is the file that was used by Segmentation Gym* to create the weights file. It contains instructions for how to make the model and the data it used, as well as instructions for how to use the model for prediction. It is a handy wee thing and mastering it means mastering the entire Doodleverse.

2. '.h5' weights file: this is the file that was created by the Segmentation Gym* function train_model.py. It contains the trained model's parameter weights. It can called by the Segmentation Gym* function seg_images_in_folder.py. Models may be ensembled.

3. '_modelcard.json' model card file: this is a json file containing fields that collectively describe the model origins, training choices, and dataset that the model is based upon. There is some redundancy between this file and the config file (described above) that contains the instructions for the model training and implementation. The model card file is not used by the program but is important metadata so it is important to keep with the other files that collectively make the model and is such is considered part of the model

4. '_model_history.npz' model training history file: this numpy archive file contains numpy arrays describing the training and validation losses and metrics. It is created by the Segmentation Gym function train_model.py

5. '.png' model training loss and mean IoU plot: this png file contains plots of training and validation losses and mean IoU scores during model training. A subset of data inside the .npz file. It is created by the Segmentation Gym function train_model.py

Additionally, BEST_MODEL.txt contains the name of the model with the best validation loss and mean IoU

References *Segmentation Gym: Buscombe, D., & Goldstein, E. B. (2022). A reproducible and reusable pipeline for segmentation of geoscientific imagery. Earth and Space Science, 9, e2022EA002332. https://doi.org/10.1029/2022EA002332 See: https://github.com/Doodleverse/segmentation_gym

**Xia, Yokoya, Adriano, & Broni-Bediako. (2022). OpenEarthMap: A Benchmark Dataset for Global High-Resolution Land Cover Mapping [Data set]. Zenodo. https://doi.org/10.5281/zenodo.7223446

Images and 4-class labels for semantic segmentation of Sentinel-2 and Landsat RGB, NIR, and SWIR satellite images of coasts (water, whitewater, sediment, other)

Description

579 images and 579 associated labels for semantic segmentation of Sentinel-2 and Landsat RGB satellite images of coasts. The 4 classes are 0=water, 1=whitewater, 2=sediment, 3=other

These images and labels have been made using the Doodleverse software package, Doodler*. These images and labels could be used within numerous Machine Learning frameworks for image segmentation, but have specifically been made for use with the Doodleverse software package, Segmentation Gym**.

Some (422) of these images and labels were originally included in the Coast Train*** data release, and have been modified from their original by reclassifying from the original classes to the present 4 classes.

The label images are a subset of the following data release**** https://doi.org/10.5281/zenodo.7335647

Imagery comes from the following 10 sand beach sites:

1. Duck, NC, Hatteras NC, USA
2. Santa Cruz CA, USA
3. Galveston TX, USA
4. Truc Vert,France
5. Sunset State Beach CA, USA
6. Torrey Pines CA, USA
7. Narrabeen, NSW, Australia
8. Elwha WA, USA
9. Ventura region, CA, USA
10. Klamath region, CA USA

Imagery are a mixture of 10-m Sentinel-2 and 15-m pansharpened Landsat 7, 8, and 9 visible-band imagery of various sizes. Red, Green, Blue, NIR, and SWIR bands only

File descriptions

1. classes.txt, a file containing the class names
2. images.zip, a zipped folder containing the 3-band RGB images of varying sizes and extents
3. nir.zip, a zipped folder containing the corresponding near-infrared (NIR) imagery
4. swir.zip, a zipped folder containing the corresponding shortwave-infrared (SWIR) imagery
5. labels.zip, a zipped folder containing the 1-band label images
6. overlays.zip, a zipped folder containing a semi-transparent overlay of the color-coded label on the image (blue=0=water, red=1=whitewater, yellow=2=sediment, green=3=other)
7. resized_images.zip, RGB images resized to 512x512x3 pixels
8. resized_nir.zip, NIR images resized to 512x512x3 pixels
9. resized_swir.zip, SWIR images resized to 512x512x3 pixels
10. resized_labels.zip, label images resized to 512x512 pixels

References

*Doodler: Buscombe, D., Goldstein, E.B., Sherwood, C.R., Bodine, C., Brown, J.A., Favela, J., Fitzpatrick, S., Kranenburg, C.J., Over, J.R., Ritchie, A.C. and Warrick, J.A., 2021. Human‐in‐the‐Loop Segmentation of Earth Surface Imagery. Earth and Space Science, p.e2021EA002085https://doi.org/10.1029/2021EA002085. See https://github.com/Doodleverse/dash_doodler.

**Segmentation Gym: Buscombe, D., & Goldstein, E. B. (2022). A reproducible and reusable pipeline for segmentation of geoscientific imagery. Earth and Space Science, 9, e2022EA002332. https://doi.org/10.1029/2022EA002332 See: https://github.com/Doodleverse/segmentation_gym

***Coast Train data release: Wernette, P.A., Buscombe, D.D., Favela, J., Fitzpatrick, S., and Goldstein E., 2022, Coast Train--Labeled imagery for training and evaluation of data-driven models for image segmentation: U.S. Geological Survey data release, https://doi.org/10.5066/P91NP87I. See https://coasttrain.github.io/CoastTrain/ for more information

**** Buscombe, Daniel, Goldstein, Evan, Bernier, Julie, Bosse, Stephen, Colacicco, Rosa, Corak, Nick, Fitzpatrick, Sharon, del Jesús González Guillén, Anais, Ku, Venus, Paprocki, Julie, Platt, Lindsay, Steele, Bethel, Wright, Kyle, & Yasin, Brandon. (2022). Images and 4-class labels for semantic segmentation of Sentinel-2 and Landsat RGB satellite images of coasts (water, whitewater, sediment, other) (v1.0) [Data set]. Zenodo. https://doi.org/10.5281/zenodo.7335647

Links

Paper: https://arxiv.org/pdf/1908.09101v2 Repository: https://github.com/Mhaiyang/ICCV2019_MirrorNet Project page: https://mhaiyang.github.io/ICCV2019_MirrorNet/index.html We got our data from: https://github.com/Charmve/Mirror-Glass-Detection

  Split info

We split the train to train and validation with the ratio 80% and 20% respectively. If you want to use the original split, you can just combine train and validation.

  License info

Refer to the… See the full description on the dataset page: https://huggingface.co/datasets/rdyzakya/Mirror-Segmentation-Dataset.

Dataset

This dataset was created by NguyenThanhNhan

Contents

Doodleverse/Segmentation Zoo Res-UNet models for 2-class (water, other) segmentation of Sentinel-2 and Landsat-7/8 1-band MNDWI images of coasts.

These Residual-UNet model data are based on MNDWI images of coasts and associated labels.

Models have been created using Segmentation Gym* using the following dataset**: https://doi.org/10.5281/zenodo.7384263

Classes: {0=other, 1=water}

File descriptions

For each model, there are 5 files with the same root name:

1. '.json' config file: this is the file that was used by Segmentation Gym* to create the weights file. It contains instructions for how to make the model and the data it used, as well as instructions for how to use the model for prediction. It is a handy wee thing and mastering it means mastering the entire Doodleverse.

2. '.h5' weights file: this is the file that was created by the Segmentation Gym* function `train_model.py`. It contains the trained model's parameter weights. It can called by the Segmentation Gym* function `seg_images_in_folder.py`. Models may be ensembled.

3. '_modelcard.json' model card file: this is a json file containing fields that collectively describe the model origins, training choices, and dataset that the model is based upon. There is some redundancy between this file and the `config` file (described above) that contains the instructions for the model training and implementation. The model card file is not used by the program but is important metadata so it is important to keep with the other files that collectively make the model and is such is considered part of the model

4. '_model_history.npz' model training history file: this numpy archive file contains numpy arrays describing the training and validation losses and metrics. It is created by the Segmentation Gym function `train_model.py`

5. '.png' model training loss and mean IoU plot: this png file contains plots of training and validation losses and mean IoU scores during model training. A subset of data inside the .npz file. It is created by the Segmentation Gym function `train_model.py`

Additionally, BEST_MODEL.txt contains the name of the model with the best validation loss and mean IoU

References

*Segmentation Gym: Buscombe, D., & Goldstein, E. B. (2022). A reproducible and reusable pipeline for segmentation of geoscientific imagery. Earth and Space Science, 9, e2022EA002332. https://doi.org/10.1029/2022EA002332 See: https://github.com/Doodleverse/segmentation_gym

** Buscombe, Daniel. (2022). Images and 2-class labels for semantic segmentation of Sentinel-2 and Landsat RGB, NIR, and SWIR satellite images of coasts (water, other) (v1.0) [Data set]. Zenodo. https://doi.org/10.5281/zenodo.7384263

https://drive.google.com/uc?id=1x6OsMmimLrwrYwiNm9EuIDh0-GThLik-" alt="">

Project Overview: The Caridina and Neocaridina Shrimp Detection Project aims to develop and improve computer vision algorithms for detecting and distinguishing between different shrimp varieties. This project is centered around aquarium fish keeping hobbyist and how computer vision can be beneficial to improving the care of dwarf shrimp. This project will focus on zoning a feeding area and tracking and counting caridina shrimp in area.

Caridina and neo-caridina shrimp are two distinct species that require different water parameters for optimal health. Neocaridina shrimp are generally more hardy and easier to keep than caridina species, while caridina shrimp are known for their striking distinctive patterns. The body structure of both species are similar. However, there are specific features that should allow enough sensitivity to at least distinguish between caridina shrimp.

Descriptions of Each Class Type: The dataset for this project includes thirteen different class types. The neo-caridina species have been grouped together to test if the model can distinguish between caridina and neo-caridina shrimp. The remaining classes are all different types of caridina shrimp.

The RGalaxyPinto and BGalaxyPinto varieties are caridina shrimp, with the only difference being their color: one is wine-red while the other dark-blue-black. Both varieties have distinctive spots on the head region and stripes on their backs, making them ideal for testing the model's ability to distinguish between color.

https://drive.google.com/uc?id=19zPYu8YbCiRHUF9K_3kCsyw0X2Tog-Ts" alt="">	https://drive.google.com/uc?id=1Ay728IysDP8yMCwPEi743Bp6mnq5Xrix" alt="">
https://drive.google.com/uc?id=1Asa3DwuWop5UDpBThHgGG6otBSyXgJTV" alt="">

The CRS-CBS Crystal Red Shrimp and Crystal Black Shrimp have similar patterns to the Panda Bee shrimp, but the hues are different. Panda shrimp tend to be a deeper and richer color than CRS-CBS shrimp, CRS-CBS tend to have thicker white rings.

https://drive.google.com/uc?id=1AXlBcHGGZ9VEnNuoxeEFZf0DTPQa5hTR" alt="">	https://drive.google.com/uc?id=1BO2DwW77AqzDrj3xP9VOEYOXSP4wgRzz" alt="">
https://drive.google.com/uc?id=19yO42UW_ai11Da3KgaEiUEHn0OnJc0As" alt="">

The Panda Bee variety, on the other hand, is known for its panda-like pattern white and black/red rings.The color rings tend to be thicker and more pronounced than the Crystal Red/Black Shrimp.

Within the Caridina species, there are various tiger varieties. These include Fancy Tiger, Raccoon Tiger, Tangerine Tiger, Orange Eyed Tiger (Blonde and Full Body). All of these have stripes along the sides of their bodies. Fancy Tiger shrimp have a similar color to CRS, but with a tiger stripe pattern. Raccoon Tiger and Orange Eyed Tiger Blonde look very similar, but the body of the Raccoon Tiger appears larger, and the Orange Eyed Tiger is known for its orange eyes. Tangerine Tigers vary in stripe pattern and can often be confused with certain neo-caridina, specifically yellow or orange varieties.

https://drive.google.com/uc?id=1APx9jQ5WUdPbv1US8ihOEBpVBjvhN0Z3" alt="">	https://drive.google.com/uc?id=1B6MbiN9FY9fomf6-P6zy-jkoGJKEiXlW" alt="">	https://drive.google.com/uc?id=1A3qYXbPkqjeK2oCJfSLAPwEsEZN9nw8NN" alt="">
https://drive.google.com/uc?id=19ukHly3uZ05FeGdW_hVBWwlHRFvgnMMC" alt="">	https://drive.google.com/uc?id=1AztJj471aIWcRYHNC1lrJse7raO2dUqm" alt="">

The remaining are popular favorites for breeding and distinct color patterns namely Bluebolt, Shadow Mosura, White Bee/Golden Bee, and King Kong Bee.

https://drive.google.com/uc?id=19yEpuJ6ENmkcImu0OfCzliITP_UnCNoM" alt="">	https://drive.google.com/uc?id=19uglS20nyTSi-_b1ls8f09cIuJUHOpSm" alt="">
https://drive.google.com/uc?id=1AbbCVRnlIQL1MlqY3MJnX9t2WVdyq2zJ" alt="">

Links to External Resources: Here are some resources that provide additional information on the shrimp varieties and other resources used in this project:

Caridina Shrimp: https://en.wikipedia.org/wiki/Bee_shrimp
Neo-Caridina Shrimp: https://en.wikipedia.org/wiki/Neocaridina
  Roboflow Polygon Zoning/Tracking/Counting:https://colab.research.google.com/github/roboflow-ai/notebooks/blob/main/notebooks/how-to-detect-and-count-objects-in-polygon-zone.ipynb
  Roboflow SAM: https://colab.research.google.com/github/roboflow-ai/notebooks/blob/main/notebooks/how-to-segment-anything-with-sam.ipynb
  Ultralytics Hub:https://github.com/ultralytics/hub

Our paper has been published on Phys. Rev. Res. (doi: 10.1103/PhysRevResearch.6.013259)

Our preprint paper is also avilable at arXiv(https://arxiv.org/abs/2310.06406), here is the abstract of our paper:

Topological defects play a key role in the structures and dynamics of liquid crystals (LCs) and other ordered systems. There is a recent interest in studying defects in different biological systems with distinct textures. However, a robust method to directly recognize defects and extract their structural features from various traditional and nontraditional nematic systems remains challenging to date. Here we present a machine learning solution, termed Machine Eye for Defects (MED), for automated defect analysis in images with diverse nematic textures. MED seamlessly integrates state-of-the-art object detection networks, Segment Anything Model, and vision transformer algorithms with tailored computer vision techniques. We show that MED can accurately identify the positions, winding numbers, and orientations of ±1/2 defects across distinct cellular contours, sparse vector fields of nematic directors, actin filaments, microtubules, and simulation images of Gay–Berne particles. MED performs faster than conventional defect detection method and can achieve over 90% accuracy on recognizing ±1/2 defects and their orientations from vector fields and experimental tissue images. We further demonstrate that MED can identify defect types that are not included in the training data, such as giant-core defects and defects with higher winding number. Remarkably, MED can provide correct structural information about ±1 defects. As such, MED stands poised to transform studies of diverse ordered systems by providing automated, rapid, accurate, and insightful defect analysis.

Repository Organization

Trained Models.zip

This directory is integral for model deployment and houses all relevant pre-trained models.

plus_vit_vecUV.pt: Pre-trained model for the Plus Transformer variant.

minus_vit_theR.pt: Pre-trained model for the Minus Transformer variant.

nanodet-plus-m_416-halfenhance: A sub-directory containing all files associated with the trained Nanodet-Plus model.

configs: Configuration files for training procedures.

Training Data.zip

This directory contains all datasets used for the training of Nanodet-Plus, Plus Transformer, and Minus Transformer models.

Code.zip

This directory features the implementation details and example use-cases showcased in Figure 2 and Figure 3c of our associated paper. The directory also includes code corresponding to the specific versions of Nanodet-Plus and SAM models cited in our study.

nanodet: Code in this folder is adapted from RangiLyu/nanodet (https://github.com/RangiLyu/nanodet). We have included the exact version used for compatibility.

segment_anything: Code sourced from Facebook Research's segment-anything (https://github.com/facebookresearch/segment-anything). The specific version used is included for compatibility.

Fig2: Code for predicting topological defects in tissue cell images, citing the following reference: T. B. Saw et al., Nature 544, 212 (2017).

Fig3c: Code for predicting topological defects in microtubules images, citing the following reference: M. Golden et al., Sci. Adv. 9, eabq6120 (2023).

Initialization Steps

Before executing any code, please ensure the following:

All files in the Trained Models directory must be available.

Download the checkpoint sam_vit_l_0b3195.pth from Facebook Research's segment-anything. (https://github.com/facebookresearch/segment-anything)

Acknowledgments

RangiLyu/nanodet (https://github.com/RangiLyu/nanodet)

Facebook Research's segment-anything (https://github.com/facebookresearch/segment-anything)

For further inquiries or issue reporting, you may contact us via email.

Contact Information: hrenae@connect.ust.hk

Doodleverse/Segmentation Zoo/Seg2Map Res-UNet models for segmentation of buildings of RGB 1024x1024 high-res. images

Models have been created using Segmentation Gym* using the following dataset**: https://github.com/FrontierDevelopmentLab/multi3net

These Residual-UNet model data are based on 1m spatial footprint images and associated labels of buildings in Houston. Imagery made available through DigitalGlobe***

Image size used by model: 1024 x 1024 x 3 pixels

classes:
other
building

File descriptions

For each model, there are 5 files with the same root name:

1. '.json' config file: this is the file that was used by Segmentation Gym* to create the weights file. It contains instructions for how to make the model and the data it used, as well as instructions for how to use the model for prediction. It is a handy wee thing and mastering it means mastering the entire Doodleverse.

2. '.h5' weights file: this is the file that was created by the Segmentation Gym* function `train_model.py`. It contains the trained model's parameter weights. It can called by the Segmentation Gym* function `seg_images_in_folder.py`. Models may be ensembled.

3. '_modelcard.json' model card file: this is a json file containing fields that collectively describe the model origins, training choices, and dataset that the model is based upon. There is some redundancy between this file and the `config` file (described above) that contains the instructions for the model training and implementation. The model card file is not used by the program but is important metadata so it is important to keep with the other files that collectively make the model and is such is considered part of the model

4. '_model_history.npz' model training history file: this numpy archive file contains numpy arrays describing the training and validation losses and metrics. It is created by the Segmentation Gym function `train_model.py`

5. '.png' model training loss and mean IoU plot: this png file contains plots of training and validation losses and mean IoU scores during model training. A subset of data inside the .npz file. It is created by the Segmentation Gym function `train_model.py`

Additionally, BEST_MODEL.txt contains the name of the model with the best validation loss and mean IoU

References
*Segmentation Gym: Buscombe, D., & Goldstein, E. B. (2022). A reproducible and reusable pipeline for segmentation of geoscientific imagery. Earth and Space Science, 9, e2022EA002332. https://doi.org/10.1029/2022EA002332 See: https://github.com/Doodleverse/segmentation_gym

**Rudner, T. G. J.; Rußwurm, M.; Fil, J.; Pelich, R.; Bischke, B.; Kopačková, V.; Biliński, P. Segmenting Flooded Buildings via Fusion of Multiresolution, Multisensor, and Multitemporal Satellite Imagery. In AAAI 2019. https://arxiv.org/pdf/1812.01756.pdf

***DigitalGlobe. 2018. DigitalGlobe Open Data Program. https://www.digitalglobe.com/opendata. Online; accessed 2018-09-01.

# Cape Hatteras Landsat8 RGB Images and Labels for Image Segmentation using the program, Segmentation Gym

## Overview
* Test datasets and files for testing the [segmentation gym](https://github.com/Doodleverse/segmentation_gym) program for image segmentation
* Data set made by Daniel Buscombe, Marda Science LLC.
* Dataset consists of a time-series of Landsat-8 images of Cape Hatteras National Seashore, courtesy of the U.S. Geological Survey.
* Imagery spans the period February 2015 to September 2021.
* Labels were created by Daniel Buscombe, Marda Science, using the labeling program [Doodler](https://github.com/Doodleverse/dash_doodler).

Download this file and unzip to somewhere on your machine (although *not* inside the `segmentation_gym` folder), then see the relevant page on the [segmentation gym wiki](https://github.com/Doodleverse/segmentation_gym/wiki) for further explanation.

## file structure

```{sh}
/Users/Someone/my_segmentation_zoo_datasets
│ ├── config
│ | └── *.json
│ ├── capehatteras_data
| | ├── fromDoodler
| | | ├──images
│ | | └──labels
| | ├──npzForModel
│ | └──toPredict
│ └── modelOut
│ └── *.png
│ └── weights
│ └── *.h5

```

## config
There are 3 config files:
1. `/config/hatteras_l8_resunet.json`
2. `/config/hatteras_l8_vanilla_unet.json`
3. `/config/hatteras_l8_resunet_model2.json`

The first two are for res-unet and unet models respectively. The last one differs from the first only with specification of kernel size. It is provided as an example of how to conduct model training experiments, modifying one hyperparameter at a time in the effort to create an optimal model.

They all contain the same essential information and differ as indicated below

```
{
"TARGET_SIZE": [768,768], # the size of the imagery you wish the model to train on. This may not be the original size
"MODEL": "resunet", # model name. Otherwise, "unet"
"NCLASSES": 4, # number of classes
"KERNEL":9, # horizontal size of convolution kernel in pixels
"STRIDE":2, # stride in convolution kernel
"BATCH_SIZE": 7, # number of images/labels per batch
"FILTERS":6, # number of filters
"N_DATA_BANDS": 3, # number of image bands
"DROPOUT":0.1, # amount of dropout
"DROPOUT_CHANGE_PER_LAYER":0.0, # change in dropout per layer
"DROPOUT_TYPE":"standard", # type of dropout. Otherwise "spatial"
"USE_DROPOUT_ON_UPSAMPLING":false, # if true, dropout is used on upsampling as well as downsampling
"DO_TRAIN": false, # if false, the model will not train, but you will select this config file, data directory, and the program will load the model weights and test the model on the validation subset
if true, the model will train from scratch (warning! this will overwrite the existing weights file in h5 format)
"LOSS":"dice", # model training loss function, otherwise "cat" for categorical cross-entropy
"PATIENCE": 10, # number of epochs of no model improvement before training is aborted
"MAX_EPOCHS": 100, # maximum number of training epochs
"VALIDATION_SPLIT": 0.6, #proportion to use for validation
"RAMPUP_EPOCHS": 20, # [LR-scheduler] rampup to maximim
"SUSTAIN_EPOCHS": 0.0, # [LR-scheduler] sustain at maximum
"EXP_DECAY": 0.9, # [LR-scheduler] decay rate
"START_LR": 1e-7, # [LR-scheduler] start lr
"MIN_LR": 1e-7, # [LR-scheduler] min lr
"MAX_LR": 1e-4, # [LR-scheduler] max lr
"FILTER_VALUE": 0, #if >0, the size of a median filter to apply on outputs (not recommended unless you have noisy outputs)
"DOPLOT": true, #make plots
"ROOT_STRING": "hatteras_l8_aug_768", #data file (npz) prefix string
"USEMASK": false, # use the convention 'mask' in label image file names, instead of the preferred 'label'
"AUG_ROT": 5, # [augmentation] amount of rotation in degrees
"AUG_ZOOM": 0.05, # [augmentation] amount of zoom as a proportion
"AUG_WIDTHSHIFT": 0.05, # [augmentation] amount of random width shift as a proportion
"AUG_HEIGHTSHIFT": 0.05,# [augmentation] amount of random width shift as a proportion
"AUG_HFLIP": true, # [augmentation] if true, randomly apply horizontal flips
"AUG_VFLIP": false, # [augmentation] if true, randomly apply vertical flips
"AUG_LOOPS": 10, #[augmentation] number of portions to split the data into (recommended > 2 to save memory)
"AUG_COPIES": 5 #[augmentation] number iof augmented copies to make
"SET_GPU": "0" #which GPU to use. If multiple, list separated by a comma, e.g. '0,1,2'. If CPU is requested, use "-1"
"WRITE_MODELMETADATA": false, #if true, the prompts `seg_images_in_folder.py` to write detailed metadata for each sample file
"do_crf": true #if true, apply CRF post-processing to outputs
}
```

## capehatteras_data
Folder containing all the model input data

```{sh}
│ ├── capehatteras_data: folder containing all the model input data
| | ├── fromDoodler: folder containing images and labels exported from Doodler using [this program](https://github.com/dbuscombe-usgs/dash_doodler/blob/main/utils/gen_images_and_labels_4_zoo.py)
| | | ├──images: jpg format files, one per label image
│ | | └──labels: jpg format files, one per image
| | ├──npzForModel: npz format files for model training using [this program](https://github.com/dbuscombe-usgs/segmentation_zoo/blob/main/train_model.py) that have been created following the workflow [documented here](https://github.com/dbuscombe-usgs/segmentation_zoo/wiki/Create-a-model-ready-dataset) using [this program](https://github.com/dbuscombe-usgs/segmentation_zoo/blob/main/make_nd_dataset.py)
│ | └──toPredict: a folder of images to test model prediction using [this program](https://github.com/dbuscombe-usgs/segmentation_zoo/blob/main/seg_images_in_folder.py)
```

## modelOut
PNG format files containing example model outputs from the train ('_train_' in filename) and validation ('_val_' in filename) subsets as well as an image showing training loss and accuracy curves with `trainhist` in the filename. There are two sets of these files, those associated with the residual unet trained with dice loss contain `resunet` in their name, and those from the UNet are named with `vanilla_unet`.

## weights
There are model weights files associated with each config files.

The Imaging Data Commons (IDC)(https://imaging.datacommons.cancer.gov/) [1] connects researchers with publicly available cancer imaging data, often linked with other types of cancer data. Many of the collections have limited annotations due to the expense and effort required to create these manually. The increased capabilities of AI analysis of radiology images provide an opportunity to augment existing IDC collections with new annotation data. To further this goal, we trained several nnUNet [2] based models for a variety of radiology segmentation tasks from public datasets and used them to generate segmentations for IDC collections.

To validate the model's performance, roughly 10% of the AI predictions were assigned to a validation set. For this set, a board-certified radiologist graded the quality of AI predictions on a Likert scale. If they did not 'strongly agree' with the AI output, the reviewer corrected the segmentation.

This record provides the AI segmentations, Manually corrected segmentations, and Manual scores for the inspected IDC Collection images.

Only 10% of the AI-derived annotations provided in this dataset are verified by expert radiologists . More details, on model training and annotations are provided within the associated manuscript to ensure transparency and reproducibility.

This work was done in two stages. Versions 1.x of this record were from the first stage. Versions 2.x added additional records. In the Version 1.x collections, a medical student (non-expert) reviewed all the AI predictions and rated them on a 5-point Likert Scale, for any AI predictions in the validation set that they did not 'strongly agree' with, the non-expert provided corrected segmentations. This non-expert was not utilized for the Version 2.x additional records.

Likert Score Definition:

Guidelines for reviewers to grade the quality of AI segmentations.

• 5 Strongly Agree - Use-as-is (i.e., clinically acceptable, and could be used for treatment without change)
• 4 Agree - Minor edits that are not necessary. Stylistic differences, but not clinically important. The current segmentation is acceptable
• 3 Neither agree nor disagree - Minor edits that are necessary. Minor edits are those that the review judges can be made in less time than starting from scratch or are expected to have minimal effect on treatment outcome
• 2 Disagree - Major edits. This category indicates that the necessary edit is required to ensure correctness, and sufficiently significant that user would prefer to start from the scratch
• 1 Strongly disagree - Unusable. This category indicates that the quality of the automatic annotations is so bad that they are unusable.

Zip File Folder Structure

Each zip file in the collection correlates to a specific segmentation task. The common folder structure is

• ai-segmentations-dcm This directory contains the AI model predictions in DICOM-SEG format for all analyzed IDC collection files
• qa-segmentations-dcm This directory contains manual corrected segmentation files, based on the AI prediction, in DICOM-SEG format. Only a fraction, ~10%, of the AI predictions were corrected. Corrections were performed by radiologist (rad*) and non-experts (ne*)
• qa-results.csv CSV file linking the study/series UIDs with the ai segmentation file, radiologist corrected segmentation file, radiologist ratings of AI performance.

qa-results.csv Columns

The qa-results.csv file contains metadata about the segmentations, their related IDC case image, as well as the Likert ratings and comments by the reviewers.

File Overview

brain-mr.zip

• Segment Description: brain tumor regions: necrosis, edema, enhancing
• IDC Collection: UPENN-GBM
• Links: model weights, github

breast-fdg-pet-ct.zip

• Segment Description: FDG-avid lesions in breast from FDG PET/CT scans QIN-Breast
• IDC Collection: QIN-Breast
• Links: model weights, github

breast-mr.zip

• Segment Description: Breast, Fibroglandular tissue, structural tumor
• IDC Collection: duke-breast-cancer-mri
• Links: model weights, github

kidney-ct.zip

• Segment Description: Kidney, Tumor, and Cysts from contrast enhanced CT scans
• IDS Collection: TCGA-KIRC, TCGA-KIRP, TCGA-KICH, CPTAC-CCRCC
• Links: model weights, github

liver-ct.zip

• Segment Description: Liver from CT scans
• IDC Collection: TCGA-LIHC
• Links: model weights, github

liver2-ct.zip

• Segment Description: Liver and Lesions from CT scans
• IDC Collection: HCC-TACE-SEG, COLORECTAL-LIVER-METASTASES
• Links: model weights, github

liver-mr.zip

• Segment Description: Liver from T1 MRI scans
• IDC Collection: TCGA-LIHC
• Links: model weights, github

lung-ct.zip

• Segment Description: Lung and Nodules (3mm-30mm) from CT scans
• IDC Collections:
  
  • Anti-PD-1-Lung
  • LUNG-PET-CT-Dx
  • NSCLC Radiogenomics
  • <a

This dataset contributes volumetric segmentations of the anatomic regions in a subset of CT images available from NCI Imaging Data Commons [1] (https://imaging.datacommons.cancer.gov/) automatically generated using the TotalSegmentation model v1.5.6 [2]. The initial release includes segmentations for the majority of the CT scans included in the National Lung Screening Trial (NLST) collection [3], [4] already available in IDC. Direct link to open this analysis result dataset in IDC (available after release of IDC v18): https://portal.imaging.datacommons.cancer.gov/explore/filters/?analysis_results_id=TotalSegmentator-CT-Segmentations.

Specifically, for each of the CT series analyzed, we include segmentations as generated by TotalSegmentator, converted into DICOM Segmentation object format using dcmqi v1.3.0 [5], and first order and shape features for each of the segmented regions, as produced by pyradiomics v3.0.1 [6]. Radiomics features were converted to DICOM Structured Reporting documents following template TID1500 using dcmqi. TotalSegmentator analysis on the NLST cohort was executed using Terra platform [7]. Implementation of the workflow that was used for performing the analysis is available at https://github.com/ImagingDataCommons/CloudSegmentator [8].

Due to the large size of the files, they are stored in the cloud buckets maintained by IDC, and the attached files are the manifests that can be used to download the actual files.

If you use the files referenced in the attached manifests, we ask you to cite this dataset and the preprint describing how it was generated [9].

Download instructions

Each of the manifests include instructions in the header on how to download the included files.

To download the TotalSegmentator segmentations (in DICOM SEG format) and pyradiomics measurements (in DICOM SR format) files using .s5cmd manifests:

1. install idc-index package: pip install --upgrade idc-index
2. download the files referenced by manifests included in this dataset by passing the .s5cmd manifest file. E.g., idc download totalsegmentator_ct_segmentations_aws.s5cmd

Other files included in the record are:

1. firstorder and shape radiomics features extracted using pyradiomics, and organized one file per segmented structure (see README file in the zip file for details on how those are organized)
   1. pyradiomics_features_csv.zip: saved in CSV format
   2. pyradiomics_features_parquet.zip: saved in Parquet format

Support

If you have any questions about this dataset, or if you experience any issues, please reach out to Imaging Data Commons support via support@canceridc.dev or (preferred) IDC Forum at https://discourse.canceridc.dev.

Long-RVOS: A Comprehensive Benchmark for Long-term Referring Video Object Segmentation

GitHub Repository: https://github.com/iSEE-Laboratory/Long_RVOS Project Page: https://isee-laboratory.github.io/Long-RVOS/ Paper: arXiv:2505.12702

  Dataset Description







  Dataset Summary

Long-RVOS is the first large-scale long-term referring video object segmentation benchmark, containing 2,000+ videos with an average duration exceeding 60 seconds. The dataset addresses… See the full description on the dataset page: https://huggingface.co/datasets/iSEE-Laboratory/Long-RVOS.