A benchmark dataset for out-of-distribution detection. ImageNet-1k is in-distribution, while iNaturalist is out-of-distribution.

The Tiny ImageNet Dataset is a dataset of 100,000 tiny (64x64) images of objects. It is a popular dataset for image classification and object detection research. The dataset consists of 200 different classes, each of which has 500 images.

The Stylized-ImageNet dataset is created by removing local texture cues in ImageNet while retaining global shape information on natural images via AdaIN style transfer. This nudges CNNs towards learning more about shapes and less about local textures.

The NINCO (No ImageNet Class Objects) dataset is introduced in the ICML 2023 paper In or Out? Fixing ImageNet Out-of-Distribution Detection Evaluation. The images in this dataset are free from objects that belong to any of the 1000 classes of ImageNet-1K (ILSVRC2012), which makes NINCO suitable for evaluating out-of-distribution detection on ImageNet-1K .

The NINCO main dataset consists of 64 OOD classes with a total of 5879 samples. These OOD classes were selected to have no categorical overlap with any classes of ImageNet-1K. Each sample was inspected individually by the authors to not contain ID objects.

Besides NINCO, included are (in the same .tar.gz file) truly OOD versions of 11 popular OOD datasets with in total 2715 OOD samples.

Further included are 17 OOD unit-tests, with 400 samples each.

Code for loading and evaluating on each of the three datasets is provided at https://github.com/j-cb/NINCO.

When using NINCO, please consider citing (besides the bibtex given below) the following data sources that were used to create NINCO:

Hendrycks et al.: ”Scaling out-of-distribution detection for real-world settings”, ICML, 2022.
Bossard et al.: ”Food-101 – mining discriminative components with random forests”, ECCV 2014.
Zhou et al.: ”Places: A 10 million image database for scene recognition”, IEEE PAMI 2017.
Huang et al.: ”Mos: Towards scaling out-of-distribution detection for large semantic space”, CVPR 2021.
Li et al.: ”Caltech 101 (1.0)”, 2022. Ismail et al.: ”MYNursingHome: A fully-labelled image dataset for indoor object classification.”, Data in Brief (V. 32) 2020. The iNaturalist project: https://www.inaturalist.org/

When using NINCO_popular_datasets_subsamples, additionally to the above, please consider citing:

Cimpoi et al.: ”Describing textures in the wild”, CVPR 2014.
Hendrycks et al.: ”Natural adversarial examples”, CVPR 2021.
Wang et al.: ”Vim: Out-of-distribution with virtual-logit matching”, CVPR 2022.
Bendale et al.: ”Towards Open Set Deep Networks”, CVPR 2016.
Vaze et al.: ”Open-set Recognition: a Good Closed-set Classifier is All You Need?”, ICLR 2022.
Wang et al.: ”Partial and Asymmetric Contrastive Learning for Out-of-Distribution Detection in Long-Tailed Recognition.” ICML, 2022.
Galil et al.: “A framework for benchmarking Class-out-of-distribution detection and its application to ImageNet”, ICLR 2023.

This is a subset of ImageNet called "ImageNet16" more suited for cases with limited computational budget and faster experimentation.

Each class has 400 train images and 100 test images.

• Credit also goes to original creators that constructed the dataset. Unfortunately, I was not able to relocated it online so I reupload it here.

If used in your work please cite as follows:

C. Kyrkou, "Toward Efficient Convolutional Neural Networks With Structured Ternary Patterns," in IEEE Transactions on Neural Networks and Learning Systems, doi: 10.1109/TNNLS.2024.3380827.

The classes corresponding to imagenet1K:

• n02009912 American_egret

• n02113624 toy_poodle

• n02123597 Siamese_cat

• n02132136 brown_bear

• n02504458 African_elephant

• n02690373 airliner

• n02835271 bicycle-built-for-two

• n02951358 canoe

• n03041632 cleaver

• n03085013 computer_keyboard

• n03196217 digital_clock

• n03977966 police_van

• n04099969 rocking_chair

• n04111531 rotisserie

• n04285008 sports_car

• n04591713 wine_bottle

From original map.txt

knife = n03041632

keyboard = n03085013

elephant = n02504458

bicycle = n02835271

airplane = n02690373

clock = n03196217

oven = n04111531

chair = n04099969

bear = n02132136

boat = n02951358

cat = n02123597

bottle = n04591713

truck = n03977966

car = n04285008

bird = n02009912

dog = n02113624

Folder Structure

-

--

 --- <filename1>.JPEG

 --- <filename2>.JPEG

 --- ....

--

--...

-

--

 --- <filename1>.JPEG

 --- <filename2>.JPEG

 --- ....

--

--...

Some preliminary results:

Model Name Accuracy (Top-1)

VGG16 85.3

ResNet50 88.2

MobileNetV2 91.0

EfficientNet B0 85.6

Massive Credit to original ImageNet authors[1] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg and Li Fei-Fei.ImageNet Large Scale Visual Recognition Challenge. IJCV, 2015

ImageNet is a large-scale visual database widely used in the field of computer vision, especially for object recognition tasks. It contains millions of labeled images, organized into multiple categories, and is used for training and evaluating image classification models. ImageNet datasets are widely used for training deep learning models, particularly Convolutional Neural Networks (CNNs). ILSVRC2012 (ImageNet Large Scale Visual Recognition Challenge 2012) is a part of ImageNet and is a competition for image classification and object detection. In ILSVRC2012, the dataset includes over 1000 categories with more than 1 million images. The goal of ILSVRC2012 is to evaluate the performance of different models in image classification and object recognition tasks, and it significantly contributed to the development of modern deep learning architectures. This competition's success helped accelerate the widespread use of deep neural networks, especially Convolutional Neural Networks (CNNs).Data availability and access.} The dataset used in this study is available at https://huggingface.co/datasets/ILSVRC/imagenet-1k, https://image-net.org/challenges/LSVRC/2012

Dataset Summary

This is a copy of the full Winter21 release of ImageNet in webdataset tar format with WEBP encoded images. This release consists of 19167 classes, 2674 fewer classes than the original 21841 class Fall11 release of the full ImageNet. The classes were removed due to these concerns: https://www.image-net.org/update-sep-17-2019.php This is the same contents as https://huggingface.co/datasets/timm/imagenet-w21-wds but encoded in webp at ~56% of the size, shard count… See the full description on the dataset page: https://huggingface.co/datasets/timm/imagenet-w21-webp-wds.

ImageNet-LT is a subset of original ImageNet ILSVRC 2012 dataset. The training set is subsampled such that the number of images per class follows a long-tailed distribution. The class with the maximum number of images contains 1,280 examples, whereas the class with the minumum number of images contains only 5 examples. The dataset also has a balanced validation set, which is also a subset of the ImageNet ILSVRC 2012 training set and contains 20 images per class. The test set of this dataset is the same as the validation set of the original ImageNet ILSVRC 2012 dataset.

The original ImageNet ILSVRC 2012 dataset must be downloaded manually, and its path should be set with --manual_dir in order to generate this dataset.

To use this dataset:

import tensorflow_datasets as tfds

ds = tfds.load('imagenet_lt', split='train')
for ex in ds.take(4):
 print(ex)

See the guide for more informations on tensorflow_datasets.

https://storage.googleapis.com/tfds-data/visualization/fig/imagenet_lt-1.0.0.png" alt="Visualization" width="500px">

Dataset Summary

ILSVRC 2012, commonly known as 'ImageNet' is an image dataset organized according to the WordNet hierarchy. Each meaningful concept in WordNet, possibly described by multiple words or word phrases, is called a "synonym set" or "synset". There are more than 100,000 synsets in WordNet, majority of them are nouns (80,000+). ImageNet aims to provide on average 1000 images to illustrate each synset. Images of each concept are quality-controlled and human-annotated. 💡… See the full description on the dataset page: https://huggingface.co/datasets/timm/imagenet-1k-wds.

ImageNet-P consists of noise, blur, weather, and digital distortions. The dataset has validation perturbations; has difficulty levels; has CIFAR-10, Tiny ImageNet, ImageNet 64 × 64, standard, and Inception-sized editions; and has been designed for benchmarking not training networks. ImageNet-P departs from ImageNet-C by having perturbation sequences generated from each ImageNet validation image. Each sequence contains more than 30 frames, so to counteract an increase in dataset size and evaluation time only 10 common perturbations are used.

Imagenette is a subset of 10 easily classified classes from the Imagenet dataset. It was originally prepared by Jeremy Howard of FastAI. The objective behind putting together a small version of the Imagenet dataset was mainly because running new ideas/algorithms/experiments on the whole Imagenet take a lot of time.

This version of the dataset allows researchers/practitioners to quickly try out ideas and share with others. The dataset comes in three variants:

• Full size
• 320 px
• 160 px

Note: The v2 config correspond to the new 70/30 train/valid split (released in Dec 6 2019).

To use this dataset:

import tensorflow_datasets as tfds

ds = tfds.load('imagenette', split='train')
for ex in ds.take(4):
 print(ex)

See the guide for more informations on tensorflow_datasets.

https://storage.googleapis.com/tfds-data/visualization/fig/imagenette-full-size-v2-1.0.0.png" alt="Visualization" width="500px">

Benchmarking the robustness to distribution shifts traditionally relies on dataset collection which is typically laborious and expensive, in particular for datasets with a large number of classes like ImageNet. An exception to this procedure is ImageNet-C (Hendrycks & Dietterich, 2019), a dataset created by applying common real-world corruptions at different levels of intensity to the (clean) ImageNet images. Inspired by this work, we introduce ImageNet-Cartoon and ImageNet-Drawing, two datasets constructed by converting ImageNet images into cartoons and colored pencil drawings, using a GAN framework (Wang & Yu, 2020) and simple image processing (Lu et al., 2012), respectively.

This repository contains ImageNet-Cartoon and ImageNet-Drawing. Checkout the official GitHub Repo for the code on how to reproduce the datasets.

If you find this useful in your research, please consider citing:

@inproceedings{imagenetshift,
 title={ImageNet-Cartoon and ImageNet-Drawing: two domain shift datasets for ImageNet},
 author={Tiago Salvador and Adam M. Oberman},
 booktitle={ICML Workshop on Shift happens: Crowdsourcing metrics and test datasets beyond ImageNet.},
 year={2022}
}

ImageNet-R is a set of images labelled with ImageNet labels that were obtained by collecting art, cartoons, deviantart, graffiti, embroidery, graphics, origami, paintings, patterns, plastic objects, plush objects, sculptures, sketches, tattoos, toys, and video game renditions of ImageNet classes. ImageNet-R has renditions of 200 ImageNet classes resulting in 30,000 images. by collecting new data and keeping only those images that ResNet-50 models fail to correctly classify. For more details please refer to the paper.

The label space is the same as that of ImageNet2012. Each example is represented as a dictionary with the following keys:

• 'image': The image, a (H, W, 3)-tensor.
• 'label': An integer in the range [0, 1000).
• 'file_name': A unique sting identifying the example within the dataset.

To use this dataset:

import tensorflow_datasets as tfds

ds = tfds.load('imagenet_r', split='train')
for ex in ds.take(4):
 print(ex)

See the guide for more informations on tensorflow_datasets.

https://storage.googleapis.com/tfds-data/visualization/fig/imagenet_r-0.2.0.png" alt="Visualization" width="500px">

ImageNet-Sketch data set consists of 50,889 images, approximately 50 images for each of the 1000 ImageNet classes. The data set is constructed with Google Image queries "sketch of ", where is the standard class name. Only within the "black and white" color scheme is searched. 100 images are initially queried for every class, and the pulled images are cleaned by deleting the irrelevant images and images that are for similar but different classes. For some classes, there are less than 50 images after manually cleaning, and then the data set is augmented by flipping and rotating the images.

Link to original evaluation code for: https://github.com/hendrycks/natural-adv-examples @article{hendrycks2021nae, title={Natural Adversarial Examples}, author={Dan Hendrycks and Kevin Zhao and Steven Basart and Jacob Steinhardt and Dawn Song}, journal={CVPR}, year={2021} }

ImageNet-PI is a relabelled version of the standard ILSVRC2012 ImageNet dataset in which the labels are provided by a collection of 16 deep neural networks with different architectures pre-trained on the standard ILSVRC2012. Specifically, the pre-trained models are downloaded from tf.keras.applications.

In addition to the new labels, ImageNet-PI also provides meta-data about the annotation process in the form of confidences of the models on their labels and additional information about each model.

For more information see: ImageNet-PI

To use this dataset:

import tensorflow_datasets as tfds

ds = tfds.load('imagenet_pi', split='train')
for ex in ds.take(4):
 print(ex)

See the guide for more informations on tensorflow_datasets.

The dataset used in the paper is CIFAR-10, STL-10, and ImageNet.

Repack Information

This repository contains a complete repack of ILSVRC/imagenet-1k in Parquet format with the following data transformations:

Images were center-cropped to square to the minimum height/width dimension. Images were then rescaled to 256x256 using Lanczos resampling. This dataset is available at benjamin-paine/imagenet-1k-256x256 Images were then rescaled to 128x128 using Lanczos resampling. This dataset is available at benjamin-paine/imagenet-1k-128x128. Images… See the full description on the dataset page: https://huggingface.co/datasets/benjamin-paine/imagenet-1k-64x64.

Dataset

This dataset was created by Giba

Released under CC0: Public Domain

Contents

Dataset

This dataset was created by João Cardoso

Released under Other (specified in description)

Contents