RSICD Image Caption Dataset

RSICD Image Caption Dataset

By Arto (From Huggingface) [source]

About this dataset

The train.csv file contains a list of image filenames, captions, and the actual images used for training the image captioning models. Similarly, the test.csv file includes a separate set of image filenames, captions, and images specifically designated for testing the accuracy and performance of the trained models.

Furthermore, the valid.csv file contains a unique collection of image filenames with their respective captions and images that serve as an independent validation set to evaluate the models' capabilities accurately.

Each entry in these CSV files includes both a filename string that indicates the name or identifier of an image file stored in another location or directory. Additionally,** each entry also provides a list (or multiple rows) o**f strings representing written descriptions or captions describing each respective image given its filename.

Considering these details about this dataset's structure, it can be immensely valuable to researchers, developers, and enthusiasts working on developing innovative computer vision algorithms such as automatic text generation based on visual content analysis. Whether it's training machine learning models to automatically generate relevant captions based on new unseen images or evaluating existing systems' performance against diverse criteria.

Stay updated with cutting-edge research trends by leveraging this comprehensive dataset containing not only captio**ns but also corresponding imag**es across different sets specifically designed to cater to varied purposes within computer vision tasks. »

How to use the dataset

Overview of the Dataset

The dataset consists of three primary files: train.csv, test.csv, and valid.csv. These files contain information about image filenames and their respective captions. Each file includes multiple captions for each image to support diverse training techniques.

Understanding the Files

• train.csv: This file contains filenames (filename column) and their corresponding captions (captions column) for training your image captioning model.
• test.csv: The test set is included in this file, which contains a similar structure as that of train.csv. The purpose of this file is to evaluate your trained models on unseen data.
• valid.csv: This validation set provides images with their respective filenames (filename) and captions (captions). It allows you to fine-tune your models based on performance during evaluation.

Getting Started

To begin utilizing this dataset effectively, follow these steps:

• Extract the zip file containing all relevant data files onto your local machine or cloud environment.
• Familiarize yourself with each CSV file's structure: train.csv, test.csv, and valid.csv. Understand how information like filename(s) (filename) corresponds with its respective caption(s) (captions).
• Depending on your specific use case or research goals, determine which portion(s) of the dataset you wish to work with (e.g., only train or train+validation).
• Load the dataset into your preferred programming environment or machine learning framework, ensuring you have the necessary dependencies installed.
• Preprocess the dataset as needed, such as resizing images to a specific dimension or encoding captions for model training purposes.
• Split the data into training, validation, and test sets according to your experimental design requirements.
• Use appropriate algorithms and techniques to train your image captioning models on the provided data.

Enhancing Model Performance

To optimize model performance using this dataset, consider these tips:

• Explore different architectures and pre-trained models specifically designed for image captioning tasks.
• Experiment with various natural language

Research Ideas

• Image Captioning: This dataset can be used to train and evaluate image captioning models. The captions can be used as target labels for training, and the images can be paired with the captions to generate descriptive captions for test images.
• Image Retrieval: The dataset can be used for image retrieval tasks where given a query caption, the model needs to retrieve the images that best match the description. This can be useful in applications such as content-based image search.
• Natural Language Processing: The dataset can also be used for natural language processing tasks such as text generation or machine translation. The captions in this dataset are descriptive ...

Dataset

This dataset was created by Rahul Verma

Contents

This benchmark data consists of a small set of actual human feedback on figure-caption pairs, along with a large set of predicted human feedback for over 100k figure-caption pairs. Please see the readme for further details.

Face2Text is an ongoing project to collect a data set of natural language descriptions of human faces. A randomly selected sample of images from the CelebA data set were used and human annotators were given a random sample of faces to describe.

Incorporating medical text annotations compensates for the quality deficiencies of image data, effectively overcoming the limitations of medical image segmentation. Many existing approaches achieve high-quality segmentation results by integrating text into the image modality. However, these approaches require matched image-text pairs during inference to maintain their performance, and the absence of corresponding text annotations results in degraded model performance. Additionally, these methods often assume that the input text annotations are ideal, overlooking the impact of poor-quality text on model performance in practical scenarios. To address these issues, we propose a novel generative medical image segmentation model, Cap2Seg (Leveraging Caption Generation for Enhanced Segmentation of COVID-19 Medical Images). Cap2Seg not only segments lesion areas but also generates related medical text descriptions, guiding the segmentation process. This design enables the model to perform optimal segmentation without requiring text input during inference. To mitigate the impact of inaccurate text on model performance, we consider the consistency between generated textual features and visual features and introduce the Scale-aware Textual Attention Module (SATaM), which reduces the model’s dependency on irrelevant or misleading text information. Subsequently, we design a word-pixel fusion decoding mechanism that effectively integrates textual features into visual features, ensuring that the text information effectively supplements and enhances the image segmentation task. Extensive experiments on two public datasets, MosMedData+ and QaTa-COV19, demonstrate that our method outperforms the current state-of-the-art models under the same conditions. Additionally, ablation studies have been conducted to demonstrate the effectiveness of each proposed module. The code is available at https://github.com/AllenZzzzzzzz/Cap2Seg.

This benchmark data consists of a small set of actual human feedback on figure-caption pairs, along with a large set of predicted human feedback for over 100k figure-caption pairs. Please see the readme for further details.

Dataset

This dataset was created by shushruth17

Contents

Dataset

This dataset was created by MANISH KUMAR VISHWAKARMA

Released under MIT

Contents

The Abstractive News Captions with High-level cOntext Representation (ANCHOR) dataset contains 70K+ samples sourced from 5 different news media organizations. This dataset can be utilized for Vision & Language tasks such as Text-to-Image Generation, Image Caption Generation, etc.

Version 1.0, October 2024

Created by

Mithun Manivannan (1), Vignesh Nethrapalli (1), Mark Cartwright (1)

1. Sound Interaction and Computer Lab, New Jersey Institute of Technology

Publication

If using this data in an academic work, please reference the DOI and version, as well as cite the following paper, which presented the data collection procedure and the first version of the dataset:

Manivannan, M., Nethrapalli, V., Cartwright, M. EmotionCaps: Enhancing Audio Captioning Through Emotion-Augmented Data Generation. arXiv preprint arXiv:2410.12028, 2024.

Description

EmotionCaps is a ChatGPT-assisted, weakly-labeled audio captioning dataset developed to bridge the gap between soundscape emotion recognition (SER) and automated audio captioning (AAC). Created through a three-stage pipeline, the dataset leverages ground-truth annotations from AudioSet SL, which are enhanced by ChatGPT using tailored prompts and emotions assigned via a soundscape emotion recognition model trained on Emo-Soundscapes Dataset. It comprises four subsets of captions for 120,071 audio clips, each reflecting a different prompt variation: WavCaps-like, Scene-Focused, Emotion Addon, and Emotion Rewrite. The average word counts for these subsets are: WavCaps-like (12.61), Scene-Focused (14.04), Emotion Addon (18.35), and Emotion Rewrite (18.65). The increase in word count for the emotion prompts illustrates the difference in sentence length when integrating emotion information into the captions.

Audio Data

The audio data is from AudioSet SL, the strongly-labled subset of 120,071 audio clips from the larger AudioSet dataset.

Synthetic Captions

The synthetic captions were generated using a three-stage pipeline, beginning with training a soundscape emotion recognition model. This model assesses the valence and arousal of each audio clip, mapping the resulting vector to an emotion identifier. Next, we leveraged the ground-truth annotations from AudioSet SL, and extracted the list of sound events. Using these sound events, we employed ChatGPT to create different variations of captions by applying distinct prompts.

We first used the WavCaps prompt for AudioSet SL as a base, the output of which we call WavCaps-like. Building on this, we created three new prompt variations (1) scene-focused which is a modified WavCaps prompt that describes the scene, (2) emotion addon which is an extension of the scene-Focused prompt, where an emotion is appended to the list of sound events to guide the caption generation, and (3) emotion rewrite which consists of two-step prompt where ChatGPT first generates the scene-focused caption, then is instructed to rewrite it with a specific emotion in mind.

Using these four prompt styles — WavCaps, Scene-Focused, Emotion Addon, and Emotion Rewrite — along with the AudioSet SL sound events and predicted emotions, we employed ChatGPT-3.5 Turbo to generate four corresponding caption variations for the dataset.

Each caption variation has been organized into separate CSV files for clarity and accessibility. All files correspond to the same set of audio clips from AudioSet SL, with the key distinction being the caption variation associated with each clip. The different subsets are designed to be used independently, as they each fulfill specific roles in understanding the impact of emotion in audio captions.

• wavcaps-like.csv: Contains captions generated using the WavCaps prompt, serving as the baseline before emotion is introduced.

• scene-focused.csv: Provides captions focused on describing the scene or environment of the audio clip, without emotion integration.

• emotion-addon.csv: Captions where emotion data is appended to the scene-focused base caption.

• emotion-rewrite.csv: Captions that are completely rewritten based on the scene-focused base caption and the assigned emotion.

This structure allows users to explore how emotional content influences captioning models by comparing the variations both with and without emotional enrichment.

Columns in CSV files

segment_id : The ID of the audio recording in AudioSet SL. These are in the form

caption : The caption generated for each audio clip, corresponding to the specific subset (e.g., WavCaps, Scene-Focused, Emotion Addon, or Emotion Rewrite) as indicated by the file name.

Conditions of use

Dataset created by Mithun Manivannan, Vignesh Nethrapalli, Mark Cartwright

The EmotionCaps dataset is offered free of charge under the terms of the Creative Commons Attribution 4.0 International (CC BY 4.0) license:

https://creativecommons.org/licenses/by/4.0/

The dataset and its contents are made available on an “as is” basis and without warranties of any kind, including without limitation satisfactory quality and conformity, merchantability, fitness for a particular purpose, accuracy or completeness, or absence of errors. Subject to any liability that may not be excluded or limited by law, New Jersey Institute of Technology is not liable for, and expressly excludes all liability for, loss or damage however and whenever caused to anyone by any use of the EmotionCaps dataset or any part of it.

Feedback

Please help us improve EmotionCaps by sending your feedback to:

• Mithun Manivannan: mithun.mani01@gmail.com
• Mark Cartwright: mcartwright@gmail.com

In case of a problem, please include as many details as possible.

Acknowledgments

This work was partially supported by the New Jersey Institute of Technology Honors Summer Research Institute (HSRI).

Image Caption Quality Dataset is a dataset of crowdsourced ratings for machine-generated image captions. It contains more than 600k ratings of image-caption pairs.

If you find this dataset useful, please drop a like!! Thank you :)

Content

Training, Test, Val splits

All images are contained in Flickr8k_Dataset. Data splits and annotations are included in Flickr8k_text.

From readme.txt

Flickr8k.token.txt - the raw captions of the Flickr8k Dataset . The first column is the ID of the caption which is "image address # caption number"

Flickr8k.lemma.txt - the lemmatized version of the above captions

Flickr_8k.trainImages.txt - The training images used in our experiments

Flickr_8k.devImages.txt - The development/validation images used in our experiments

Flickr_8k.testImages.txt - The test images used in our experiments

ExpertAnnotations.txt is the expert judgments. The first two columns are the image and caption IDs. Caption IDs are #<0-4>. The next three columns are the expert judgments for that image-caption pair. Scores range from 1 to 4, with a 1 indicating that the caption does not describe the image at all, a 2 indicating the caption describes minor aspects of the image but does not describe the image, a 3 indicating that the caption almost describes the image with minor mistakes, and a 4 indicating that the caption describes the image.

Acknowledgements

Original Authors: Cyrus Rashtchian, Peter Young, Micah Hodosh, and Julia Hockenmaier. Collecting Image Annotations Using Amazon's Mechanical Turk. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk.

Credits to Jason Brownlee for organizing original ZIP archive. (https://machinelearningmastery.com/develop-a-deep-learning-caption-generation-model-in-python/).

Lenkashell/safety-image-generation-captions-1 dataset hosted on Hugging Face and contributed by the HF Datasets community

Dataset Card for STAIR-Captions

  Dataset Summary

STAIR Captions is a large-scale dataset containing 820,310 Japanese captions. This dataset can be used for caption generation, multimodal retrieval, and image generation.

  Supported Tasks and Leaderboards

[More Information Needed]

  Languages

The language data in JDocQA is in Japanese (BCP-47 ja-JP).

  Dataset Structure





  Data Instances

[More Information Needed]… See the full description on the dataset page: https://huggingface.co/datasets/shunk031/STAIR-Captions.

Captioning Hardware Market Outlook

According to our latest research, the global captioning hardware market size reached USD 1.34 billion in 2024, driven by surging demand for accessible content and regulatory mandates across industries. The market is exhibiting robust momentum, with a CAGR of 7.2% anticipated from 2025 to 2033. By the end of the forecast period, the captioning hardware market is projected to attain a value of USD 2.52 billion by 2033. This growth is underpinned by increasing investments in digital broadcasting infrastructure, rising awareness around inclusivity, and rapid technological advancements in media delivery and accessibility solutions.

Several key factors are fueling the expansion of the captioning hardware market. One of the primary drivers is the global rise in regulatory requirements for content accessibility. Governments and regulatory bodies in North America, Europe, and Asia Pacific have implemented strict mandates that require broadcasters, educational institutions, and public organizations to provide accessible content for individuals with hearing impairments. The enforcement of laws such as the Americans with Disabilities Act (ADA) and the European Accessibility Act has compelled organizations to invest in advanced captioning hardware, ensuring compliance and avoiding hefty penalties. This regulatory landscape is expected to remain a significant growth catalyst throughout the forecast period as more countries adopt similar standards and expand their scope to include digital and online content.

Another significant growth factor is the exponential increase in digital media consumption across multiple platforms. The proliferation of streaming services, online video content, and live broadcasts has dramatically heightened the need for efficient and reliable captioning solutions. Captioning hardware, known for its real-time processing capabilities and high accuracy, is being rapidly adopted by broadcasters and content creators to cater to a diverse, global audience. Furthermore, the integration of artificial intelligence and machine learning technologies into captioning hardware is enhancing the quality and speed of caption generation, making it an indispensable tool for media companies aiming to stay competitive in a dynamic market. As digital transformation accelerates worldwide, the demand for robust captioning hardware is expected to surge further.

The captioning hardware market is also benefiting from the growing emphasis on inclusivity and corporate social responsibility. Organizations across sectors, including education, corporate, and government, are increasingly recognizing the importance of making their content accessible to all individuals. This cultural shift toward inclusivity is prompting investments in captioning hardware as a means to foster engagement, improve communication, and enhance learning outcomes. In the education sector, for instance, captioning hardware is being deployed in classrooms and lecture halls to support students with hearing impairments and facilitate remote learning. Similarly, corporations are utilizing captioning solutions for webinars, conferences, and training sessions to ensure all employees can participate effectively. This trend is expected to continue, amplifying market growth across various end-user segments.

Subtitling and Captioning have become integral components of the media landscape, especially as content consumption transcends geographical and linguistic boundaries. The distinction between the two lies in their application; while subtitling primarily caters to translating spoken dialogue into text for viewers who do not understand the language, captioning is more comprehensive, providing text for all audio elements, including sound effects and speaker identification. This dual approach not only enhances accessibility for audiences with hearing impairments but also broadens the reach of content to non-native speakers. As the demand for multilingual content continues to rise, the integration of subtitling and captioning technologies is becoming increasingly crucial for media companies aiming to engage global audiences effectively.

From a regional perspective, North America currently dominates the captioning hardware market, accounting for the largest revenue share in 2024. The region's leadership can be attributed to a mature media

Captions Generator for Shorts Market Outlook

According to our latest research, the global Captions Generator for Shorts market size reached USD 1.12 billion in 2024, demonstrating robust adoption across digital content platforms. The market is experiencing a strong compound annual growth rate (CAGR) of 19.6% from 2025 to 2033, fueled by the surging demand for automated content solutions. By 2033, the market is projected to reach USD 5.19 billion, reflecting the transformative impact of AI-driven captioning technologies and the proliferation of short-form video content on social media and marketing platforms worldwide.

The rapid expansion of the Captions Generator for Shorts market is primarily driven by the explosive growth of short-form video content, particularly on social media platforms such as TikTok, Instagram Reels, and YouTube Shorts. As consumers increasingly favor bite-sized, visually engaging content, content creators and brands are compelled to enhance accessibility and engagement through accurate and contextually relevant captions. The integration of advanced artificial intelligence and natural language processing technologies has significantly improved the efficiency and accuracy of automated captions, reducing manual effort and enabling real-time captioning at scale. This technological evolution is attracting a diverse range of users, from individual content creators to large enterprises seeking to optimize their digital communication strategies.

Another significant growth factor for the Captions Generator for Shorts market is the rising emphasis on inclusivity and compliance with accessibility regulations. Governments and organizations worldwide are enacting stringent guidelines to ensure digital content is accessible to all, including individuals with hearing impairments. This regulatory landscape is compelling businesses, educational institutions, and media companies to adopt automated captioning solutions, not only to avoid legal repercussions but also to broaden their audience reach. Furthermore, the ability to generate multilingual captions is facilitating global content distribution, allowing creators to tap into new markets and demographics with minimal localization costs.

The market is also benefiting from the increasing adoption of video marketing strategies by enterprises across various sectors. As video content continues to outperform other formats in terms of engagement and conversion rates, businesses are leveraging captions generators to enhance search engine optimization (SEO), improve viewer retention, and deliver clear messaging across diverse audiences. The proliferation of cloud-based deployment models is making these solutions more accessible and scalable, enabling organizations to integrate captioning capabilities seamlessly into their existing workflows. The convergence of AI, cloud computing, and multimedia content creation is expected to further accelerate market growth in the coming years.

Regionally, North America and Asia Pacific are emerging as dominant players in the Captions Generator for Shorts market, driven by high internet penetration, widespread adoption of social media, and the presence of leading technology providers. North America, in particular, is witnessing strong demand from both individual creators and enterprise clients, while Asia Pacific is experiencing rapid growth due to the increasing popularity of short-form video platforms and a burgeoning creator economy. Europe is also showing steady progress, supported by regulatory initiatives and growing awareness of digital accessibility. Latin America and the Middle East & Africa, though currently smaller markets, are expected to register notable growth rates as digital transformation initiatives gain momentum.

Component Analysis

The Component segment of the Captions Generator for Shorts market is bifurcated into software and services, each playing a distinct role in the market’s value chain. Software solutions, which encompass AI-driven captioning tools, plug-ins, and integrated platforms, account for the largest share of the market. These software offerings are increasingly being adopted due to their ability to deliver high-quality, real-time captions with minimal human intervention. The integration of machine learning and natural language processing algorithms has dramatically enhanced the accuracy and contextual relevance of generated captions, making them indispensable for content creators and enterprises alike. Addit

Captions Generator for Shorts Market Outlook

According to our latest research, the Global Captions Generator for Shorts market size was valued at $356 million in 2024 and is projected to reach $1.14 billion by 2033, expanding at a robust CAGR of 13.8% during the forecast period of 2025–2033. The primary growth driver for this market is the exponential rise in short-form video content consumption across social media platforms, which has necessitated the adoption of automated captioning tools for accessibility, engagement, and compliance. As content creators and enterprises increasingly prioritize inclusivity and global reach, the demand for advanced, AI-powered captions generators for shorts continues to surge, reshaping how digital media is produced and consumed worldwide.

Regional Outlook

North America commands the largest share of the global captions generator for shorts market, accounting for approximately 38% of the total market value in 2024. This dominance is attributed to the region’s mature digital infrastructure, widespread adoption of social media, and a highly active ecosystem of content creators and enterprises. The presence of leading technology developers and a strong focus on accessibility regulations, such as the Americans with Disabilities Act (ADA), have further catalyzed the adoption of captions generator solutions. Additionally, North America’s media and entertainment sector, which is consistently at the forefront of innovation, has embraced these tools to enhance viewer engagement, improve SEO, and comply with accessibility mandates. As a result, the region continues to witness steady investments in AI-driven video technologies, reinforcing its leadership position in the market.

The Asia Pacific region is emerging as the fastest-growing market for captions generator for shorts, projected to register a CAGR of 16.3% through 2033. This rapid expansion is fueled by the explosive growth of mobile internet usage, the proliferation of short-form video platforms such as TikTok and YouTube Shorts, and increasing digital literacy across countries like China, India, and Southeast Asia. Government initiatives to promote digital content creation, coupled with rising investments from global technology giants, have accelerated the adoption of automated captioning solutions. Furthermore, the region’s vast multilingual landscape has heightened the need for advanced, AI-powered caption generators capable of supporting multiple languages and dialects, thereby driving innovation and market penetration.

In emerging economies within Latin America and the Middle East & Africa, the market for captions generator for shorts is witnessing gradual adoption, primarily hindered by infrastructural limitations, lower digital penetration, and budget constraints among small and medium enterprises. However, localized demand is on the rise, particularly as regional content creators and educational institutions recognize the value of captions in expanding audience reach and improving accessibility. Policy reforms aimed at bridging the digital divide and enhancing media inclusivity are expected to gradually stimulate market growth. Nonetheless, challenges such as inconsistent regulatory frameworks and limited access to advanced AI technologies continue to impact the pace of adoption in these regions.

Report Scope

Overview: The Image and Text Pair Dataset is a curated collection of images paired with descriptive textual captions or subtitles. This dataset is designed to support various natural language processing and computer vision tasks, such as image captioning, text-to-image retrieval, and multimodal machine learning research. It serves as a valuable resource for training and evaluating models that can understand and generate meaningful relationships between visual content and textual descriptions.

Contents: The dataset consists of the following components:

Images: The dataset includes a set of image files in common formats such as JPEG or PNG. Each image captures a different scene, object, or concept. These images are diverse and cover a wide range of visual content.

Textual Captions or Subtitles: For each image, there is an associated textual caption or subtitle that describes the content of the image. These captions provide context, details, or descriptions of the visual elements in the images. The text data is in natural language and is designed to be human-readable.

Use Cases: The Image and Text Pair Dataset can be utilized for various machine learning and AI tasks, including but not limited to:

Image Captioning: Training and evaluating models to generate textual descriptions for given images. Text-to-Image Retrieval: Enabling models to retrieve images based on textual queries. Multimodal Learning: Supporting research in multimodal models that understand and bridge the gap between textual and visual data. Natural Language Processing: Serving as a source of textual data for NLP tasks like text generation, summarization, and sentiment analysis. Dataset Size: The dataset contains a specific number of image and text pairs. The exact number may vary depending on the dataset's source and purpose. It may range from a few dozen pairs to thousands or more, depending on its intended application.

Data Sources: The source of this dataset may vary. In this case, the images and captions have been uploaded to a platform like Kaggle. They could be sourced from a variety of places, including user-generated content, public image collections, or custom data creation.

Research and Applications: Researchers and practitioners can use this dataset to advance the state of the art in various AI fields, particularly in areas where understanding and generating text-image relationships are critical. It can be a valuable resource for building models that can comprehend and describe visual content, as well as for developing innovative applications in areas like image recognition, image search, and content recommendation.

Please note that the specifics of the dataset, including the number of image-caption pairs, data sources, and licensing, can vary depending on the actual dataset you have uploaded to Kaggle or any other platform. The above description is a generalized template and can be adapted to your specific dataset's details.

Dataset

This dataset was created by Phạm Phú Hòa

Contents

The Song Describer Dataset: a Corpus of Audio Captions for Music-and-Language Evaluation

A retro-futurist drum machine groove drenched in bubbly synthetic sound effects and a hint of an acid bassline.

The Song Describer Dataset (SDD) contains ~1.1k captions for 706 permissively licensed music recordings. It is designed for use in evaluation of models that address music-and-language (M&L) tasks such as music captioning, text-to-music generation and music-language retrieval. More information about the data, collection method and validation is provided in the paper describing the dataset.

If you use this dataset, please cite our paper:

The Song Describer Dataset: a Corpus of Audio Captions for Music-and-Language Evaluation, Manco, Ilaria and Weck, Benno and Doh, Seungheon and Won, Minz and Zhang, Yixiao and Bogdanov, Dmitry and Wu, Yusong and Chen, Ke and Tovstogan, Philip and Benetos, Emmanouil and Quinton, Elio and Fazekas, György and Nam, Juhan, Machine Learning for Audio Workshop at NeurIPS 2023, 2023