lmms-lab/Video-MME dataset hosted on Hugging Face and contributed by the HF Datasets community

Video-MME stands for Video Multi-Modal Evaluation. It is the first-ever comprehensive evaluation benchmark specifically designed for Multi-modal Large Language Models (MLLMs) in video analysis¹. This benchmark is significant because it addresses the need for a high-quality assessment of MLLMs' performance in processing sequential visual data, which has been less explored compared to their capabilities in static image understanding.

The Video-MME benchmark is characterized by its: 1. Diversity in video types, covering 6 primary visual domains with 30 subfields for broad scenario generalizability. 2. Duration in the temporal dimension, including short-, medium-, and long-term videos ranging from 11 seconds to 1 hour, to assess robust contextual dynamics. 3. Breadth in data modalities, integrating multi-modal inputs such as video frames, subtitles, and audios. 4. Quality in annotations, with rigorous manual labeling by expert annotators for precise and reliable model assessment¹.

The benchmark includes 900 videos totaling 256 hours, manually selected and annotated, resulting in 2,700 question-answer pairs. It has been used to evaluate various state-of-the-art MLLMs, including the GPT-4 series and Gemini 1.5 Pro, as well as open-source image and video models¹. The findings from Video-MME highlight the need for further improvements in handling longer sequences and multi-modal data, which is crucial for the advancement of MLLMs¹.

(1) [2405.21075] Video-MME: The First-Ever Comprehensive Evaluation .... https://arxiv.org/abs/2405.21075. (2) Video-MME. https://video-mme.github.io/home_page.html. (3) Video-MME: Welcome. https://video-mme.github.io/. (4) undefined. https://doi.org/10.48550/arXiv.2405.21075.

Click to add a brief description of the dataset (Markdown and LaTeX enabled).

Provide:

a high-level explanation of the dataset characteristics explain motivations and summary of its content potential use cases of the dataset

topyun/Video-MME-Long dataset hosted on Hugging Face and contributed by the HF Datasets community

Reacherx/Video-MME-Pro dataset hosted on Hugging Face and contributed by the HF Datasets community

vid-modeling/videomme dataset hosted on Hugging Face and contributed by the HF Datasets community

Jialuo21/VideoMME dataset hosted on Hugging Face and contributed by the HF Datasets community

Colombia Exports: fob: MME: RTC: Radio & TV Receivers, Apparatus for Recording & Reproducing Sound & Video, etc. data was reported at 876.003 USD th in Apr 2019. This records an increase from the previous number of 237.477 USD th for Mar 2019. Colombia Exports: fob: MME: RTC: Radio & TV Receivers, Apparatus for Recording & Reproducing Sound & Video, etc. data is updated monthly, averaging 356.856 USD th from Feb 2013 (Median) to Apr 2019, with 75 observations. The data reached an all-time high of 1,725.693 USD th in Jun 2013 and a record low of 60.660 USD th in Oct 2016. Colombia Exports: fob: MME: RTC: Radio & TV Receivers, Apparatus for Recording & Reproducing Sound & Video, etc. data remains active status in CEIC and is reported by National Statistics Administrative Department. The data is categorized under Global Database’s Colombia – Table CO.JA011: Exports: CPC Ver2 AC.

js-hyun/preprocess-videomme-data dataset hosted on Hugging Face and contributed by the HF Datasets community

This dataset contains image and video QA test sets for SITE-Bench evaluation.

VideoEval-Pro

VideoEval-Pro is a robust and realistic long video understanding benchmark containing open-ended, short-answer QA problems. The dataset is constructed by reformatting questions from four existing long video understanding MCQ benchmarks: Video-MME, MLVU, LVBench, and LongVideoBench into free-form questions. The paper can be found here. The evaluation code and scripts are available at: TIGER-AI-Lab/VideoEval-Pro

  Dataset Structure

Each example in the… See the full description on the dataset page: https://huggingface.co/datasets/TIGER-Lab/VideoEval-Pro.

哥伦比亚 Imports Value: cif: MME: RTC: Radio & TV Receivers, Apparatus for Recording & Reproducing Sound & Video, etc.在2019-04达75,673.668 美元 千，相较于2019-03的83,199.965 美元 千有所下降。哥伦比亚 Imports Value: cif: MME: RTC: Radio & TV Receivers, Apparatus for Recording & Reproducing Sound & Video, etc.数据按月度更新，2013-02至2019-04期间平均值为75,673.668 美元 千，共75份观测结果。该数据的历史最高值出现于2013-05，达440,430.861 美元 千，而历史最低值则出现于2015-01，为45,553.582 美元 千。CEIC提供的哥伦比亚 Imports Value: cif: MME: RTC: Radio & TV Receivers, Apparatus for Recording & Reproducing Sound & Video, etc.数据处于定期更新的状态，数据来源于National Statistics Administrative Department，数据归类于Global Database的哥伦比亚 – Table CO.JA034: Imports: Value: CPC Ver 2 AC。