📊 MMLongBench-Doc Evaluation Results

Official evaluation results: GPT-4.1 (2025-04-14) & GPT-4o (2024-11-20) 📄 Paper: MMLongBench-Doc, NeurIPS 2024 Datasets and Benchmarks Track (Spotlight)

Accepted by NeurIPS 2024 Datasets and Benchmarks Track

We introduce the RePair puzzle-solving dataset, a large-scale real world dataset of fractured frescoes from the archaelogical campus of Pompeii. Our dataset consists of over 1000 fractured frescoes. The RePAIR stands as a realistic computational challenge for methods for 2D and 3D puzzle solving, and serves as a benchmark that enables the study of fractured object reassembly and presents new challenges for geometric shape understanding. Please visit our website for more dataset information, access to source code scripts and for an interactive gallery viewing of the dataset samples.

Semi Truths Dataset: A Large-Scale Dataset for Testing Robustness of AI-Generated Image Detectors (NeurIPS 2024 Track Datasets & Benchmarks Track)

Recent efforts have developed AI-generated image detectors claiming robustness against various augmentations, but their effectiveness remains unclear. Can these systems detect varying degrees of augmentation?

To address these questions, we introduce Semi-Truths, featuring 27, 600 real images, 223, 400 masks, and 1, 472, 700… See the full description on the dataset page: https://huggingface.co/datasets/semi-truths/Semi-Truths.

The paper has been accepted in NeurIPS 2024 (Dataset & Benchmark Track). paper repo

☠️ Warning: The samples presented by this paper may be considered offensive or vulgar.

❗️ Ethics Statement

The opinions and findings contained in the samples of our presented dataset should not be interpreted as representing the views expressed or implied by the authors. We acknowledge the risk of malicious actors attempting to reverse-engineer memes. We sincerely hope that users will employ the dataset responsibly and appropriately, avoiding misuse or abuse. We believe the benefits of our proposed resources outweigh the associated risks. All resources are intended solely for scientific research and are prohibited from commercial use.

📜 Chinese Harmful Meme

To adapt to the Chinese online environment, we introduce the definition of Chinese harmful memes:

Chinese harmful memes are multimodal units consisting of an image and Chinese inline text that have the potential to cause harm to an individual, an organization, a community, a social group, or society as a whole. These memes can range from offense or joking that perpetuate harmful stereotypes towards specific social entities, to memes that are more subtle and general but still have the potential to cause harm. It is important to note that Chinese harmful memes can be created and spread intentionally or unintentionally. They often reflect and reinforce underlying negative values and cultural attitudes on the Chinese Internet, which are detrimental from legal or moral perspectives.

📜 ToxiCN MM

According to the definition, we identify the most common harmful types of memes on Chinese platforms, including targeted harmful, general offense, sexual innuendo, and dispirited culture. We focus on these harmful types when constructing the dataset.

During the annotation, we label memes from two aspects: harmful types (i.e., the above four types) and modality combination (i.e., analyzing toxicity through fused or independent features, including Text-Image Fusion, Harmful Text, and Harmful Image). Finally, we present the ToxiCN MM dataset, which contains 12,000 samples.

Considering the potential risk of abuse, please fill out the following form to request the datasets: https://forms.gle/UN61ZNfTgMZKfMrv7. After we get your request, we will send the dataset to your email as soon as possible. The dataset labels and captions generated by GPT-4V have been saved as train_data_discription.json and test_data_discription.json in the ./data/ directory. Here we simply describe each fine-grain label.

📜 Detector

We present a Multimodal Knowledge Enhancement Detector for effective detection. It incorporates contextual information of meme content to enhance the detector's understanding of Chinese memes generated by the LLM. The requirements.txt file lists the specific dependencies of the project.

❗️ Licenses

This work is licensed under a Creative Commons Attribution- NonCommercial-NoDerivatives 4.0 International License (CC BY-NC-ND 4.0).

Poster

https://github.com/user-attachments/assets/c3cb7793-33f2-4e3e-ad72-e0d84530c658" alt="poster_original">

Cite

If you want to use the resources, please cite the following paper. The camera-ready version of the paper will be released after the conference: ~~~ @article{lu2024towards, title={Towards Comprehensive Detection of Chinese Harmful Memes}, author={Lu, Junyu and Xu, Bo and Zhang, Xiaokun and Wang, Hongbo and Zhu, Haohao and Zhang, Dongyu and Yang, Liang and Lin, Hongfei}, journal={arXiv preprint arXiv:2410.02378}, year={2024} } ~~~

WikiDBs is an open-source corpus of 100,000 relational databases. We aim to support research on tabular representation learning on multi-table data. The corpus is based on Wikidata and aims to follow certain characteristics of real-world databases.

WikiDBs was published as a spotlight paper at the Dataset & Benchmarks track at NeurIPS 2024.

WikiDBs contains the database schemas, as well as table contents. The database tables are provided as CSV files, and each database schema as JSON. The 100,000 databases are available in five splits, containing 20k databases each. In total, around 165 GB of disk space are needed for the full corpus. We also provide a script to convert the databases into SQLite.

BLEnD

This is the official repository of BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages (Submitted to NeurIPS 2024 Datasets and Benchmarks Track). 24/12/05: Updated translation errors25/05/02: Updated multiple choice questions file (v1.1)

  About

Large language models (LLMs) often lack culture-specific everyday knowledge, especially across diverse regions and non-English languages. Existing benchmarks for evaluating LLMs' cultural… See the full description on the dataset page: https://huggingface.co/datasets/nayeon212/BLEnD.

The dataset contains UAV footage of wild antelopes (blackbucks) in grassland habitats. It can be mainly used for two tasks: Multi-object tracking (MOT) and Re-Identification (Re-ID). We provide annotations for the position of animals in each frame, allowing us to offer very long videos (up to 3 min) completely annotated while maintaining the identity of each animal in the video. The Re-ID dataset offers two videos, that capture the movement of some animals simultaneously from two different UAVs. The Re-ID task is to find the same individual in two videos taken simultaneously from a slightly different perspective. The relevant paper will be published in the NeurIPS 2024 Dataset and Benchmarking Track. https://nips.cc/virtual/2024/poster/97563 Resolution: 5.4 K MOT: 12 videos ( MOT17 Format) Re-ID: 6 sets (each with a pair of drones) (Custom) Detection: 320 Images (COCO, YOLO)