This release contains data sets for experiments with document-level machine translation. The data sets have been used in previous studies and provided here for replicability and comparison with other systems. The data sets are taken from the English-German news translation task at WMT 2019 and the English-German bitext in the OpenSubtitles collection v2016 from OPUS. All data sets are sentence aligned with corresponding lines being aligned to each other. Document boundaries are marked with empty lines (on both sides of the parallel corpus).The data set has been used in the following publication:@inproceedings{scherrer-tiedemann-loaiciga-2019, title = "Analysing concatenation approaches to document-level NMT in two different domains", author = {Scherrer, Yves and Tiedemann, J{"o}rg and Lo{\'a}iciga, Sharid}, booktitle = "Proceedings of the Third Workshop on Discourse in Machine Translation", month = nov, year = "2019", address = "Hong-Kong", publisher = "Association for Computational Linguistics",}Please, cite that paper if you use the data set in your own work.

eole-nlp/synth-greedy-decoded-doclevel dataset hosted on Hugging Face and contributed by the HF Datasets community

A/B Test Supertext vs DeepL

We release all evaluation data and scripts for further analysis and reproduction of the accompanying paper: A comparison of translation performance between DeepL and Supertext. The data consists of document-level translations by Supertext and DeepL as well as accompanying ratings by professional translators. Please find more details in the paper. Please note that the empty lines correspond to paragraph boundaries (i.e., double line breaks) in the original… See the full description on the dataset page: https://huggingface.co/datasets/Supertext/mt-doclevel-ab-test.

WebNovelTrans/kunpeng-doc-level-webnovel-instruction dataset hosted on Hugging Face and contributed by the HF Datasets community

Dataset Card for Dataset Name

  Dataset Summary

The benchmark datasets for document-level machine translation.

  Supported Tasks

Document-level Machine Translation Tasks.

  Languages

English-German

  Dataset Structure





  Data Instances

TED: iwslt17, News: nc2016, Europarl: europarl7

  Data Fields

Pure text that each line represents a sentence and multiple lines separated by '

  Data Splits

train… See the full description on the dataset page: https://huggingface.co/datasets/gshbao/DocNMT.

This dataset was constructed by merging individual sentences from the Flores dataset based on matching domain, topic, and URL attributes. The result is a long-context, document-level parallel benchmark. For more details on the domains and dataset statistics, please refer to the original paper and the dataset.