Bimodal dataset on Inner Speech

Code available: https://github.com/LTU-Machine-Learning/Inner_Speech_EEG_FMRI

Publication available: https://www.biorxiv.org/content/10.1101/2022.05.24.492109v3

Abstract: The recognition of inner speech, which could give a `voice' to patients that have no ability to speak or move, is a challenge for brain-computer interfaces (BCIs). A shortcoming of the available datasets is that they do not combine modalities to increase the performance of inner speech recognition. Multimodal datasets of brain data enable the fusion of neuroimaging modalities with complimentary properties, such as the high spatial resolution of functional magnetic resonance imaging (fMRI) and the temporal resolution of electroencephalography (EEG), and therefore are promising for decoding inner speech. This paper presents the first publicly available bimodal dataset containing EEG and fMRI data acquired nonsimultaneously during inner-speech production. Data were obtained from four healthy, right-handed participants during an inner-speech task with words in either a social or numerical category. Each of the 8-word stimuli were assessed with 40 trials, resulting in 320 trials in each modality for each participant.
The aim of this work is to provide a publicly available bimodal dataset on inner speech, contributing towards speech prostheses.

Short Dataset description: The dataset consists of 1280 trials in each modality (EEG, FMRI). The stimuli contain 8 words, selected from 2 different categories (social, numeric): Social: child, daughter, father, wife Numeric: four, three, ten, six

There are 4 subjects in total: sub-01, sub-02, sub-03, sub-05. Initially, there were 5 participants, however, sub-04 data was rejected due to high fluctuations. Details of valid data are available in the file participants.tsv.

For questions please contact: foteini.liwicki@ltu.se

This dataset contains key characteristics about the data described in the Data Descriptor Thinking out loud: an open-access EEG-based BCI dataset for inner speech recognition. Contents:

    1. human readable metadata summary table in CSV format


    2. machine readable metadata file in JSON format

Original Paper: Foteini Simistira Liwicki, Vibha Gupta, Rajkumar Saini, Kanjar De, Nosheen Abid, Sumit Rakesh, Scott Wellington, Holly Wilson, Marcus Liwicki, Johan Eriksson (2022), Bimodal pilot study on inner speech decoding reveals the potential of combining EEG and fMRI, Url: https://www.biorxiv.org/content/10.1101/2022.05.24.492109v1

Overview: The dataset involves a study for inner speech detection using bimodal electroencephalography (EEG) and Functional Magnetic Resonance Imaging (FMRI). The dataset consits of 1280 trials from 4 adult healthy subjects (2 male + 2 female) for each modality. The dataset consits of 1280 trials in each modality (EEG + FMRI). The stimulus consists of 4 words from 2 categories: 1. Social: child, daughter, father, wife 2. Numeric: four, three, ten, six

There are four subjects sub01, sub02, sub03, sub05 files. Intitially there were 5 participants, however sub04 data was rejected due to high gluctuations. Details of valid data are available at participants.tsv file.

Experimental paradigm: To Do

EEG DATA: RAW EEG Data Device: BioSemi Active2 EEG data was sampled

FMRI DATA:

All the data was acquired via SIEMENS MAGNETOM Prisma All DICOM files were converted to NFTII using MRIcroGL 1.2.20211006 x86-64 FPC which uses dcm2niix for the conversion. FMRI task event file for each subject session conatins the following columns Innerword - Can be any of the 8 words child, daughter, father, wife, four, three, ten, six Condition - Can be Social or Numeric Stimulus Onset time in Seconds Stimulus Offset time in Seconds

Task Events are available in subX_seesionY.tsv where X stands for subject number 1,2,3 and 5 and Y stands for session number 1 or 2.

For FMRI BOLD files the following parameters were used. Slices:68 TR:2160 ms TE:30.00 ms

For FMRI field maps GRE Field maps: TR: 662.0 ms TE1: 4.92ms TE2: 7.38ms

Inner Speech Dataset.

Author: Nieto Nicolás

Code available at https://github.com/N-Nieto/Inner_Speech_Dataset

Abstract: Surface electroencephalography is a standard and noninvasive way to measure electrical brain activity. Recent advances in artificial intelligence led to significant improvements in the automatic detection of brain patterns, allowing increasingly faster, more reliable and accessible Brain-Computer Interfaces. Different paradigms have been used to enable the human-machine interaction and the last few years have broad a mark increase in the interest for interpreting and characterizing the "inner voice" phenomenon. This paradigm, called inner speech, raises the possibility of executing an order just by thinking about it, allowing a “natural” way of controlling external devices. Unfortunately, the lack of publicly available electroencephalography datasets, restricts the development of new techniques for inner speech recognition. A ten-subjects dataset acquired under this and two others related paradigms, obtain with an acquisition systems of 136 channels, is presented. The main purpose of this work is to provide the scientific community with an open-access multiclass electroencephalography database of inner speech commands that could be used for better understanding of the related brain mechanisms.

Conditions = Inner Speech, Pronounced Speech, Visualized Condition

Classes = "Arriba/Up", "Abajo/Down", "Derecha/Right", "Izquierda/Left"

Total Trials = 5640

Please contact us at this e-mail address if you have any doubts: nnieto@sinc.unl.edu.ar

Speech is essential for human communication, but millions of people lose the ability to speak due to conditions such as amyotrophic lateral sclerosis (ALS) or stroke. Assistive technologies like brain-computer interfaces (BCIs), which can convert brain signals into speech, offer hope for these patients. However, these technologies still face challenges in decoding accuracy. This issue is especially challenging for tonal languages like Mandarin Chinese. Their processing requires phoneme encoding and precise tonal information handling, which complicates the decoding of brain signals. Furthermore, most existing speech datasets are based on Indo-European languages, which hinders our understanding of how tonal information is encoded in the brain. To address this, we introduce a comprehensive open dataset, which includes multimodal signals from 30 subjects using Mandarin Chinese across overt, silent, and imagined speech modes, covering electroencephalogram (EEG), surface electromyogram (sEMG), and speech recordings. Unlike many datasets that focus on a single speech mode, this one integrates three speech modes, providing a more comprehensive view of speech-related activity. Incorporating Mandarin facilitates an in-depth examination of the inner mechanisms that encode tonal variations and their interaction with motor and auditory speech representations. This is crucial for enhancing tonal language decoding in BCIs. Beyond BCI applications, this dataset lays a valuable groundwork for exploring the neural encoding of tonal languages, investigating tone-related brain dynamics, and improving assistive communication strategies. It supports cross-linguistic speech processing research and contributes to data-driven neural speech decoding technology innovations.