The use of intracranial recordings to decode human language: Challenges and opportunities

ABSTRACT

Decoding speech from intracranial recordings serves two main purposes: understanding the neural correlates of speech processing and decoding speech features for targeting speech neuroprosthetic devices. Intracranial recordings have high spatial and temporal resolution, and thus offer a unique opportunity to investigate and decode the electrophysiological dynamics underlying speech processing. In this review article, we describe current approaches to decoding different features of speech perception and production – such as spectrotemporal, phonetic, phonotactic, semantic, and articulatory components – using intracranial recordings. A specific section is devoted to the decoding of imagined speech, and potential applications to speech prosthetic devices. We outline the challenges in decoding human language, as well as the opportunities in scientific and neuroengineering applications.

AUTHORS

• Stéphanie Martin

• José del R. Millán

• Robert T. Knight

• Brian Pasley

Date: 2016

DOI: 10.1016/j.bandl.2016.06.003

View PDF

Word pair classification during imagined speech using direct brain recordings

ABSTRACT

People that cannot communicate due to neurological disorders would bene t from an internal speech decoder. Here, we showed the ability to classify individual words during imagined speech from electrocorticographic signals. In a word imagery task, we used high gamma (70–150 Hz) time features with a support vector machine model to classify individual words from a pair of words. To account for temporal irregularities during speech production, we introduced a non-linear time alignment into the SVM kernel. Classi cation accuracy reached 88% in a two-class classi cation framework (50% chance level), and average classi cation accuracy across fteen word-pairs was signi cant across ve subjects (mean = 58%; p < 0.05). We also compared classi cation accuracy between imagined speech, overt speech and listening. As predicted, higher classi cation accuracy was obtained in the listening and overt speech conditions (mean = 89% and 86%, respectively; p < 0.0001), where speech stimuli were directly presented. The results provide evidence for a neural representation for imagined words in the temporal lobe, frontal lobe and sensorimotor cortex, consistent with previous ndings in speech perception and production. These data represent a proof of concept study for basic decoding of speech imagery, and delineate a number of key challenges to usage of speech imagery neural representations for clinical applications.

AUTHORS

• Stéphanie Martin

• Peter Brunner

• Iñaki Iturrate

• José del R. Millán

• Gerwin Schalk

• Robert T. Knight

• Brian Pasley

Date: 2016

DOI: 10.1038/srep25803

View PDF

Understanding and Decoding Thoughts in the Human Brain

ABSTRACT

Many people cannot communicate because of their physical problems, such as paralysis. These patients cannot speak with their friends, but their brains are still working well. They can think by themselves and would bene t from a device that could read their minds and translate their thoughts into audible speech. In our study, we placed electrodes beneath patients’ skulls, directly at the surface of the brain, and measured brain activity while the patients were thinking. We then tried to decode and translate the words that they imagined into audible sounds. We showed that we could decode some parts of the sound of what patients were thinking. This was our rst attempt at translating thoughts to speech, and we hope to get much better, as many patients who cannot speak but have thoughts in their minds could bene t from a “speech decoder.”

AUTHORS

• Stéphanie Martin

• Christian Mikutta

• Robert T. Knight

• Brian Pasley

Date: 2016

DOI: 10.3389/frym.2016.00004

View PDF

Decoding spectrotemporal features of overt and covert speech from the human cortex

Authors:

• Stéphanie Martin

• Peter Brunner

• Chris Holdgraf

• Hans-Jochen Heinze

• Nathan E. Crone

• Jochem W. Rieger

• Gerwin Schalk

• Robert T. Knight

• Brian Pasley

Date: 2014

DOI: 10.3389/fneng.2014.00014

PubMed: 4034498

View PDF

Abstract:

Auditory perception and auditory imagery have been shown to activate overlapping brain regions. We hypothesized that these phenomena also share a common underlying neural representation. To assess this, we used electrocorticography intracranial recordings from epileptic patients performing an out loud or a silent reading task. In these tasks, short stories scrolled across a video screen in two conditions: subjects read the same stories both aloud (overt) and silently (covert). In a control condition the subject remained in a resting state. We first built a high gamma (70–150 Hz) neural decoding model to reconstruct spectrotemporal auditory features of self-generated overt speech. We then evaluated whether this same model could reconstruct auditory speech features in the covert speech condition. Two speech models were tested: a spectrogram and a modulation-based feature space. For the overt condition, reconstruction accuracy was evaluated as the correlation between original and predicted speech features, and was significant in each subject (p < 10−5; paired two-sample t-test). For the covert speech condition, dynamic time warping was first used to realign the covert speech reconstruction with the corresponding original speech from the overt condition. Reconstruction accuracy was then evaluated as the correlation between original and reconstructed speech features. Covert reconstruction accuracy was compared to the accuracy obtained from reconstructions in the baseline control condition. Reconstruction accuracy for the covert condition was significantly better than for the control condition (p < 0.005; paired two-sample t-test). The superior temporal gyrus, pre- and post-central gyrus provided the highest reconstruction information. The relationship between overt and covert speech reconstruction depended on anatomy. These results provide evidence that auditory representations of covert speech can be reconstructed from models that are built from an overt speech data set, supporting a partially shared neural substrate.