Imagined speech can be decoded from low- and cross-frequency intracranial EEG features

Abstract:

Reconstructing intended speech from neural activity using brain-computer interfaces holds great promises for people with severe speech production deficits. While decoding overt speech has progressed, decoding imagined speech has met limited success, mainly because the associated neural signals are weak and variable compared to overt speech, hence difficult to decode by learning algorithms. We obtained three electrocorticography datasets from 13 patients, with electrodes implanted for epilepsy evaluation, who performed overt and imagined speech production tasks. Based on recent theories of speech neural processing, we extracted consistent and specific neural features usable for future brain computer interfaces, and assessed their performance to discriminate speech items in articulatory, phonetic, and vocalic representation spaces. While high-frequency activity provided the best signal for overt speech, both low- and higher-frequency power and local cross-frequency contributed to imagined speech decoding, in particular in phonetic and vocalic, i.e. perceptual, spaces. These findings show that low-frequency power and cross-frequency dynamics contain key information for imagined speech decoding.

Authors:

• Timothée Proix

• Jaime Delgado Saa

• Andy Christen

• Stephanie Martin

• Brian N. Pasley

• Robert T. Knight

• Xing Tian

• David Poeppel

• Werner K. Doyle

• Orrin Devinsky

• Luc H. Arnal

• Pierre Mégevand

• Anne-Lise Giraud

Date: 2021

DOI: https://doi.org/10.1038/s41467-021-27725-3

View PDF

Rapid tuning shifts in human auditory cortex enhance speech intelligibility

ABSTRACT

Experience shapes our perception of the world on a moment-to-moment basis. This robust perceptual effect of experience parallels a change in the neural representation of stimulus features, though the nature of this representation and its plasticity are not well-understood. Spectrotemporal receptive field (STRF) mapping describes the neural response to acoustic features, and has been used to study contextual effects on auditory receptive fields in animal models. We performed a STRF plasticity analysis on electrophysiological data from recordings obtained directly from the human auditory cortex. Here, we report rapid, automatic plasticity of the spectrotemporal response of recorded neural ensembles, driven by previous experience with acoustic and linguistic information, and with a neurophysiological effect in the sub-second range. This plasticity reflects increased sensitivity to spectrotemporal features, enhancing the extraction of more speech-like features from a degraded stimulus and providing the physiological basis for the observed ‘perceptual enhancement’ in understanding speech.

AUTHORS

• Chris Holdgraf

• Wendy de Heer

• Brian Pasley

• Jochem W. Rieger

• Nathan E. Crone

• Jack J. Lin

• Robert T. Knight

• Frédéric E. Theunissen

Date: 2016

DOI: 10.1038/ncomms13654

View PDF

The use of intracranial recordings to decode human language: Challenges and opportunities

ABSTRACT

Decoding speech from intracranial recordings serves two main purposes: understanding the neural correlates of speech processing and decoding speech features for targeting speech neuroprosthetic devices. Intracranial recordings have high spatial and temporal resolution, and thus offer a unique opportunity to investigate and decode the electrophysiological dynamics underlying speech processing. In this review article, we describe current approaches to decoding different features of speech perception and production – such as spectrotemporal, phonetic, phonotactic, semantic, and articulatory components – using intracranial recordings. A specific section is devoted to the decoding of imagined speech, and potential applications to speech prosthetic devices. We outline the challenges in decoding human language, as well as the opportunities in scientific and neuroengineering applications.

AUTHORS

• Stéphanie Martin

• José del R. Millán

• Robert T. Knight

• Brian Pasley

Date: 2016

DOI: 10.1016/j.bandl.2016.06.003

View PDF

Word pair classification during imagined speech using direct brain recordings

ABSTRACT

People that cannot communicate due to neurological disorders would bene t from an internal speech decoder. Here, we showed the ability to classify individual words during imagined speech from electrocorticographic signals. In a word imagery task, we used high gamma (70–150 Hz) time features with a support vector machine model to classify individual words from a pair of words. To account for temporal irregularities during speech production, we introduced a non-linear time alignment into the SVM kernel. Classi cation accuracy reached 88% in a two-class classi cation framework (50% chance level), and average classi cation accuracy across fteen word-pairs was signi cant across ve subjects (mean = 58%; p < 0.05). We also compared classi cation accuracy between imagined speech, overt speech and listening. As predicted, higher classi cation accuracy was obtained in the listening and overt speech conditions (mean = 89% and 86%, respectively; p < 0.0001), where speech stimuli were directly presented. The results provide evidence for a neural representation for imagined words in the temporal lobe, frontal lobe and sensorimotor cortex, consistent with previous ndings in speech perception and production. These data represent a proof of concept study for basic decoding of speech imagery, and delineate a number of key challenges to usage of speech imagery neural representations for clinical applications.

AUTHORS

• Stéphanie Martin

• Peter Brunner

• Iñaki Iturrate

• José del R. Millán

• Gerwin Schalk

• Robert T. Knight

• Brian Pasley

Date: 2016

DOI: 10.1038/srep25803

View PDF

Understanding and Decoding Thoughts in the Human Brain

ABSTRACT

Many people cannot communicate because of their physical problems, such as paralysis. These patients cannot speak with their friends, but their brains are still working well. They can think by themselves and would bene t from a device that could read their minds and translate their thoughts into audible speech. In our study, we placed electrodes beneath patients’ skulls, directly at the surface of the brain, and measured brain activity while the patients were thinking. We then tried to decode and translate the words that they imagined into audible sounds. We showed that we could decode some parts of the sound of what patients were thinking. This was our rst attempt at translating thoughts to speech, and we hope to get much better, as many patients who cannot speak but have thoughts in their minds could bene t from a “speech decoder.”

AUTHORS

• Stéphanie Martin

• Christian Mikutta

• Robert T. Knight

• Brian Pasley

Date: 2016

DOI: 10.3389/frym.2016.00004

View PDF

Decoding Speech for Understanding and Treating Aphasia

Authors:

• Brian Pasley

• Robert T. Knight

Date: 2015

DOI: 10.1016/B978-0-444-63327-9.00018-7

View PDF

Abstract:

Aphasia is an acquired language disorder with a diverse set of symptoms that can affect virtually any linguistic modality across both the comprehension and production of spoken language. Partial recovery of language function after injury is common but typically incomplete. Rehabilitation strategies focus on behavioral training to induce plasticity in underlying neural circuits to maximize linguistic recovery. Understanding the different neural circuits underlying diverse language functions is a key to developing more effective treatment strategies. This chapter discusses a systems identification analytic approach to the study of linguistic neural representation. The focus of this framework is a quantitative, model-based characterization of speech and language neural representations that can be used to decode, or predict, speech representations from measured brain activity. Recent results of this approach are discussed in the context of applications to understanding the neural basis of aphasia symptoms and the potential to optimize plasticity during the rehabilitation process.

Decoding spectrotemporal features of overt and covert speech from the human cortex

Authors:

• Stéphanie Martin

• Peter Brunner

• Chris Holdgraf

• Hans-Jochen Heinze

• Nathan E. Crone

• Jochem W. Rieger

• Gerwin Schalk

• Robert T. Knight

• Brian Pasley

Date: 2014

DOI: 10.3389/fneng.2014.00014

PubMed: 4034498

View PDF

Abstract:

Auditory perception and auditory imagery have been shown to activate overlapping brain regions. We hypothesized that these phenomena also share a common underlying neural representation. To assess this, we used electrocorticography intracranial recordings from epileptic patients performing an out loud or a silent reading task. In these tasks, short stories scrolled across a video screen in two conditions: subjects read the same stories both aloud (overt) and silently (covert). In a control condition the subject remained in a resting state. We first built a high gamma (70–150 Hz) neural decoding model to reconstruct spectrotemporal auditory features of self-generated overt speech. We then evaluated whether this same model could reconstruct auditory speech features in the covert speech condition. Two speech models were tested: a spectrogram and a modulation-based feature space. For the overt condition, reconstruction accuracy was evaluated as the correlation between original and predicted speech features, and was significant in each subject (p < 10−5; paired two-sample t-test). For the covert speech condition, dynamic time warping was first used to realign the covert speech reconstruction with the corresponding original speech from the overt condition. Reconstruction accuracy was then evaluated as the correlation between original and reconstructed speech features. Covert reconstruction accuracy was compared to the accuracy obtained from reconstructions in the baseline control condition. Reconstruction accuracy for the covert condition was significantly better than for the control condition (p < 0.005; paired two-sample t-test). The superior temporal gyrus, pre- and post-central gyrus provided the highest reconstruction information. The relationship between overt and covert speech reconstruction depended on anatomy. These results provide evidence that auditory representations of covert speech can be reconstructed from models that are built from an overt speech data set, supporting a partially shared neural substrate.

Reconstructing Speech from Human Auditory Cortex

Authors:

• Brian Pasley

• Stephen V. David

• Nima Mesgarani

• Adeen Flinker

• Shihab A. Shamma

• Nathan E. Crone

• Robert T. Knight

• Edward F. Chang

Date: 2012

DOI: 10.1371/journal.pbio.1001251

PubMed: 22303281

View PDF

Abstract:

How the human auditory system extracts perceptually relevant acoustic features of speech is unknown. To address this question, we used intracranial recordings from nonprimary auditory cortex in the human superior temporal gyrus to determine what acoustic information in speech sounds can be reconstructed from population neural activity. We found that slow and intermediate temporal fluctuations, such as those corresponding to syllable rate, were accurately reconstructed using a linear model based on the auditory spectrogram. However, reconstruction of fast temporal fluctuations, such as syllable onsets and offsets, required a nonlinear sound representation based on temporal modulation energy. Reconstruction accuracy was highest within the range of spectro-temporal fluctuations that have been found to be critical for speech intelligibility. The decoded speech representations allowed readout and identification of individual words directly from brain activity during single trial sound presentations. These findings reveal neural encoding mechanisms of speech acoustic parameters in higher order human auditory cortex.

State-dependent variability of neuronal responses to transcranial magnetic stimulation of the visual cortex

Authors:

• Brian Pasley

• Elena A. Allen

• Ralph D. Freeman

Date: 2009

View PDF

Abstract:

Electrical brain stimulation is a promising tool for both experimental and clinical applications. However, the effects of stimulation on neuronal activity are highly variable and poorly understood. To investigate the basis of this variability, we performed extracellular recordings in the visualcortex following application of transcranialmagneticstimulation (TMS). Our measurements of spiking and local field potential activity exhibit two types of response patterns which are characterized by the presence or absence of spontaneous discharge following stimulation. This variability can be partially explained by state-dependent effects, in which higher pre-TMS activity predicts larger post-TMS responses. These results reveal the possibility that variability in the neural response to TMS can be exploited to optimize the effects of stimulation. It is conceivable that this feature could be utilized in real time during the treatment of clinical disorders.

Transcranial magnetic stimulation elicits coupled neural and hemodynamic consequences

Authors:

• Brian Pasley

• Elena A. Allen

• T. Duong

• Ralph D. Freeman

Date: 2007

View PDF

Abstract:

Transcranial magnetic stimulation (TMS) is an increasingly common technique used to selectively modify neural processing. However, application of TMS is limited by uncertainty concerning its physiological effects. We applied TMS to the cat visual cortex and evaluated the neural and hemodynamic consequences. Short TMS pulse trains elicited initial activation (~1 minute) and prolonged suppression (5 to 10 minutes) of neural responses. Furthermore, TMS disrupted the temporal structure of activity by altering phase relationships between neural signals. Despite the complexity of this response, neural changes were faithfully reflected in hemodynamic signals; quantitative coupling was present over a range of stimulation parameters. These results demonstrate long-lasting neural responses to TMS and support the use of hemodynamic-based neuroimaging to effectively monitor these changes over time.

Analysis of oxygen metabolism implies a neural origin for the negative BOLD response in human visual cortex

Authors:

• Brian Pasley

• B. A. Inglis

• Ralph D. Freeman

Date: 2007

PubMed: 17113313

View PDF

Abstract:

The sustained negative blood oxygenation level-dependent (BOLD) response in functional MRI is observed universally, but its interpretation is controversial. The origin of the negative response is of fundamental importance because it could provide a measurement of neural deactivation. However, a substantial component of the negative response may be due to a non-neural hemodynamic artifact. To distinguish these possibilities, we have measured evoked BOLD, cerebral blood flow (CBF), and oxygen metabolism responses to a fixed visual stimulus from two different baseline conditions. One is a normal resting baseline, and the other is a lower baseline induced by a sustained negative response. For both baseline conditions, CBF and oxygen metabolism responses reach the same peak amplitude. Consequently, evoked responses from the negative baseline are larger than those from the resting baseline. The larger metabolic response from negative baseline presumably reflects a greater neural response that is required to reach the same peak amplitude as that from resting baseline. Furthermore, the ratio of CBF to oxygen metabolism remains approximately the same from both baseline states (approximately 2:1). This tight coupling between hemodynamic and metabolic components implies that the magnitude of any hemodynamic artifact is inconsequential. We conclude that the negative response is a functionally significant index of neural deactivation in early visual cortex.

Subcortical discrimination of unperceived objects during binocular rivalry

Authors:

• Brian Pasley

• L. C. Mayes

• R. T. Schultz

Date: 2004

DOI: dx.doi.org/10.1016/S0896-6273(04)00155-2

PubMed: 15066273

View PDF

Abstract:

Rapid identification of behaviorally relevant objects is important for survival. In humans, the neural computations for visually discriminating complex objects involve inferior temporal cortex (IT). However, less detailed but faster form processing may also occur in a phylogenetically older subcortical visual system that terminates in the amygdala. We used binocular rivalry to present stimuli without conscious awareness, thereby eliminating the IT object representation and isolating subcortical visual input to the amygdala. Functional magnetic resonance imaging revealed significant brain activation in the left amygdala but not in object-selective IT in response to unperceived fearful faces compared to unperceived nonface objects. These findings indicate that, for certain behaviorally relevant stimuli, a high-level cortical representation in IT is not required for object discrimination in the amygdala.