Encoding and decoding analysis of music perception using intracranial EEG

Abstract:

Music perception engages multiple brain regions, however the neural dynamics of this core human experience remains elusive. We applied predictive models to intracranial EEG data from 29 patients listening to a Pink Floyd song. We investigated the relationship between the song spectrogram and the elicited high-frequency activity (70-150Hz), a marker of local neural activity. Encoding models characterized the spectrotemporal receptive fields (STRFs) of each electrode and decoding models estimated the population-level song representation. Both methods confirmed a crucial role of the right superior temporal gyri (STG) in music perception. A component analysis on STRF coefficients highlighted overlapping neural populations tuned to specific musical elements (vocals, lead guitar, rhythm). An ablation analysis on decoding models revealed the presence of unique musical information concentrated in the right STG and more spatially distributed in the left hemisphere. Lastly, we provided the first song reconstruction decoded from human neural activity.

Authors:

• Ludovic Bellier

• Anaïs Llorens

• Déborah Marciano

• Gerwin Schalk

• Peter Brunner

• Robert T. Knight

• Brian N. Pasley

Date: 2022

DOI: https://doi.org/10.1101/2022.01.27.478085

View PDF

The use of intracranial recordings to decode human language: Challenges and opportunities

Abstract:

Decoding speech from intracranial recordings serves two main purposes: understanding the neural correlates of speech processing and decoding speech features for targeting speech neuroprosthetic devices. Intracranial recordings have high spatial and temporal resolution, and thus offer a unique opportunity to investigate and decode the electrophysiological dynamics underlying speech processing. In this review article, we describe current approaches to decoding different features of speech perception and production – such as spectrotemporal, phonetic, phonotactic, semantic, and articulatory components – using intracranial recordings. A specific section is devoted to the decoding of imagined speech, and potential applications to speech prosthetic devices. We outline the challenges in decoding human language, as well as the opportunities in scientific and neuroengineering applications.

Authors:

• Stephanie Martin

• José del R. Millán

• Robert T. Knight

• Brian N. Pasley

Date: 2019

DOI: http://dx.doi.org/10.1016/j.bandl.2016.06.003

View PDF

Decoding Inner Speech Using Electrocorticography: Progress and Challenges Toward a Speech Prosthesis

ABSTRACT

Certain brain disorders resulting from brainstem infarcts, traumatic brain injury, cerebral palsy, stroke, and amyotrophic lateral sclerosis, limit verbal communication despite the patient being fully aware. People that cannot communicate due to neurological disorders would benefit from a system that can infer internal speech directly from brain signals. In this review article, we describe the state of the art in decoding inner speech, ranging from early acoustic sound features, to higher order speech units. We focused on intracranial recordings, as this technique allows monitoring brain activity with high spatial, temporal, and spectral resolution, and therefore is a good candidate to investigate inner speech. Despite intense efforts, investigating how the human cortex encodes inner speech remains an elusive challenge, due to the lack of behavioral and observable measures. We emphasize various challenges commonly encountered when investigating inner speech decoding, and propose potential solutions in order to get closer to a natural speech assistive device.

AUTHORS

• Stephanie Martin

• Iñaki Iturrate

• José del R. Millán

• Robert T. Knight

• Brian N. Pasley

Date: 2018

DOI: 10.3389/fnins.2018.00422

View PDF