Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Research Highlight
  • Published:

Speech recognition and processing

An electrocorticography-based speech decoder for neural speech prostheses

Nature Reviews Electrical Engineering volume 1, page 284 (2024)Cite this article

Subjects

Neural speech prostheses based on brain–computer interface (BCI) systems and machine learning technologies are fundamental tools for helping restore speech loss due to neurological deficits. Despite the progress, advances in speech decoding are still hampered by the limited timeframe of data available for training personalized models and the inherent variability in speech production. Now, writing in Nature Machine Intelligence, Adeen Flinker and colleagues introduce an electrocorticography (ECoG)-to-speech framework that overcomes data scarcity and enhances natural-sound speech synthesis.

The approach is based on a specific acoustic space that can synthesize speech using differentiable digital signal processing and an acoustic pre-training that is blind to the neural data and enable to decode realistic vocalizations with limited data. The training process involves semi-supervised learning that initially uses only speech signals, followed by supervised learning based on ground-truth spectrograms (using measures of spectrogram difference and short-time objective intelligibility, STOI). The team used various deep learning architectures (including 3D ResNet, 3D Swin Transformer and LSTM) that reply on different causality directions for the ECoG decoder and trained the models using a portion of the data and validated them on the remaining set. Robust decoding performance were achieved across participants with both high-density and low-density electrode sampling on the cortex and high accuracy was demonstrated across different models, with causal decoding with convolutional architectures showing superior performance.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

References

Original article

  • Chen, X. et al. A neural speech decoding framework leveraging deep learning and speech synthesis. Nat. Mach. Intell. 6, 467–480 (2024)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Nature Reviews Electrical Engineering https://www.nature.com/natrevelectreng/

    Silvia Conti

Authors
  1. Silvia Conti
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Silvia Conti.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Conti, S. An electrocorticography-based speech decoder for neural speech prostheses. Nat Rev Electr Eng 1, 284 (2024). https://doi.org/10.1038/s44287-024-00054-5

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s44287-024-00054-5

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing