The neurointerface expands the dictionary
The improved system for converting brain activity into sentences does not require the ability to pronounce any sounds.
Neurophysiologists have improved a system that, with the help of deep learning and a language model, transforms the brain activity of paralyzed people into sentences: it can be used by people who have completely lost their speech, and its vocabulary is 1,152 words. A description of the system and the results of its testing on a patient with anarthria are published in Nature Communications (Metzger et al., Generalizable spelling using a speech neurosthesis in an individual with severe limb and vocal paralysis).
Some illnesses and serious injuries can lead to speech loss. And if with some disorders the patient cannot formulate sentences (as, for example, with Broca's aphasia), then with paralysis the ability to articulate them is lost due to the loss of muscle strength.
One of the areas of assistance to people with paralysis concerns the development of systems for interacting with other people. Due to the fact that the brains of such people are still able to create signals for the corresponding muscle movement (but not to produce the movements themselves), signal conversion systems are able to decode brain activity. Among the researchers involved in such developments is a team of American neurophysiologists and engineers who have created a system in which a brain implant reads the activity of neurons in speech motor zones and converts them into whole phrases.
Now the same team, with the participation of Edward F. Chang from the University of California, has expanded past research and supplemented the speech conversion system.
In the development and testing of the system in the previous work of the authors and the current project, one patient with anarthria (lack of speech associated with neuromuscular damage), who was 36 years old at the time of the study, took part in the development and testing of the system. Since the patient still had the ability to produce some sounds (for example, mooing), with the previous system, he controlled the neuroprosthesis, trying to speak aloud, and the dictionary was only 50 words.
In the new study, the authors collected data on the participant's brain activity during the passage of two tasks. In the first task, at a certain signal, he had to try to pronounce a word or letter that appeared on the screen, to himself or out loud, depending on the sample. If there was the word "right" on the screen and an arrow pointing to the right, the participant had to try to squeeze his right hand. The researchers used this data to train and optimize models for signal detection and classification.
In the second task, the participant had to try to pronounce a phrase shown on the screen, or an arbitrary phrase, depending on the sample. These data were then used for additional optimization of the model and its evaluation.
Testing a model with a dictionary of 1,152 words showed a median error of 6.13 percent for decoding letters and 10.53 percent for decoding words. Of the 150 sentences, 70 percent were decoded without errors, the median decoding rate was 29.41 letters and 6.86 sentences per minute. This turned out to be faster than the auxiliary device that the patient used in everyday life.
The authors also tested the difference in the activation of neurons and the work of the conversion system when trying to pronounce words aloud and to themselves. They found that, despite similar activity patterns, these processes are not identical: training and testing conducted on the same data showed higher accuracy of speech conversion than cross-validation (p<0.01).
Thus, the authors managed to improve the speech conversion system by expanding the dictionary and training the model to process the activation of neurons when trying to pronounce words to themselves (without using any sounds). This means that the system will also be able to help people who are completely paralyzed and cannot make sounds when trying to pronounce a phrase, and data simulation has shown that the system's vocabulary can be expanded to 9170 words, which will allow patients to communicate freely.
Portal "Eternal youth" http://vechnayamolodost.ru