The UCSF team has made surprising progress and reports today in the New England Journal of Medicine that they have used these electrodes to decode speech in real time. The subject was a 36-year-old man researchers call “Bravo-1,” who, after a severe stroke, has lost his ability to form intelligible words and can only growl or moan. In their report, Chang’s group says that with the electrodes on the surface of his brain, Bravo-1 was able to form sentences on a computer at a rate of about 15 words per minute. The technology involves measuring neural signals in the part of the motor cortex associated with Bravo-1’s efforts to move his tongue and vocal tracts as he imagines speaking.
To achieve this result, Chang’s team asked Bravo-1 to imagine saying one of 50 common words almost 10,000 times, feeding the patient’s neural signals to a deep learning model. After training the model to match words with neural signals, the team was able to correctly determine the word Bravo-1 thought it said 40% of the time (the luck results would have been around 2%). Even so, his sentences were full of errors. “Hello how are you?” might come out “Hungry how are you.”
But scientists improved performance by adding a language model, a program that evaluates the most likely word sequences in English. This increased the accuracy to 75%. With this cyborg approach, the system could predict that Bravo-1’s phrase “I right my nurse” actually meant “I love my nurse”.
As remarkable as the result is, there are over 170,000 words in English, and therefore performance would drop outside of Bravo-1’s restricted vocabulary. This means that the technique, while it can be useful as a medical aid, is not close to what Facebook had in mind. “We see applications for the foreseeable future in clinical assistive technology, but that’s not where our business is,” says Chevillet. “We are focused on consumer applications, and there is a very long way to go for that. “
Facebook’s decision to give up brain reading does not shock researchers studying these techniques. “I can’t say I’m surprised, because they had hinted that they were considering a short time frame and that they were going to reassess things,” said Marc Slutzky, professor at Northwestern whose former student Emily Mugler was. a key hire for Facebook. his project. “Speaking of experience, the goal of speech decoding is a big challenge. We are still far from a practical and comprehensive solution.
Still, Slutsky says the UCSF project is an “awesome next step” that demonstrates both remarkable possibilities and some limitations in the science of brain reading. “It remains to be seen if you can decode free speech,” he says. “A patient who says ‘I want a glass of water’ as opposed to ‘I want my medicine’, well, that’s different.” He says if artificial intelligence models could be trained for longer and on the brains of multiple people, they could improve quickly.
As UCSF’s research continued, Facebook was also paying other centers, like the Johns Hopkins Applied Physics Lab, to find out how to pump light through the skull to read neurons non-invasively. Like MRI, these techniques rely on the detection of reflected light to measure the amount of blood flow to regions of the brain.
It is these optical techniques that remain the main stumbling block. Even with recent improvements, some by Facebook, they are not able to pick up neural signals with sufficient resolution. Another problem, says Chevillet, is that the blood flow detected by these methods occurs five seconds after a group of neurons fires, making it too slow to control a computer.