They don’t have to think about it. Skilled typists rely on muscle memory to do what they do. Their brains associate the location of letters on the keyboard without conscious knowledge of where the letters actually are.
Now researchers have devised a system that allows a person to communicate directly with a computer from his or her brain. The approach enables communication at a rate more than twice as fast as previous typing-by-brain experiments.
Researchers at Stanford University performed the study on a 65-year-old man with a spinal cord injury. The researchers implanted an electrode array in the man’s brain. The scientists described the experiment recently in the journal Nature.
Typing by Just Thinking About It
“The big news from this paper is the very high speed,” says Cynthia Chestek, a biomedical engineer at the University of Michigan, who was not involved in the study. “It’s at least half way to able-bodied typing speed, and that’s why this paper is in Nature.”
For years, researchers have been experimenting with ways to allow people to directly communicate with computers with their thoughts. This kind of technology offers a life-giving communication method for people who are paralyzed and unable to speak.
Successful BCI typing-by-brain approaches so far typically involve a person imagining moving a cursor around a digital keyboard. Electrodes record brain activity. Machine learning algorithms decipher the patterns associated with those thoughts, translating them into typed words.
The fastest of these previous typing-by-brain experiments allowed people to type about 40 characters, or 8 words, per minute.
Imagining Hand Movements
That we can do this at all is impressive, but in real life that speed of communication is quite slow. The Stanford researchers were able to more than double that speed. They did it with a system that decodes brain activity associated with handwriting.
In the new system, the participant imagines the hand movements he would make to write sentences.
“We ask him to actually try to write—to try to make his hand move again, and he reports this somatosensory illusion of actually feeling like his hand is moving,” says Frank Willett, a researcher at Stanford who collaborated on the experiment.
A microelectrode array in the participant’s brain records the electrical activity of neurons as he tries to write. “He hasn’t moved his hand or tried to write in more than ten years and we still got these beautiful patterns of neural activity,” says Willett.
A machine learning algorithm then decodes the brain patterns associated with each letter. A computer displays the letters on a screen. The participant was able to communicate at about 90 characters, or 18 words, per minute.
The authors say people the same age as the study participant can type a bit faster. Typically, they type around 23 words per minute on a smartphone. Able-bodied adults can type on a full keyboard at an average of about 40 words per minute.
The Stanford researchers achieved the feat by repurposing a machine learning algorithm that was originally developed for speech recognition. The deep learning algorithm trained for few hours to recognize the participant’s neural activity when he imagined handwriting.
Neural networks are typically trained to recognize speech and images over tens of thousands of hours. They use audio data and millions of images, Willett says. So the challenge with the handwriting experiment was to achieve high accuracy with a limited amount of data.
To overcome this, the team applied data augmentation techniques, says Willett. “We only had the opportunity to collect maybe 100-500 different sentences that we could ask the participant to write,” Willett says.
“So we took those sentences and chopped them up into individual letters and rearranged them into an infinite number of different sentences, and we found that that really helped teach these algorithms,” he adds.
A Promising Breakthrough
It was also difficult to decode when, exactly, the man was writing a letter and when he wasn’t. To help with this, Willett and his team borrowed a tool from speech recognition. They used a hidden Markov model. This helped label the relevant data.
Once the data was labeled, the neural network could more easily be taught to associate certain patterns with letters. Willett says it’s the unique art of handwriting that makes it a faster way to communicate using BCI.
“Why it works so much better than plain typing…is because each handwritten letter has a different pen trajectory associated with it, and a very different pattern of finger motions and motor actions. This evokes a unique pattern of neural activity that’s easy to distinguish,” he says.
By contrast, point-and-click systems make straight line movements to different keys. This evokes similar patterns of neural activity that aren’t easily distinguished, which slows down the system, Willett says.
The techniques and algorithms presented in the experiment are applicable to other areas of research. These include connecting the brain to prosthetic hands, says Chestek at the University of Michigan.
“Regardless of whether this is the best way to do communication, the overall approach is really promising for motor control, generally,” Chestek says.
The algorithms, in their current form, have to be trained for and customized for each participant. They also have to be recalibrated over time because neurons tend to change over time. The electrode array may move around slightly.
As a next step, Willett says he hopes to reduce the amount of initial training time. He says researchers hope to one day develop a way for the algorithms to automatically recalibrate, too.
So, how fast can you type the word, “Amazing”?
Parts of this article originally appeared on the IEEE Spectrum.
