Highlights
- Scientists have developed a non-invasive ‘mind-captioning’ technique that turns brain activity into descriptive sentences.
- The model uses fMRI scans and language algorithms to predict what a person is seeing or recalling.
- The research could one day help people with speech impairments, but also raises privacy concerns.
Turning thoughts into sentences
Decoding the human mind may sound like science fiction, but researchers have taken a major step towards it. Scientists have developed a new artificial intelligence system that can translate a person’s brain activity into written descriptions of what they are seeing or imagining.
The method, known as ‘mind captioning’, analyses data from functional magnetic resonance imaging (fMRI), a non-invasive brain scan that measures blood flow to detect neural activity. The findings, published in Science Advances, show the technique can produce detailed sentences describing what’s happening in a person’s mind.
“This is hard to do,” said Alex Huth, a computational neuroscientist at the University of California, Berkeley. “It’s surprising you can get that much detail.”
How the mind-captioning model works
Researchers have previously managed to predict what someone is looking at or hearing using brain activity. However, understanding more complex visual scenes or abstract thoughts has remained a challenge.
To overcome this, Tomoyasu Horikawa and his team at NTT Communication Science Laboratories in Japan combined two types of AI models. The first model analysed text captions from more than 2,000 short videos and converted them into numerical “meaning signatures”. A second model was trained to match those meaning signatures to patterns of brain activity recorded from six volunteers as they watched the same videos.
Once trained, the decoder could read new brain scans and predict the meaning signature behind what the person was viewing. Another language model then generated sentences that best matched those decoded meanings.
For instance, when a participant watched a clip of a person jumping off a waterfall, the system gradually refined its guesses until it produced the sentence: “A person jumps over a deep waterfall on a mountain ridge.”
The same approach worked when participants recalled the videos from memory, showing that similar brain patterns are used for both viewing and remembering.
Potential to help communication
Because the process relies on non-invasive brain imaging, it could pave the way for new assistive tools that help people who have lost the ability to speak, such as those recovering from strokes or living with paralysis.
“If we can do that using these artificial systems, maybe we can help out people with communication difficulties,” said Huth, who previously worked on a related brain-decoding project in 2023.
Privacy and ethical questions
While the findings are promising, experts say the technology also raises concerns about mental privacy. If such systems become more advanced, they could, in theory, expose personal thoughts or emotions.
Both Horikawa and Huth stressed that current methods are limited; they require participants’ consent, controlled conditions, and cannot read private thoughts or intentions.
“Nobody has shown you can do that, yet,” said Huth.
