Mind Reading with fMRI

From the Nature Editor's summary:

Recent functional magnetic resonance imaging (fMRI) studies have shown that, based on patterns of activity evoked by different categories of visual images, it is possible to deduce simple features in the visual scene, or to which category it belongs. Kay et al. take this approach a tantalizing step further. Their newly developed decoding method, based on quantitative receptive field models that characterize the relationship between visual stimuli and fMRI activity in early visual areas, can identify with high accuracy which specific natural image an observer saw, even for an image chosen at random from 1,000 distinct images. This prompts the thought that it may soon be possible to decode subjective perceptual experiences such as visual imagery and dreams, an idea previously restricted to the realm of science fiction.

The abstract from Kay et al., followed by one figure:

A challenging goal in neuroscience is to be able to read out, or decode, mental content from brain activity. Recent functional magnetic resonance imaging (fMRI) studies have decoded orientation, position, and object category from activity in visual cortex. However, these studies typically used relatively simple stimuli (for example, gratings) or images drawn from fixed categories (for example, faces, houses), and decoding was based on previous measurements of brain activity evoked by those same stimuli or categories. To overcome these limitations, here we develop a decoding method based on quantitative receptive-field models that characterize the relationship between visual stimuli and fMRI activity in early visual areas. These models describe the tuning of individual voxels for space, orientation and spatial frequency, and are estimated directly from responses evoked by natural images. We show that these receptive-field models make it possible to identify, from a large set of completely novel natural images, which specific image was seen by an observer. Identification is not a mere consequence of the retinotopic organization of visual areas; simpler receptive-field models that describe only spatial tuning yield much poorer identification performance. Our results suggest that it may soon be possible to reconstruct a picture of a person's visual experience from measurements of brain activity alone.


Figure Legend - The experiment consisted of two stages. In the first stage, model estimation, fMRI data were recorded while each subject viewed a large collection of natural images. These data were used to estimate a quantitative receptive-field model for each voxel. In the second stage, image identification, fMRI data were recorded while each subject viewed a collection of novel natural images. For each measurement of brain activity, we attempted to identify which specific image had been seen. This was accomplished by using the estimated receptive-field models to predict brain activity for a set of potential images and then selecting the image whose predicted activity most closely matches the measured activity.