Category-Specific Cortical Activity Precedes Retrieval During Memory Search

^* To whom correspondence should be addressed. E-mail: polyn{at}psych.upenn.edu

Human memory can be characterized as an elaborate network of stored representations (1, 2). Recalling a particular event involves reactivating the constellation of representations that was active during that event, a phenomenon that Tulving has referred to as "mental time travel" (3). One of the major puzzles of human memory is how we enact this process of mental time travel. More concretely: When we are instructed to recall a particular event, how do we manage to select representations corresponding to that event, as opposed to representations from other events (4, 5)?

Several theorists have argued that recalling an event involves a process of contextual reinstatement (6, 7). When asked to recall memories of a certain type, a person activates knowledge about the general properties of those events and then uses this general knowledge to constrain the search for memories of the target events. For example, in trying to remember a trip to the zoo, a person could use their general knowledge of the kinds of animals that are typically found at zoos as a contextual cue for specific memories of seeing those animals. If specific details are recalled, these details can be used to further refine the retrieval cue, which leads to recall of additional details, and so on. Over time, the person continues to probe memory, and the set of representations that are active at recall increasingly comes to resemble the set of representations that were active during the targeted event. Whereas a number of behavioral memory studies have found evidence consistent with the contextual reinstatement hypothesis (8–10), this kind of evidence is necessarily indirect. We can infer (based on theoretical grounds) that the observed patterns of behavioral data arise from increased match between cues at test and stored memory traces, but these studies do not directly measure cue-trace match.

We used functional magnetic resonance imaging (fMRI) to more directly test the contextual reinstatement hypothesis. In neural terms, the contextual reinstatement hypothesis leads to a number of predictions. The most basic prediction is that, when subjects try to recall specific details from a particular episode or type of episode, the pattern of brain activity (during recall) will progressively come to resemble the pattern of activity that was present during the to-be-remembered episode. Furthermore, it should be possible to relate the reinstatement of brain activity to behavioral recall performance on a time-varying basis. The likelihood of recalling details from a particular episode (at a particular moment during the recall test) should be strongly related to how well subjects—at that moment—have reinstated activity from the to-be-remembered episode. According to the contextual reinstatement hypothesis, this association occurs because subjects use reinstated activity in a top-down fashion to cue for additional details (i.e., better reinstatement creates a better cue). However, this association could also occur if reinstated activity passively reflects recall of specific details (i.e., more recall leads to more reinstatement) and has nothing to do with cueing. To tease apart these ideas, we need to examine the temporal dynamics of how brain activity from the study phase is reinstated during recall. The contextual reinstatement hypothesis posits that if subjects use reinstated activity to cue memory, reinstatement should precede the recall of specific details. The alternative hypothesis (that reinstated brain activity is a passive reflection of the recall of specific details) posits that reinstated activity should occur during and after behavioral recall, but not before.

The main predictions of the contextual reinstatement hypothesis are summarized as follows: (i) Contextual reinstatement should build up gradually during the recall period. (ii) Fluctuations in contextual reinstatement should correlate with fluctuations in recall performance. (iii) Contextual reinstatement should precede recall of individual items. Recent neuroimaging studies of human memory retrieval have found that components of brain activity recorded during the study period are reinstated during the recall period, but these studies did not measure how contextual reinstatement changes over time during the recall test (11–15).

To test the predictions of the contextual reinstatement hypothesis, we designed a study that would allow us to directly measure contextual reinstatement in a time-varying manner. Unlike the aforementioned imaging studies (which used recognition or cued recall tests), our study used a free-recall paradigm in which subjects were asked to recall studied items in the absence of specific cues. The lack of specific environmental information driving retrieval in the free-recall paradigm places stronger demands on contextual reinstatement processes (16). Also, we used newly developed, multivariate pattern-analysis methods (17–20) to compare patterns of brain activity at the time of recall (test) to those observed during the initial encoding (study). This method increased our sensitivity in measuring how well study-phase brain activity was being reinstated at test.

Over the course of the experiment, subjects studied three lists, each of which contained 30 study items. Each list was composed of three different types of study items: photographs of famous faces, photographs of famous locations, and photographs of common objects. These photographs were presented with a name written in text above them (for example, a photograph of actor Jack Nicholson with the words "Jack Nicholson" above the picture). Subjects performed a different judgment on each class of stimuli to orient them to the salient features of those stimuli (21). Overall, the goal was to create a distinctive mental context associated with each stimulus class at study that could subsequently be tracked during the recall phase. At the end of the experiment, subjects were given a final free-recall test (lasting 3 min) where they were asked to recall all of the items they studied, in any order and regardless of category. fMRI data were acquired during both the study and recall periods (21).

The goal of the fMRI analysis was to track reinstatement (at the time of retrieval) of the contexts associated with studying face, location, and object stimuli. In order to do this, we first sought to identify patterns of brain activity associated with each category during study. Data analysis was carried out on an individual-subject basis. For each subject, we trained a neural-network pattern classifier to discriminate between patterns of whole-brain activity associated with face, location, and object stimuli during the study phase (21). Next, the trained network was used to classify whole-brain patterns of activity in the same individual (each pattern corresponding to 1.8 s of scanning) during the recall period. For each brain volume (scan), the classifier was used to produce an estimate of how well that scan matched the patterns of activity associated with the face, location, and object contexts from the study phase. By applying the classifier to successive brain scans acquired during the recall phase, we were able to derive a graded, time-varying estimate of the extent to which subjects were reinstating the face, location, and object study contexts. These time-varying estimates of the reinstatement of each context were then compared to the record of actual verbal recalls made by the subject.

Consistent with the contextual reinstatement hypothesis, we found that category-specific brain activity during the final free-recall period corresponded to the category of verbal recall (Fig. 1). This correspondence was quantified by correlating the classifier's estimates of category-specific activity (for each 1.8-s scan) with the record of verbal recalls for each category. A nonparametric statistical analysis confirmed that the classifier estimates correlated more with recall from the matching category than with recall from other categories; this effect was significant (P < 0.05) in seven of the nine individual subjects. A second nonparametric statistical analysis revealed that these effects were significant at the group level at P < 0.001 (table S3) (21).

Fig. 1. Correspondence between the classifier's estimates of contextual reinstatement and verbal recalls for a single representative subject. Time is represented on the x axis; each time point represents one complete brain scan (lasting 1.8 s). For each brain scan, the classifier produced an estimate of the match between the current testing pattern and each of the three study contexts (strength of estimate appears on the y axis). The blue, red, and green lines correspond to the face, location, and object classifier estimates, respectively. The blue, red and green dots correspond to the face, location, and object recalls made by the subject (larger dots correspond to multiple items recalled during a single scan). The recall events are shifted forward by three time points to account for lag to the peak hemodynamic response (27). For illustrative purposes, the classifier estimate lines were temporally smoothed by replacing each point with the mean of that point and the immediately neighboring estimates (however, the correlations reported here were computed based on unsmoothed classifier estimates, as was the event-related average shown in Fig. 2). [View Larger Version of this Image (36K GIF file)]

An event-related average of the classifier estimates, constructed based on a subset of recall events (see below), revealed that category-specific brain activity reliably appeared before the verbalization of the recalled item (Fig. 2). This anticipatory rise was significant beginning at 5.4 s before recall, suggesting that subjects cue with general information about a category when trying to recall specific items from that category. To minimize the possibility that estimates of category-related activity (before recall) would be influenced by other recall events, we only included recall events in the event-related average if no same-category items were recalled in the preceding 14.4 s. For qualifying recall events, Fig. 2 plots the average classifier estimate for the currently recalled category in the time intervals surrounding the time of recall (t = 0). The anticipatory rise was computed relative to a baseline plot, showing classifier estimates for categories that were not recalled at t = 0 or in the 14.4 s preceding t = 0. This analysis was not corrected for the lag in the hemodynamic response; thus, the increase in category-related brain activity most likely preceded recall by substantially more than the 5.4 s observed in the blood oxygen level–dependent response.

Fig. 2. Event-related average of the classifier's estimates of contextual reinstatement for the time intervals surrounding recall events. Recall events were excluded from the event-related average if a same-category recall was made in the preceding 8 scans (equivalent to 14.4 s). The dotted line at t = 0 represents the scan on which the verbal recall was made. The "currently recalled" plot (black line) was constructed by averaging together classifier estimates for categories that were recalled at t = 0, but not in 14.4 s preceding t = 0. The "baseline" plot (purple line) was constructed by averaging together classifier estimates for categories that were not recalled at t = 0 or in the 14.4 s preceding t = 0. The "recently recalled" plot (red line) was constructed by averaging together classifier estimates for categories that were not recalled at t = 0, but were recalled (at some point) during the 14.4 s preceding t = 0 (28). The three plots have not been shifted to account for hemodynamic lag effects. Statistical comparisons focused on the difference between the currently recalled and baseline plots, because these two plots were matched for lack of recalls in the 14.4 s preceding t = 0. The currently recalled and baseline plots differ significantly starting at t = –3 (i.e., 3 scans or 5.4 s before the verbal recall). Significance was calculated using a one-tailed, paired-samples t test on the within-subject difference between the two plots (points marked with stars and circles differ at P < 0.01 and P < 0.05, respectively). [View Larger Version of this Image (27K GIF file)]

Given the neural network classifier's success at predicting overt recall based on patterns of brain activity, we ran an analysis to determine which brain regions were contributing to the classification. Figure 3 presents maps displaying which voxels exerted the strongest influence in detecting each of the three study contexts (21). The four representative axial slices show that canonical category-selective areas [such as the fusiform face area and parahippocampal place area (22–24)], in addition to textured patterns in other brain areas, activate during the study of these item types.

Fig. 3. The classifier-derived importance maps. Each voxel was assigned an "importance value" for each study context based on its influence on the classifier's estimate of reinstatement for that context (21). These maps average over all 9 subjects (whereas all classification was done on a within-subject basis). Each row depicts the map for a different study context, and each column depicts a different axial slice, spaced 12 mm apart. Voxels with positive importance values are colored red; voxels with negative importance values are colored blue. The colors fade into transparency as importance values approach zero, as shown on the color bar. FG, fusiform gyrus; PG, parahippocampal gyrus; MFG, medial frontal gyrus; PC, posterior cingulate; MTG, middle temporal gyrus (24). [View Larger Version of this Image (136K GIF file)]

The finding that canonical category-selective areas were contributing to the classification suggests that these areas could be driving the observed increase in contextual reinstatement before recall. However, follow-up analyses indicated that voxels outside of peak category-selective areas are also important for establishing this result (21). We identified peak category-selective regions of interest (ROIs) by using a group general linear model analysis, applied to study-phase data (see table S5 for a list of identified regions). In one analysis, we tracked category-related activity at recall by computing (at each time point) the average activity of each category-specific ROI. In another analysis, we used our standard pattern-classification procedure, but we limited the analysis to the union of the voxels in the peak category-selective ROIs. Both ROI-based analyses showed a significant correspondence between category-specific brain activity and behavior (tables S7 and S8). However, in both cases the observed correspondence was smaller than the correspondence obtained using our primary analysis method, and neither of the ROI-based analysis methods was sensitive enough to detect the anticipatory rise in category-selective activity shown in Fig. 2 (figs. S10 and S11). Taken together, these results suggest that including voxels outside of the peak category-selective ROIs improves our ability to detect subtle changes in the reinstatement of category-related activity.

The work described here is one of a growing number of fMRI studies illustrating the benefits of multivoxel pattern-classification techniques (17–20, 25, 26). These studies have demonstrated that, by efficiently extracting the information present in multivoxel patterns of brain activity, it is possible to detect subtle distinctions between cognitive states using relatively thin time slices of brain data (on the order of seconds). Whereas previous applications of classification techniques have focused on brain activity elicited by specific perceptual cues, our study shows that classification algorithms can be used to extract a time-varying trace of the subjects' cognitive state as they search through memory in the absence of specific cues. Our results ground Tulving's speculations about mental time travel in neural fact. As subjects search for memories from a particular event, their brain state progressively comes to resemble their brain state during the sought-after event, and the degree of match predicts what kinds of information the subjects will retrieve. By providing a direct view of how subjects are cueing memory, the methods presented here constitute a powerful new tool that researchers can use to test and refine theories of how people mine the recesses of the past.

References and Notes

THIS ARTICLE HAS BEEN CITED BY OTHER ARTICLES:

ADVERTISEMENT