The effect of object processing in content-dependent source memory
© Park et al.; licensee BioMed Central Ltd. 2013
Received: 11 February 2013
Accepted: 9 July 2013
Published: 13 July 2013
Skip to main content
© Park et al.; licensee BioMed Central Ltd. 2013
Received: 11 February 2013
Accepted: 9 July 2013
Published: 13 July 2013
Previous studies have suggested that the study condition of an item influences how the item is encoded. However, it is still unclear whether subsequent source memory effects are dependent upon stimulus content when the item and context are unitized. The present fMRI study investigated the effect of encoding activity sensitive to stimulus content in source memory via unitization. In the scanner, participants were instructed to integrate a study item, an object in either a word or a picture form, with perceptual context into a single image.
Subsequent source memory effects independent of stimulus content were identified in the left lateral frontal and parietal regions, bilateral fusiform areas, and the left perirhinal cortex extending to the anterior hippocampus. Content-dependent subsequent source memory effects were found only with words in the left medial frontal lobe, the ventral visual stream, and bilateral parahippocampal regions. Further, neural activity for source memory with words extensively overlapped with the region where pictures were preferentially processed than words, including the left mid-occipital cortex and the right parahippocampal cortex.
These results indicate that words that were accurately remembered with correct contextual information were processed more like pictures mediated by integrated imagery operation, compared to words that were recognized with incorrect context. In contrast, such processing did not discriminate subsequent source memory with pictures. Taken together, these findings suggest that unitization supports source memory for both words and pictures and that the requirement of the study task interacts with the nature of stimulus content in unitized source encoding.
Neural correlates of successful memory formation have been investigated with the functional magnetic resonance imaging (fMRI) subsequent memory procedure, in which neural activity during encoding for later remembered items is compared with activity for later forgotten items. Subsequent memory effects are often expressed as enhanced encoding activity for remembered items in several brain regions, such as the left prefrontal cortex, the fusiform area, and the medial temporal lobe (MTL) [1–3]. Further, subsequent memory tends to reveal the influence of the study condition in which the item is experienced at the cortical level. Seminal behavioral studies of memory have suggested that memory representation results from how an item is processed at encoding [4–6]. From this perspective, episodic memory is the by-product of study processing based on the circumstances in which an event is experienced. In accordance with this theoretical principle, previous fMRI studies of subsequent memory effects have shown that cortical encoding activity for successful memory of an item depends upon the nature of the study task, the modality of the study item, and the material of the study item [2, 7–9].
In these studies, the differences in study conditions, including study task (semantic vs. non-semantic) or stimulus content such as modality (visual vs. auditory) and material (verbal vs. pictorial), dissociated the cortical regions where neural activity for successful memory formation was identified. For example, subsequent memory of words was predicted in the left inferior prefrontal cortex and the left fusiform region, whereas encoding activity for pictures was identified in the right inferior prefrontal cortex and bilateral fusiform areas . The MTL effects of memory formation also showed left lateralized activation for words and right lateralized activation for pictures and faces . Semantic encoding of successfully remembered words was related with activity in the left and medial prefrontal regions; however, non-semantic subsequent memory effects were found in the right prefrontal cortex, bilateral intra-parietal sulci and bilateral fusiform gyri . Moreover, subsequent memory effects for visually presented words were reported in the right fusiform cortex, while auditory subsequent memory effects were identified in bilateral superior temporal sulci .
However, relatively little is known about whether the formation of associations, as opposed to encoding of single items, also exhibits study condition-specific effects of encoding. The extant findings of task-dependent effects in memory associations are less consistent. For example, a study that compared task effects on associative memory of word pairs using semantic and phonological tasks found that extensive activity in the left inferior prefrontal cortex predicted word pairs that were later successfully remembered as an intact pair in both encoding tasks; task-specific effects, however, were negligible . Another study that compared semantic versus perceptual study conditions for associative memory of word pairs reported that the same left inferior prefrontal regions were involved in subsequent associative memory effects common to both conditions; however, perceptual associative encoding effects were also found in the left temporo-occipital and bilateral parietal cortices as well as the right parahippocampal cortex .
Other studies of subsequent source memory (item-source associations) showed that neural correlates of source memory differed depending upon study task or stimulus content [13, 14]. Subsequent source memory of words studied with a semantic task was localized in the left temporo-occipital cortex, while successful source memory effects with a non-semantic encoding task were found in the right temporo-occipital region, revealing hemispheric differences between subsequent source memory effects that depend upon the study task . Subsequent source memory effects also varied by stimulus content, such that source encoding activity for words were found in the left inferior prefrontal cortex, while successful source memory effects for objects were identified in the left perirhinal cortex .
In addition to these cortical effects, the MTL is known to be critical for forming associations [12, 15, 16]. From the memory process view, the MTL subregions are specialized for different memory processes. That is, the hippocampus and the parahippocampal cortex are integral to the formation of associations based on recollection, whereas the perirhinal cortex tends to support familiarity-based item recognition [17–21]. Functional dissociation within the MTL for different memory processes has been supported by findings that increased activity in the hippocampus predicts subsequent recognition based on recollection (e.g., source memory) but not familiarity [22–24], whereas familiarity-based judgments (e.g., item memory) have been related with increased perirhinal activity during encoding [18, 22, 23]. It was further proposed that the perirhinal cortex could support source memory based on familiarity through unitization of an item and context . As opposed to the conventional way of binding (i.e., non-unitized encoding), in which separate item and context are to be combined through associations, unitization is assumed to bind context to the item as an item feature. From this view, source memory through unitization may be supported by familiarity. Further, source memory via unitization was supported by perirhinal activity, which is consistent with the proposal of the perirhinal cortex for familiarity-based source recognition [25, 26].
It has also been suggested that MTL subregions are involved in encoding of heterogeneous classes of stimulus content [22, 27]. First, the hippocampus is involved in processing of content-independent binding. For instance, encoding activity in the anterior hippocampus was identified for recollection of both objects and scenes, reflecting the content-independent role of the hippocampus . Other MTL subregions selectively engage in processing of different types of content. Specifically, the parahippocampal cortex is selective to visuo-spatial processing such as scene processing [29–33]. The perirhinal cortex is involved in object processing, and the role of perirhinal cortex in object encoding was found across different recognition judgments (i.e., Remember, Know, and New) and source memory . Similarly, a MTL subsequent memory study that examined source encoding with objects and scenes reported the dissociation of the MTL cortex by content: the perirhinal cortex contributed to source encoding for objects, but the parahippocampal cortex supported source encoding of scenes .
Though previous studies suggested content-dependent encoding effects, most of those studies examined content-dependent effects with non-unitized encoding tasks or focused on MTL activity. As alluded to earlier, unitization of an item and context may change the way that the item and context are bound; therefore, unitized source memory may be supported by different encoding activity at the cortical level. Considering that the memory process involved in unitized encoding may be different from the memory process for non-unitized encoding, an interesting question is whether subsequent memory effects specific to stimulus content would appear with unitized encoding both at the cortical and MTL levels. This is important, as it investigates how encoding activity specific to each trial (i.e., stimulus content) is influenced by study processing across trials (i.e., encoding operation) in forming associations. Previous studies that used non-unitized encoding tasks reported content-independent effects in the hippocampus and content-dependent effects in the cortical areas. However, it is not clear whether unitized encoding is sensitive to content-dependent subsequent memory effect. Additionally, given that non-unitized encoding tends to elicit hippocampal activity for source memory formation across different types of content, it is of interest whether unitized encoding would show similar MTL source memory effects across verbal and pictorial content. In what follows, we report a study of the formation of source memory for words and pictures via unitization.
Participants were presented a list of study items (i.e. words and pictures) on a colored background and were instructed to imagine the item in the background color. This meant that participants actively had to generate a mental presentation for words but not necessarily for pictures. Thus, imagery-based operations required by the study task would have different effects depending on whether the content was verbal or pictorial. By comparing encoding activity of words and pictures varied by later source memory performance (source correct vs. source incorrect), we investigated the effect of imagery-based operations in source memory by stimulus content.
Previous studies of subsequent memory effects have reported encoding activity for perceptual processing of visually-presented stimuli in the parietal and occipital cortices, as well as the fusiform cortex [10, 13, 34–36]. Further, perirhinal activity has been reported for the formation of source memory with objects through unitization [29, 31, 32]. Thus, it is likely that parietal/occipital activity as well as perirhinal activity would contribute to source encoding of both types of content. On the other hand, participants had to generate a visuo-spatial representation of the word and place it in the color context for words, whereas pictures were already presented in the necessary visual form. That said, visuo-spatial processing and binding would be critical for forming successful source memory with words by engaging in the study task, but less so with pictures. Then, neural activity for visuo-spatial processing and binding would distinguish words with contextual information from words without correct context to a greater extent than such activity with pictures, which would emerge as content-dependent subsequent source memory effects. Further, as pictures were presented in the pictorial form during study, the brain regions that showed picture-preferential processing (pictures > words) regardless of later source memory effects would be the areas where subsequent source memory effects selective to words would be identified.
For subsequent source memory effects independent of stimulus content, unitization of item and color would be supported by encoding of visually presented objects, as evidenced by increased activity in the parieto-occipital cortices, the fusiform gyrus, and the perirhinal cortex, in addition to the established role of hippocampal activity in source memory formation overall.
Study requirement for engaging in visuo-spatial processing and binding for source memory formation would affect words more than pictures, and this would call for greater subsequent source memory effects selective to words in the ventral visual pathway and the parahippocampal cortex.
Subsequent source memory effects for words would indicate that these words were processed more like pictures, based on the overlap of regions where pictures were preferentially processed compared to words such as parietal and occipital regions.
Mean test response proportions (SEM) and mean study reaction times (in ms) as a function of stimulus content and memory judgment
Study reaction times are also shown in Table 1, segregated by stimulus content and later source memory judgment. A 2 × 2 ANOVA with stimulus content (word, picture) and source judgment (source correct, source incorrect, item miss) as factors was conducted on these data. The ANOVA showed the main effect of stimulus content, indicating longer reaction times for pictures than for words, F [1,23] = 8.59, p < .005. However, there was neither a main effect of source judgment nor an interaction between stimulus content and source judgment on study times.
Analyses of subsequent source memory effects were based on contrasts between encoding activity of study items that were later accurately endorsed with the studied color (source correct) as opposed to encoding activity of study items that were later recognized but with an incorrect color (source incorrect). Item misses were not included in the analysis due to an insufficient number of trials.
Subsequent source memory effects independent of stimulus content
Coordinates (x y z)
Z (# of voxels)
L superior frontal gyrus
L inferior frontal gyrus
L inferior orbito-frontal cortex
L angular gyrus/Inferior/superior parietal lobule/ Mid occipital cortex
L fusiform gyrus
L perirhinal cortex/anterior hippocampus
Subsequent source memory effects selective to words
Coordinates (x y z)
Z (# of voxels)
L medial frontal gyrus
L middle frontal gyrus
L fusiform gyrus
L middle occipital gyrus
L parahippocampal cortex
R parahippocampus/ hippocampus
The present study investigated content-dependent source memory effects by means of an integrated imagery operation. Successful source memory that did not differ between the two stimulus content types was predicted in the left inferior frontal gyrus (LIFG), the lateral parietal/occipital lobes, and the fusiform gyrus. In addition, MTL subsequent memory effects independent of stimulus content were identified in the left perirhinal cortex extending to the left anterior hippocampus. The formation of source memory selective to words recruited activity in the left frontal cortex, the ventral visual stream, and the bilateral parahippocampal cortices extending to the vicinity of the right posterior hippocampus. Picture-dependent subsequent source memory effects did not emerge over threshold. Finally, subsequent source memory effects for words extensively overlapped with the areas where pictures were preferentially processed.
Content-independent subsequent source memory effects were identified in a number of cortical regions. First, subsequent source memory effects were evident in the LIFG extending to the superior frontal regions. LIFG effects have been found in the formation of associative memory [11, 12, 37, 38] and source memory [14, 34] with both semantic and non-semantic encoding tasks. The current finding of LIFG involvement in integrated source encoding is consistent with the proposal that the LIFG is crucial for the formation of associations regardless of the type of study processing [11, 38]. Encoding activity for source memory independent of stimulus content was also found in the left lateral parietal cortex, spanning from the angular gyrus to the superior parietal/occipital regions. These regions, especially the intra-parietal sulcus, are known to engage in goal-directed attentional processing [39, 40], multi-modal processing [35, 41], and perceptual processing [35, 36, 42], in addition to encoding of both item and source memories [7, 10, 18, 43]. Previous studies also reported subsequent source memory effects in the LIFG and the left parietal regions with non-unitized encoding tasks. Thus, the present findings of the LIFG and the left parietal activity through integrated binding of item and context indicates that neural activity in these regions supports the formation of successful source memory mediated by perceptual processing through unitized encoding as well as non-unitized encoding.
In parallel with these cortical effects, content-independent subsequent source memory effects were identified in the left perirhinal cortex extending to the left anterior hippocampus, which is consistent with previous reports of perirhinal activity for source memory formation with objects [14, 29, 33] and prior findings of hippocampal activity for source encoding in general [12, 31, 32]. The formation of associations via unitization of an item and its contextual details is known to be supported by perirhinal activity, as the unitized representation of the item with context as an object can be recognized based on familiarity [44, 45]. The current finding of perirhinal activity in source encoding with both words and pictures extends previous findings of perirhinal involvement in associations [18, 19, 32, 33, 46] in that source memory for both verbal and pictorial content is supported by perirhinal activity through unitization. Further, activity in the hippocampus has been found in subsequent source memory in a content-general manner, which confirms the established role of the hippocampus in memory associations overall [19, 28, 31].
While investigating content-dependent subsequent source memory effects, we found only word-dependent effects. These effects were evident in the visual ventral stream, the region critical for object identification (i.e., the “what” pathway), which suggests that accurate source memory of words was mediated by visuo-spatial object processing more so than inaccurate source memory of words. That is, words that were subsequently remembered with source information tended to be the ones with more object-like processing during encoding, whereas words that were not accompanied with correct source information tended to be the ones that were not visualized well enough for imagery operation to occur at a level comparable to those that elicited source correct judgments.a However, the level of object processing did not distinguish subsequent memory of pictures between source correct versus source incorrect. Considering that pictures were presented in the visual object form during encoding, object processing would likely occur for all pictures regardless of whether they were accompanied with correct contextual information or not, hence the smaller impact of object processing in subsequent source memory effects with pictures. This interpretation is supported by the significant interaction between stimulus content and source judgment revealing the differences in parameter estimates only with source correct words versus source incorrect words (see Results). In addition, insular activity findings complement previous findings of insular involvement in multi-modal processing of stimuli  as well as source encoding [34, 47].
Word-dependent source memory effects were also identified in the bilateral parahippocampal cortices, extending to the vicinity of the right posterior hippocampus. These MTL effects reflect that activity of word-dependent source memory exhibits not only extensive object processing but also additional visuo-spatial and contextual processing for the words that were later remembered with correct source, compared with words without correct source, consistent with prior reports of parahippocampal activity for both spatial and nonspatial contexts [46, 48]. On the other hand, there was no overlap between word-dependent source memory effects and the region where words were preferentially processed than pictures. Prior studies have shown overlap of content-dependent subsequent memory effects in the areas where corresponding content are preferentially processed [10, 38, 43]. The current, seemingly contradictory findings illustrate that successfully remembered words with accurate context are indeed the outcome of encoding in more of an ‘object’ manner due to the requirement of the study task. These findings suggest that adopting pictorial processing by engaging in the imagery-based encoding task was crucial for the formation of successful source memory only with words due to the interaction between the nature of study processing and the stimulus content.b In sum, the present findings demonstrate that the requirement of trial-general study processing influences the way that trial-specific stimulus content is experienced. Importantly, the present findings show that unitization is supported for different types of stimulus content.
We investigated neural activity specific to stimulus content via integrated source encoding with words and pictures. As predicted, subsequent source memory effects were evident in the left lateral frontal and parietal cortex as well as the left perirhinal cortex extending to the hippocampus, which is indicative of the significance of these regions in source memory formation for integrated object-context. Content-dependent source memory effects were found only with words in the ventral visual stream and the parahippocampal cortex, which reflects more visuo-spatial processing and contextual binding with words for successful source memory due to the requirement of imagery encoding operation. Collectively, the present findings demonstrate the effect of trial-general study processing in subsequent source memory effects depending on stimulus content through unitization.
Twenty-four subjects participated in the experiment (16 females; 18–29 years). They were recruited from the University of Texas−Arlington community and compensated for their participation. All subjects reported being right−handed, being native English speakers, and having no history of neurological disease. Informed consent was obtained prior to participation. The experiment was approved by the University of Texas−Arlington and University of Texas Southwestern Medical Center Institutional Review Boards.
The experimental stimulus pool consisted of 175 concrete words (4–9 letter nouns) and 175 line-drawings depicting nameable objects. Three-hundred critical items were selected from the stimulus pool to construct the study and test lists. For each subject, a study list of 200 critical items (100 words and 100 line-drawing pictures) was created from the pool by random selection of items without replacement. Critical items were pseudo-randomly assigned to one of four color backgrounds (blue, green, red, or yellow), 25 words, and 25 pictures for each color. All words were presented in black, all pictures were presented in black-white line-drawing form, and both words and pictures were overlaid on the colored background. Each study list comprised an intermixed presentation of critical items with 100 null trials interspersed among the experimental trials. A test list was consisted 200 studied items intermixed with 100 new items (50 words and 50 pictures). Test items were presented on a gray background. Both study and test list sequences were constrained by no more than four consecutive presentations of a type of content or background color. An additional 50 items, 25 words, and 25 pictures were used for practice trials. The composition and ordering of the study and test lists were made anew for each subject.
All items were projected onto a screen viewed by subjects via a mirror mounted on the scanner head. For the study list, critical items were presented on one of four color backgrounds, while null events were presented on a gray background. For the test list, all items were presented on the gray background. All items were presented at a maximum visual angle of 7.1° × 7.1°.
Subjects were given instructions and practice for experimental tasks prior to the experiment proper. The experiment consisted of two study-test cycles. For each study phase, a fixation cross was presented for 200 ms as a warning signal for the upcoming item, followed by a study item superimposed on the colored background for 2000 ms. The study item was replaced by a response prompt (‘+’) for 1300 ms prior to the onset of the next trial. Stimulus onset asynchrony was 3500 ms. Subjects were instructed to form an image of the object (e.g., ‘UMBRELLA’) in the background color (e.g., ‘RED’) and to make a pleasantness judgment for the colored object (‘RED UMBRELLA’). Note that the pleasantness judgment was made based on the mental image that integrated object and color. This study instruction was aimed to facilitate unitization of the object and color context. Subjects indicated their judgments by pressing a button with the index or middle finger of their right hand. The finger assignment to each response was counterbalanced across subjects. Four colors (blue, green, red, or yellow) were used for backgrounds, and 50 items (half words and half pictures) were presented in each colored background. For null trials, a fixation character (‘+’) was continuously displayed for 3500 ms on a gray background, and no response was required.
The test phase followed approximately 5 min after the end of the study phase. Test items were presented on the gray background. The content of the stimulus was kept consistent between study and test (e.g., if a stimulus was studied in a word form, it was also tested in the word form). Subjects were asked to judge whether the test item was previously studied in the experiment and, if it had been studied, to indicate in which color it was presented; subjects responded using a one-step source recognition response. Each study and test session lasted about 20 minutes.
A Philips Achieva 3T MR scanner (Philips Medical Systems, Andover, MA) fitted with a 32-channel RF receiver head coil was used to acquire both T1-weighted anatomical volume images (256 × 238 matrix, 1 mm3 voxels) and T* 2-weighted echo-planar images (EPIs) (80 × 80 matrix, 3 mm × 3 mm in-plane resolution, axial acquisition, flip angle 70°, TE 30 ms) per volume. Each EPI volume comprised thirty 3 mm-thick axial slices acquired in a descending sequential order and separated by 1 mm, providing coverage of almost the entire brain. Data were acquired during the study and test phases in four scanning sessions comprising 270 volumes each, with a repetition time (TR) of 2 s. Five additional volumes were collected at the beginning of each run, but these were discarded to allow for T1 equilibration. The 3.5 s SOA allowed for an effective sampling rate of the hemodynamic response of 2 Hz.
Data preprocessing and statistical analyses were performed with Statistical Parametric Mapping (SPM 8, Wellcome Department of Cognitive Neurology, London, UK: http://www.fil.ion.ucl.ac.uk), implemented in MATLAB 9 (Mathworks, Natick, MA). For each subject, functional images were registered to the first image of each scan session and temporally realigned to the middle slice using sinc interpolation. The resulting data were normalized to a standard EPI template based on the Montreal Neurological Institute (MNI) reference brain and resampled into 3 mm isotropic voxels using nonlinear basis functions . The normalized images were smoothed with an isotropic 8 mm full-width half-maximum Gaussian kernel. The time series in each voxel were high-pass filtered to 1/128 Hz to remove low-frequency noise and scaled to a grand mean of 100 across both voxels and scans. T1-weighted anatomical images were coregistered to the mean EPI volume and normalized to a standard T1 template of the MNI brain. After normalization, an across-subject mean anatomical image was created as the study-specific template for identifying brain regions.
Statistical analysis was performed on the study phase data using a two-stage mixed effects model for investigating encoding activity associated with content-dependent source memory effects through unitization of the item and context. In the first stage, neural activity was modeled by delta functions at stimulus onset. The event-related blood oxygen-level dependent (BOLD) response was modeled by convolving these neural functions with a canonical hemodynamic response function (HRF) and its temporal and dispersion derivatives. In addition, six regressors were modeled for movement-related variance, and session-specific constant terms were employed to model the mean image intensity in each session. Parameter estimates for events of interest were measured for each subject using a General Linear Model. Non-sphericity of the error covariance was accommodated by an AR(1) model in which the temporal autocorrelation was estimated by pooling over suprathreshold voxels . The parameters for each covariate and the hyperparameters governing the error covariance were estimated using Restricted Maximum Likelihood (ReML). In the second stage, linear contrasts of these subject-specific parameter estimates were computed, treating subjects as a random effect.
For the analysis of subsequent source memory effects by stimulus content, four events of interest were defined: ‘word-source correct’ (studied words that were recognized with correct study color); ‘word-source incorrect’ (studied words that were recognized albeit with incorrect study color); ‘picture-source correct’ (studied pictures that were accurately judged with study color); and ‘picture-source incorrect’ (studied pictures that were responded with incorrect color). All other study trials including item misses (studied items that were incorrectly judged as new) and no responses were modeled as events of no-interest, due to an insufficient number of trials.c
At the whole brain level, only effects surviving a voxel-wise threshold of p < .001 and a corrected cluster-wise threshold of p < .05 (greater than 16 voxels) were interpreted. When exclusive masking was applied to identify voxels where effects were not shared between two contrasts, the mask threshold was set at a two-tailed threshold of p < .1. Note that the more liberal the threshold of an exclusive mask, the more conservative the masking procedure. The threshold for the inclusive mask was set to p < .01. The peak voxels of clusters exhibiting reliable effects are reported in MNI coordinates. For the regions identified from main analyses, region-specific parameter estimates were extracted by peak voxel activity of a cluster for each subject and subjected to group-level statistical tests. The significance level for the region-specific parameter estimates was set to p < .05.
aWe thank Reviewer for clarification of this interpretation.
bAt the MTL level, subsequent source memory effects dependent on words may be accounted by recollection. That is, words that were recognized with correct source information recruited more activity related with recollection process in the parahippocampal cortex and the hippocampus due to binding of the item and context, compared with words without source memory. To put it differently, source correct words were more likely the words that were accompanied with the recollection process of binding the item and context compared with source incorrect words, whereas subsequent source memory dependent on pictures was not significantly distinguished by recollection process. We thank Reviewer for this suggestion.
cOnly 7 out of 24 participants had sufficient numbers of trials (> 10) for both words and pictures.
We thank Kellen Gandy and James Schaeffer for help in of the manuscript preparation.
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.