- Research article
- Open Access
A transition from unimodal to multimodal activations in four sensory modalities in humans: an electrophysiological study
BMC Neuroscience volume 9, Article number: 116 (2008)
To investigate the long-latency activities common to all sensory modalities, electroencephalographic responses to auditory (1000 Hz pure tone), tactile (electrical stimulation to the index finger), visual (simple figure of a star), and noxious (intra-epidermal electrical stimulation to the dorsum of the hand) stimuli were recorded from 27 scalp electrodes in 14 healthy volunteers.
Results of source modeling showed multimodal activations in the anterior part of the cingulate cortex (ACC) and hippocampal region (Hip). The activity in the ACC was biphasic. In all sensory modalities, the first component of ACC activity peaked 30–56 ms later than the peak of the major modality-specific activity, the second component of ACC activity peaked 117–145 ms later than the peak of the first component, and the activity in Hip peaked 43–77 ms later than the second component of ACC activity.
The temporal sequence of activations through modality-specific and multimodal pathways was similar among all sensory modalities.
In previous studies using magnetoencephalograms (MEGs) to monitor tactile , auditory , visual [3, 4] and pain [5, 6] systems, we found very similar mechanisms of sensory processing among these sensory modalities. In brief, several 'early' activities appear serially with a time delay of about 4 ms at each step followed by one or two 'late' activities. In general, the 'early' activity reverses polarity twice with an interval of 10 ms, which results in a characteristic triphasic waveform, while the 'late' activity is long-lasting without a polarity reversal at such a short interval . For example, following tactile stimulation, 'early' activations are elicited in area 3b, area 1 and the posterior parietal cortex in this order with a delay of 3–4 ms between each step, and then a long-lasting 'late' activity is evoked in the secondary somatosensory area. We postulate that a basic role of the 'early' activity is to receive inputs from the thalamus or convergent inputs from the thalamus and/or adjacent cortical areas and to send this information to the next point quickly, while the long-lasting 'late' activity is involved in recognition of the stimuli .
In the present study, we sought to compare mechanisms of sensory processing at latencies later than the 'late' activity among these sensory modalities (vision, audition, touch and pain). At first, we expected there to be unimodal and multimodal activities. Although there is a large and growing number of studies on multimodal interaction using electroencephalography (EEG) and MEG [8–13] as well as multimodal activation and interaction using functional magnetic resonance imaging (fMRI) [14–17], it is unclear whether or not the timing of the transition from unimodal to multimodal cortical activations is different among modalities. The present study manipulated the interstimulus interval (ISI) at three levels to examine the transition from unimodal and multimodal activations. In general, evoked potentials around and after 100 ms, like N1 and P3a/P3b, are more sensitively increased by increasing the ISI than earlier responses. If the difference in response amplitude between different ISIs shows the same scalp distribution among modalities, the difference in amplitude should originate from the same generator. In addition, a source analysis is helpful to estimate the location of the generator. In general, the manipulation of the ISI more strongly affects P3a and P3b, and the non-specific activities of N1, which are considered indices of orienting attention , compared with activities ("late" activity) within 100 ms. Because the original N1 response in any modality largely includes modality-specific activities showing a different scalp distribution depending on the modality, a comparison between the scalp distributions of the original N1 might not provide a clear-cut result. Therefore, we chose a simple manipulation of the ISI to extract more effectively the non-specific activities, to obtain a clearer result and to enable us to estimate more reliably and simply the location of the activity. Another reason for this choice is that the activities obtained by manipulating the ISI may be associated with orienting attention and later processes reflected by the non-specific N1 and P3a/P3b.
We expected the non-specific, possibly multimodal, activities obtained by manipulating the ISI are clearly found at later than 100 ms and have the same scalp distribution among modalities, but the "late" activities within 100 ms to have a different scalp distribution among modalities. The multimodal activations were expected to be in the anterior cingulate gyrus or hippocampus on the basis of a large number of previous studies performing source analyses [5, 19–25] and intracranial recordings [26–29], whereas unimodal activities ("late" activities) are estimated to be in areas specific to each modality. Of final and special interest was whether or not the timing of the transition from unimodal to multimodal activations is the same among modalities.
The experiment was performed on 14 (four females and ten males) healthy right-handed volunteers, aged 23–52 years (mean, 32 ± 8). The study was approved in advance by the Ethics Committee of the National Institute for Physiological Sciences, Okazaki, Japan, and written consent was obtained from all the subjects.
The EEG activity was recorded using 27 scalp electrodes placed on Fp1/Fp2, F3/F4, F7/F8, F9/F10, C3/C4, T7/T8 (T3/T4), T9/T10, P3/P4, P7/P8 (T5/T6), P9/P10, O1/O2, Fpz, Fz, Cz, Pz and Oz according to the 10–10 system. The reference electrode was placed on the nose. The impedance of the electrode was kept below 5 kΩ. The EEG signals were recorded with a bandpass filter of 0.1–100 Hz at a sampling rate of 1000 Hz, and then digitally filtered with a 70-Hz low-pass filter. The window of analysis was from 100 ms before to 500 ms after the stimulus onset, and the prestimulus period was used as the DC baseline. Four frontal electrodes (Fp1, Fp2, F9 and F10) were used for the rejection of trials containing artifacts due to blinks.
There were three different interstimulus interval (ISI) conditions for each modality, 0.5–0.7 s, 1.8–2.2 s and 9–11 s (2 Hz, 0.5 Hz and 0.1 Hz conditions). Therefore, there were 12 conditions (4 modalities × 3 ISIs). In each condition, 56 stimuli were presented in four separate blocks (10–17 stimuli in each block). Subjects were instructed to count the number of stimuli silently, and asked to report it in each block. If the answer was incorrect for one block out of four, the accuracy rate for the condition was 75%. In each block, the first stimulus was not included in the recording. The order of the 12 conditions was randomized among subjects.
Auditory-evoked potentials (AEPs) were elicited with a 1000 Hz tone (50 ms plateau, 5 ms rise/fall) that was presented binaurally through headphones at 60 dB SPL. For somatosensory-evoked potentials (SEPs), a square wave pulse 0.5 ms in duration was delivered to the right index finger through ring electrodes with the anode and cathode at the first and second phalangeal space, respectively. The stimulus intensity was two times the sensory threshold (1.0 ± 0.4 mA). For visual-evoked potentials (VEPs), a figure of a star (white on a black background, 5.3 × 5.3° visual angle) was presented for 48 ms at the center of a screen 1.5 m in front of the subject. Noxious stimuli-evoked potentials (pain-related SEPs, pSEPs) were elicited by intra-epidermal electrical stimulation  using a concentric bipolar needle electrode  that could stimulate cutaneous A-delta fibers selectively. The electrical stimulus was a square wave pulse of 1.0 ms applied to the dorsum of the right hand between the first and second metacarpal bones. The stimulus intensity was two times the pain threshold (0.2 ± 0.1 mA). The mean visual analogue scale score (0–100) for the painful sensation was 31 ± 20.
Conventional averaged waveforms of the 12 conditions were obtained for each subject. Then, two difference waveforms were obtained in each modality. That is, we obtained the difference waveform between the 2 Hz and 0.5 Hz conditions by subtraction of the waveform of the 2 Hz condition from the 0.5 Hz-waveform (Sub1), and the difference waveform between the 0.5 Hz and 0.1 Hz conditions by subtraction of the 0.5 Hz-waveform from the 0.1 Hz-waveform (Sub2). Therefore, the Sub1 waveform indicated the activity that was increased in the 0.5 Hz condition as compared with the 2 Hz condition, and the Sub2 waveform indicated the activity that was increased in the 0.1 Hz condition as compared with the 0.5 Hz condition. The grand-averaged waveforms of the three conditions (2 Hz, 0.5 Hz, 0.1 Hz), Sub1 and Sub2 across all subjects were obtained and used for the analysis of topography. The averaged waveforms for the 2 Hz condition, Sub1 and Sub2 were used for source modeling.
In each waveform, the root mean square (RMS) across the 27 channels was calculated, and the field distribution was examined at several RMS peaks. The similarity of the field distribution at a certain latency point between different conditions or between different modalities was examined by determining the correlation coefficient, r . In addition to the grand-averaged waveform, the similarity of the topography among modalities was also assessed using data of the 0.1 Hz condition of individual subjects.
A multi-dipole analysis was performed to separate temporally overlapping multiple sources by using the brain electric source analysis (BESA) software package (NeuroScan, Mclean, VA). Model adequacy was assessed by examining: 1) percent variance , 2) F-ratio (ratio of reduced chi-square values before and after adding a new source)  and 3) residual waveforms (that is, the difference between the recorded data and the model) as described previously . Since 27 EEG channels were used in the present study, only four dipoles were allowed to be included in a model to calculate the F-ratio (the degree of freedom of a chi-square of a model is 27-6N, N = number of dipoles). This is the main reason why we used the difference waveform (Sub1 and Sub2) in this study. Our preliminary analysis showed that more than six dipoles were necessary to explain the 0.1 Hz waveform. The subtraction procedure could reduce the number of dipoles because of the presence of common source activities with similar source strength among conditions. Therefore, source modeling was applied to the 2 Hz, Sub1 and Sub2 waveforms though the goal of the analysis was to clarify the temporal sequence of each source activity in the 0.1 Hz condition, where both early and late activities were expected to be present. It is well known that the middle-long latency components of evoked potentials are sensitive to ISI (e.g. [33, 34]). Only when the fourth and fifth dipoles contributed almost equally to explain the recorded data (for example, two sources in the bilateral fusiform gyrus in the 2 Hz-VEPs), was a fifth dipole included in the model. BESA uses a spherical four-shell model (the brain, cerebrospinal fluid, bone and skin). The location of each cortical source was expressed in Talairach coordinates. To confirm the reliability of results of BESA using grand-averaged waveforms, data of each subject obtained in the 0.1 Hz condition were also subjected to the source modeling. The method was identical to that for the averaged waveform. It was sometimes difficult to analyze individual data using a multi-dipole method because of the low S/N ratio. This was the main reason that we used grand-averaged data in this study. However, the evoked responses in the 0.1 Hz condition were usually enough large for this analysis at least for detecting the late activities that the present study targeted. Therefore, data of individual subjects for the 0.1 Hz condition were used for the BESA and topographical analyses, and the results were compared to those for the grand-averaged data.
The counting task was very easy for the subjects. The mean correct answer rate was 98.2%. A two-way analysis of variance showed that the ISI (F = 3.42, P = 0.04) but not modality (F = 1.11, P = 0.35) was a significant factor modulating the correct answer rate. Bonferroni/Dunn's post hoc test indicated that the correct answer rate was significantly (P < 0.05) higher for the 0.1 Hz condition (99.4%) than the 2 Hz condition (96.4%).
Waveform and topography
In the waveforms of the 2 Hz and 0.5 Hz conditions, there were two RMS peaks at around 82 and 190 ms (Fig. 1). At both peaks, the field distribution was highly correlated between the two conditions (r = 0.98 and 0.99, respectively). However, the distribution was slightly different at the P7/8 and P9/10 electrodes. In the 2 Hz condition, the activity at the first RMS peak (82 ms) was negative at almost all the electrodes, but a positive activity was clearly detected at the P7/8 and P9/10 electrodes in the 0.5 Hz condition, indicating the presence of additional bilateral sources in the 0.5 Hz condition at this latency point. In confirmation of this, the Sub1 waveform showed a positivity at these electrodes at around 80 ms. In the 0.1 Hz condition, an additional negativity at around 115 ms and positivity at around 300 ms emerged. In the large positive deflection at 150–400 ms, there were two RMS peaks at 255 and 288 ms. At 255 ms, the positivity was maximal at Cz but the positive peak shifted slightly posteriorly (maximal at Pz followed by P3/4) at 288 ms, indicating that at least two distinct source activities were responsible for shaping the large positive deflection. The Sub2 waveform confirmed that there were additional large negative/positive sequential components in the 0.1 Hz condition as compared with the 0.5 Hz condition.
In the 2 Hz-SEPs, a weak positivity at around 225 ms and weak activities at earlier latencies were evoked (Fig. 1). The positivity with the maximal amplitude at Cz was enhanced in the 0.5 Hz condition. In the 0.5 Hz condition, an additional negativity (maximal at T7) and concomitant positivity (Fz and Pz) at around 86 ms appeared. In the 0.1 Hz condition, a large negativity (140 ms) and positivity (200–350 ms) appeared in addition. Like waveforms of the 0.1 Hz-AEPs, the large positivity in this condition had a maximal amplitude at Cz at around 268 ms but the location of the positive peak shifted more posteriorly with an increase in latency. The field distribution pattern at the first RMS peak of the large positivity (268 ms) was correlated with that in the 0.1 Hz-AEPs (255 ms, r = 0.97 for the grand-averaged waveform and 0.9 ± 0.09 for the individual data), and that of the second RMS peak of the 0.1 Hz-SEPs (299 ms) was also highly correlated with that in the 0.1 Hz-AEPs (288 ms, r = 0.99 and 0.91 ± 0.07). Likewise, both the negativity (142 ms) and positivity (295 ms) of the Sub2-SEP waveform were significantly correlated with those of the Sub2-AEP waveform (125 and 267 ms)(r = 0.91 and 0.97, respectively), suggesting that similar cortical activities contributed to shape the waveform of the 0.1 Hz condition for AEPs and SEPs.
Visual stimuli at 2 and 0.5 Hz evoked similar positive/negative/positive sequential components (Fig. 2). Two additional positive components peaking at 274 and 355 ms appeared in the 0.1 Hz condition. The field distribution pattern was very similar among the three ISI conditions for the first positivity peaking at around 90 ms (r = 0.92–0.98), the negativity peaking at around 140 ms (0.98–0.99) and the second positivity peaking at around 200 ms (0.97–0.98). The Sub1 waveform showed a component peaking at 128 ms, with the maximal negativity at O1 and O2 and a positivity at Cz. Similar to AEPs and SEPs, an additional negativity (168 ms) and two sequential positive components (304 and 360 ms) appeared in the 0.1 Hz condition as compared with the 0.5 Hz condition.
In the 2 Hz-pSEPs, no clear component was evoked. In the 0.5 Hz condition, a positivity peaking at 298 ms with a maximal amplitude at Cz was evoked (Fig. 2). In the 0.1 Hz condition, a large positivity peaking at Cz additionally appeared as in the other modalities. The field distribution pattern at the peak RMS of the positivity (354 ms) was similar to those of the 0.1 Hz-AEPs (288 ms, r = 0.96 for the grand-averaged waveform and 0.92 ± 0.05 for individual waveforms), 0.1 Hz-SEPs (299 ms, r = 0.98 and 0.93 ± 0.03) and 0.1 Hz-VEPs (355 ms, r = 0.91 and 0.81 ± 0.13). There was also an additional component at around 150 ms in the 0.1 Hz-pSEP as compared with the 0.5 Hz condition. At the peak RMS (146 ms), the negativity was maximal at T7 (and T8 in the ipsilateral hemisphere) and the positivity was maximal at the midline electrodes (Fz and Pz). The field distribution pattern at this latency was very similar to that at 86 ms of the 0.1 Hz-SEPs (r = 0.97 and 0.88 ± 0.07). Such a high correlation of the complicated distribution pattern between different modalities was noteworthy. The Sub2 waveform consisted of a large negativity and positivity with similar field distribution patterns in other sensory modalities (r = 0.61–0.91 for the negativity, r = 0.93–0.97 for the positivity). The relatively low correlation coefficient for the negativity was due to the concomitant existence of modality-specific activity at the latency of the negativity that was enhanced in the 0.1 Hz condition, especially in VEPs.
Procedures of source modeling
We tried to seek the source solution responsible for the prominent component of the potentials whose topography showed high correlation among different conditions or different modalities. This suggests that there are similar generators in different conditions or different modalities. We repeated source estimation at around the peak latency of the potential component to select a robust solution having the highest GOF (or highest improvement of GOF). Once the best first source was determined, we tried to find the second source at around the peak latency of the potentials that were remained to be explained by the first source. Usually, the peak of the residual waveform was similar to that for the original waveform where the topography showed high correlation among conditions or modalities.
For the waveform of the 2 Hz-AEPs (Fig. 3), we started the analysis at the first peak of RMS (82 ms). The best single source was estimated to be located in the left supratemporal plane (-55, -29, 12 in Talairach coordinates) probably corresponding to the planum temporale (PT) according to previous studies (for review, see . Since the first source left substantial activity unexplained at this latency point (residual variance = 25%), we tried to find a second source at this latency. The best second source was estimated to be in the right PT (41, -27, 15), and the GOF was increased from 75 to 98% (F = 5.9, P < 0.0002) by the addition of the second source. This two-source model successfully explained the data around 82 ms, but left some activities at around 200 ms unexplained. The best source to explain the residual activity was estimated to be in the middle part of the cingulate gyrus (1, -23, 44). By adding this source, the GOF was increased from 44 to 98% (p < 0.0001). This three-dipole model provided a mean GOF value of 87% (0–500 ms), and no additional dipoles significantly improved the fit. By using similar procedures, four sources in the bilateral PT and bilateral superior temporal gyrus (STG) were estimated for the Sub1 waveform, indicating that these four source activities were stronger in the 0.5 Hz condition than the 2 Hz condition or additionally appeared in the 0.5 Hz condition. In the Sub2 waveform, bilateral STG sources were responsible for the early activity at around 90 ms. After fitting these two sources, however, large parts of the main negative/positive components were left unexplained. To explain the residual activity, the best source was estimated to be in the posterior part of the anterior cingulate cortex (ACC). This source markedly improved the fit (e.g., the GOF at 125 ms increased from 21 to 93%). However, residual activity was still clear at around 300 ms. To explain the residual activity, we had to add two more sources to the model since no single source significantly improved the fit, but residual activity was evident. The best sources were estimated to be located in the medial part of the temporal lobe in the parahippocampal gyrus of both hemispheres. After the addition of these two sources, the GOF at 300 ms increased from 64 to 98% (P < 0.02).
Multimodal activations and modality-specific activations
Similar procedures were applied to SEPs, VEPs and pSEPs. Figures 3 and 4 show the results of source modeling. The Talairach coordinates of each cortical source are shown in Table 1. In SEPs, VEPs and pSEPs, sources in the ACC and Hip contributed to the Sub2 waveform, like in the AEPs. That is, the ACC was the main source of activity responsible for the large negative/positive vertex potentials and the bilateral sources in Hip were also responsible for the later part of the positivity, which was consistent with similar scalp topographies of evoked potentials among modalities. The locations of the ACC and Hip sources were similar among sensory modalities (Fig. 5B).
Several modality-specific activations were estimated to occur in each sensory modality. Among them, sources in the STG (AEPs), left opercular region (OP, SEPs), bilateral middle occipital gyrus (MOG, VEPs) and left OP (pSEPs) appeared to be sensitive to the ISI, that is, they were more strongly activated in longer ISI conditions.
Time course of each source activity in the 0.1 Hz condition
Since all sources detected in the analysis described above were considered to be active in the 0.1 Hz condition, they were applied to the waveform of the 0.1 Hz condition to determine the actual time course of each cortical source activity. Figure 5A shows the source strength as a function of time of the main modality-specific source activity and multimodal activity for the waveform of the 0.1 Hz condition. The peak latency of each activity is shown in Table 1. The ACC activity was biphasic and the difference in latency between the first and second peaks was similar among modalities (117–145 ms). The activity in Hip always peaked later than the second peak of the ACC activity, and the delay (48–77 ms) was similar among modalities. In addition, the temporal sequence between the main modality-specific activation and ACC activation was similar among modalities, that is, the first component of ACC activity peaked later than the modality-specific activation by 30–56 ms. These results suggested that there were similar time courses of the multimodal activations in the ACC and Hip as well as a similar timing of the flow from the modality-specific area to the multimodal circuit among all the sensory modalities.
Results of source modeling for the 0.1 Hz waveform of individual subjects are shown in Fig. 5D. The time course of each cortical activity and the sequential pattern of activation through the sensory-specific areas, ACC and Hip were very similar to those for the grand-averaged data (Table 1). In Fig. 5D, only the main modality-specific activity and multimodal activity are shown. In addition to these sources, a significant dipole was estimated to be located in right OP (n = 5) and left S1 (n = 2) for SEP, right OP (n = 6) for pSEP, and V1 (n = 5) and the fusiform gyrus (n = 3) for VEP.
In our previous studies using MEG to monitor auditory , tactile , pain [5, 6] and visual [3, 7] systems, we showed that there were similar sequential activations through 'early' and 'late' sensory cortical areas among these sensory modalities. The results of the present study demonstrated similar time courses of activation through modality-specific areas and multimodal areas. Since the main modality-specific activations in the present study correspond to the 'late' activity in nature, these findings suggest a common temporal sequence of activations: modality-specific 'early' activity to modality-specific 'late' activity-ACC-Hip (Fig. 5C).
In the present study, we used a three-step multiple source analysis to find cortical sources responsible for evoked potentials in the 0.1 Hz condition: 2 Hz, Sub1 (0.5 Hz-2 Hz) and Sub2 (0.1 Hz-0.5 Hz). To study the timing of sequential activations among several cortical areas, it is apparent that results would be more convincing if the subtraction procedures were not necessary. However, the present results showed that this method made it easy to find several major sources of activities responsible for evoked potentials of each ISI condition. For example, the very similar Sub2 waveforms and their topographies of all modalities clearly showed the usefulness of this method. On the other hand, however, there is a possibility that we missed minor contributors during our three-step analysis, especially weak activities that contributed to all three ISI conditions equally.
Multimodal activation in the ACC and hippocampal region
The present results demonstrated that the main common components of the evoked potentials were the negative-positive vertex potentials that arise mainly from the ACC. It is well known that noxious stimuli evoke negative/positive vertex potentials with a scalp distribution similar to those in the present study. Source modeling of scalp potentials evoked by laser stimuli [21–23] showed that the ACC is the main generator of the vertex potentials, which was confirmed later in many studies, including intracranial recordings by Lenz et al. (1998) . Several studies have demonstrated that at least some of the vertex potentials reflect sensory non-specific events [37–39]. Very similar biphasic potentials were also evoked in the ACC in response to auditory and visual stimuli in an intracranial recording study . As for the negative/positive vertex potentials following tactile stimulation, the field distribution of the negative/positive potentials of the 0.1 Hz-SEPs was highly correlated with those in other modalities (e.g. r = 0.96 between SEP-N140 and AEP-N115; r = 0.99 between SEP-P299 and AEP-P288) indicating that the main generator of these potentials is not in the modality-specific area. An intracranial recording study by Allison et al. (1992)  supported this view by showing that the scalp negative/positive vertex potentials are not generated in the sensorimotor cortex. There have been a few studies whose results were consistent with the present findings that the main generator of the negative/positive vertex potentials in response to tactile stimuli is the ACC. An SEP study by Waberski et al. (2002)  reported the possible contribution of the ACC to N140, and an MEG study by Inui et al. (2003a)  reported the ACC activity following tactile stimulation with a peak at 128–150 ms that was coexistent with the peak (134 ms on average) of the simultaneously recorded scalp (Cz) negativity.
The cingulate cortex is an anatomically and functionally heterogeneous area, and is considered to serve cognitive, emotional, motor, nociceptive and visuospatial functions [41–44]. According to the traditional dichotomy of the cingulate cortex (anterior and posterior), the activation in the present study (BA 24/32) is located in the posterior part of the ACC. Functionally, this area in the cingulate cortex is coexistent with the cognitive subdivision of the ACC, which is activated by numerous cognitive/attentional tasks (for review, see Bush et al. 2000 ). The posterior part of the ACC is also a part of diffuse cortical networks sensitive to stimulus salience , stimulus changes  and oddball paradigms [20, 45–47], and is thought to be the major structure in the anterior attentional system proposed by Posner and Peterson (1990)  and a key site in Mesulam's (1990)  interconnected network for directed attention. Therefore, the similar location and time course of the ACC activity among all sensory modalities in this study suggests that the ACC activity is related to modality non-specific functions. Since only a very simple counting task was used in the present study and the ACC activation was robustly obtained in the long ISI condition, the ACC activation may be related to involuntary shifts of attention to the stimulus presented against the 'silent' background [50–54]. In a classical review analyzing the component structure of auditory N1, Näätänen and Picton (1987)  described that the N1 consists of the modality-specific and non-specific components. One of the non-specific N1 components was predicted to be generated in the frontal lobe or deeper structure and to be associated with attentional triggering mechanisms.
Activation in the Hip has usually been studied under oddball paradigms as will be described below, and reports describing Hip activity in response to simple sensory stimuli are rare. There are several EEG and MEG studies showing Hip activity following noxious stimulation [5, 23, 55, 56] and following tactile stimulation . In these studies, the Hip activity appeared later than the modality-specific and ACC activities, which is in agreement with the present results. The important roles of the hippocampus are regarded as the memory function like the memory storage, retrieval and consolidation [57–59] or attention . As described below, the Hip activity seems to be one of the major sources generating the scalp-recorded P300 (P3b), both of which we measured in this study. A series of classical psychophysical studies by Donchin and colleagues have suggested that the P3b reflects the update of context in the working memory store accompanied by the allocation of attentional resources . These imply that the Hip activity we observed is associated with the memory or attentional function. It could be at least related to the more voluntary process at the later stage than the modality non-specific process that would be reflected by the ACC activity.
Possible involvement of the ACC and Hip activities in generating the P300
In ERPs recorded under oddball paradigms, in which an infrequent target stimulus and a frequent non-target stimulus are presented in a random order, a large positive component peaking at 300 ms or more after the stimulus (P300) is elicited in response to the target stimulus . The P300 component is considered to reflect fundamental cognitive processes (for reviews, see [61, 63–65]). While task-relevant deviant stimuli elicit the parietocentral P300 or P3b, task-irrelevant salient stimuli inserted in the repeated target and non-target stimuli under three-stimulus paradigms  elicit an earlier positive deflection, the frontocentral P3a or novelty P300 [50, 51, 67]. The P3a has been interpreted as a neural correlate of the orienting response .
Both the temporal sequence and the topography of the two distinct positive components in the present study therefore resemble those of the P3a and P3b. Although the present study did not employ discrimination tasks such as those in the oddball paradigm, subjects had to count the stimuli presented against a silent background at a random ISI, and thus the 'odd' or 'infrequent' aspect of the stimulus discrimination was maintained to a certain degree and the P300 component might have been elicited. Supporting this notion, Polich et al. (1994)  compared the P300 elicited with auditory stimuli using a typical oddball paradigm with that elicited from a single stimulus procedure and concluded that the single-stimulus task produces the P300 in the same fashion as those elicited with the oddball paradigm. A source modeling study using ERPs by Tarkka and Stokic (1998)  produced similar findings.
The notion that the biphasic ACC activity in the present study might have contributed to both the negativity (fronto-central part of processing negativity or non-specific N1 component ; ) and the subsequent P3a in oddball paradigms is consistent with a recent source modeling study of scalp potentials , in which the main contributor to the N2/P3a components was the ACC. A source modeling study by Dien et al. (2003)  also found that the P3a had a source in the ACC. The hippocampal region being one of the neural origins of the P300 is consistent with previous studies using scalp potentials [69, 71, 72], MEG  and intracranial recordings [26–29]. The finding in the intracranial recording studies that the peak of the focal activity recorded from the medial temporal lobe occurred 35–100 ms later than the positive peak recorded from the scalp [28, 29, 74] is noteworthy, indicating that the activity in this area is mainly responsible for the late part of the P300.
Temporal sequence of modality-specific and multimodal activations
Several modality-specific source activities were identified in each sensory modality. In general, these activities were less sensitive to the ISI than those from the ACC and Hip. However, some of these sources were activated more strongly in longer ISI conditions. These sources included STG (auditory), OP (tactile and pain) and MOG (visual). Although the significance of the sensitivity of these sources to the ISI was not clear in this study, they appear to be involved in attentional/cognitive aspects of sensory processing to a certain degree rather than the projecting function. These activities show several common features: 1) they peaked 30–56 ms earlier than the ACC source activity; 2) they lasted about 100 ms; and 3) their response latency was too long for the early (primary) activity. According to our previous findings in MEG studies, these activities correspond to the 'late' activity that follows several 'early' activities with a characteristic triphasic time course. Therefore, the present results together with our previous findings suggest a common temporal sequence of activation across all the sensory modalities: 1) the 'early' sensory cortex, 2) the 'late' sensory cortex, 3) the ACC, and then 4) the hippocampal region, each step of which roughly corresponds to 1) the quick projection of information to the next, 2) receiving, perception and integration of sensory information that has been refined at earlier stages, 3) involuntary shift of attention to perceived stimuli, and 4) voluntary aspect of cognition, memory and execution.
Comparison with other studies on multimodal interaction
There is growing evidence for multimodal activation and interaction in humans (see review, [15, 75]. Multimodal convergence has been reported in animal studies in the posterior parietal cortex, temporo-parietal junction and cingulate cortex [76–80]. It has been also reported that the cortical areas which have been considered to be exclusively modality-specific also respond to stimuli from different sensory modalities [80–84]. Most interestingly, neuronal activity can be modulated by non-auditory influences even at the primary cortical level in the primary auditory cortex [85–88]. This multimodal interaction was expected to be neural correlates of multisensory behavioral interactions observed in humans. In behavioral studies, an accelerated reaction time and an illusion of perception have been observed as a result of multisensory interaction, such as the McGurk illusion [89–91], the hearing hand illusion [92, 93], and the parchment-skin illusion . MEG and EEG studies have successfully demonstrated neural correlates of such multisensory interactions in humans [8–12, 95–103].
The present study was planned to reveal the basic time course of unimodal and multimodal activations in a different framework from these previous studies. First of all, we attempted to know whether a similar cortical area is activated in response to sensory inputs coming individually from different modalities. Therefore, stimuli were presented individually from each modality, not the simultaneous presentation from different modalities often used in studies of multisensory interaction. In this regard, Downar et al. (2000) reported similar findings to the present study using fMRI [16, 17], though the technique used and stimulus environment were different from the present ones. Then, we analyzed the difference in timing between unimodal and multimodal activations, and found a similar time course from unimodal to multimodal activations across the different modalities including vision, audition, touch and pain. This unimodal-multimodal transition expressed as a similar time course is associated with functions produced by manipulation of the ISI. Considering that the activities tested contribute to N1 and P3a/b, the unimodal-multimodal transition implies that the orienting attention and the context update are represented by electrophysiological correlates with a similar delay among those modalities tested.
The present study revealed the temporal sequence of activations through modality-specific and multimodal pathways among all sensory modalities including vision, audition, touch and pain. The timing of the transition from unimodal to multimodal activations was similar among all the modalities. Take together with our previous studies investigating early cortical activities, there is thus the similar temporal sequence of activation among all sensory modalities.
anterior cingulate cortex
auditory evoked potential
brain electric source analysis
middle occipital gyrus
root mean square
somatosensory evoked potentials
visual evoked potential.
Inui K, Wang X, Tamura Y, Kaneoke Y, Kakigi R: Serial processing in the human somatosensory system. Cereb Cortex. 2004, 14 (8): 851-857. 10.1093/cercor/bhh043.
Inui K, Okamoto H, Miki K, Gunji A, Kakigi R: Serial and parallel processing in the human auditory cortex: a magnetoencephalographic study. Cereb Cortex. 2006, 16 (1): 18-30. 10.1093/cercor/bhi080.
Inui K, Sannan H, Miki K, Kaneoke Y, Kakigi R: Timing of early activity in the visual cortex as revealed by simultaneous MEG and ERG recordings. NeuroImage. 2006, 30 (1): 239-244. 10.1016/j.neuroimage.2005.09.003.
Inui K, Tsuji T, Kakigi R: Temporal analysis of cortical mechanisms for pain relief by tactile stimuli in humans. Cereb Cortex. 2006, 16 (3): 355-365. 10.1093/cercor/bhi114.
Inui K, Tran TD, Qiu Y, Wang X, Hoshiyama M, Kakigi R: A comparative magnetoencephalographic study of cortical activations evoked by noxious and innocuous somatosensory stimulations. Neuroscience. 2003, 120 (1): 235-248. 10.1016/S0306-4522(03)00261-6.
Inui K, Wang X, Qiu Y, Nguyen BT, Ojima S, Tamura Y, Nakata H, Wasaka T, Tran TD, Kakigi R: Pain processing within the primary somatosensory cortex in humans. Eur J Neurosci. 2003, 18 (10): 2859-2866. 10.1111/j.1460-9568.2003.02995.x.
Inui K, Kakigi R: Temporal analysis of the flow from V1 to the extrastriate cortex in humans. Journal of neurophysiology. 2006, 96 (2): 775-784.
Molholm S, Ritter W, Javitt DC, Foxe JJ: Multisensory visual-auditory object recognition in humans: a high-density electrical mapping study. Cereb Cortex. 2004, 14 (4): 452-465. 10.1093/cercor/bhh007.
Molholm S, Ritter W, Murray MM, Javitt DC, Schroeder CE, Foxe JJ: Multisensory auditory-visual interactions during early sensory processing in humans: a high-density electrical mapping study. Brain Res Cogn Brain Res. 2002, 14 (1): 115-128. 10.1016/S0926-6410(02)00066-6.
Lutkenhoner B, Lammertmann C, Simoes C, Hari R: Magnetoencephalographic correlates of audiotactile interaction. NeuroImage. 2002, 15 (3): 509-522. 10.1006/nimg.2001.0991.
Giard MH, Peronnet F: Auditory-visual integration during multimodal object recognition in humans: a behavioral and electrophysiological study. J Cogn Neurosci. 1999, 11 (5): 473-490. 10.1162/089892999563544.
Teder-Salejarvi WA, Di Russo F, McDonald JJ, Hillyard SA: Effects of spatial congruity on audio-visual multimodal integration. J Cogn Neurosci. 2005, 17 (9): 1396-1409. 10.1162/0898929054985383.
Teder-Salejarvi WA, McDonald JJ, Di Russo F, Hillyard SA: An analysis of audio-visual crossmodal integration by means of event-related potential (ERP) recordings. Brain Res Cogn Brain Res. 2002, 14 (1): 106-114. 10.1016/S0926-6410(02)00065-4.
Beauchamp MS, Argall BD, Bodurka J, Duyn JH, Martin A: Unraveling multisensory integration: patchy organization within human STS multisensory cortex. Nature neuroscience. 2004, 7 (11): 1190-1192. 10.1038/nn1333.
Calvert GA: Crossmodal processing in the human brain: insights from functional neuroimaging studies. Cereb Cortex. 2001, 11 (12): 1110-1123. 10.1093/cercor/11.12.1110.
Downar J, Crawley AP, Mikulis DJ, Davis KD: A multimodal cortical network for the detection of changes in the sensory environment. Nature neuroscience. 2000, 3 (3): 277-283. 10.1038/72991.
Downar J, Crawley AP, Mikulis DJ, Davis KD: A cortical network sensitive to stimulus salience in a neutral behavioral context across multiple sensory modalities. Journal of neurophysiology. 2002, 87 (1): 615-620.
Naatanen R, Picton T: The N1 wave of the human electric and magnetic response to sound: a review and an analysis of the component structure. Psychophysiology. 1987, 24 (4): 375-425. 10.1111/j.1469-8986.1987.tb00311.x.
Dien J, Spencer KM, Donchin E: Localization of the event-related potential novelty response as defined by principal components analysis. Brain Res Cogn Brain Res. 2003, 17 (3): 637-650. 10.1016/S0926-6410(03)00188-5.
Crottaz-Herbette S, Menon V: Where and when the anterior cingulate cortex modulates attentional response: combined fMRI and ERP evidence. J Cogn Neurosci. 2006, 18 (5): 766-780. 10.1162/jocn.2006.18.5.766.
Tarkka IM, Treede RD: Equivalent electrical source analysis of pain-related somatosensory evoked potentials elicited by a CO2 laser. J Clin Neurophysiol. 1993, 10 (4): 513-519. 10.1097/00004691-199310000-00009.
Bromm B, Chen AC: Brain electrical source analysis of laser evoked potentials in response to painful trigeminal nerve stimulation. Electroencephalography and clinical neurophysiology. 1995, 95 (1): 14-26. 10.1016/0013-4694(95)00032-T.
Valeriani M, Rambaud L, Mauguiere F: Scalp topography and dipolar source modelling of potentials evoked by CO2 laser stimulation of the hand. Electroencephalography and clinical neurophysiology. 1996, 100 (4): 343-353. 10.1016/0168-5597(96)95625-7.
Turak B, Louvel J, Buser P, Lamarche M: Event-related potentials recorded from the cingulate gyrus during attentional tasks: a study in patients with implanted electrodes. Neuropsychologia. 2002, 40 (1): 99-107. 10.1016/S0028-3932(01)00071-9.
Waberski TD, Gobbele R, Darvas F, Schmitz S, Buchner H: Spatiotemporal imaging of electrical activity related to attention to somatosensory stimulation. NeuroImage. 2002, 17 (3): 1347-1357. 10.1006/nimg.2002.1222.
Halgren E, Squires NK, Wilson CL, Rohrbaugh JW, Babb TL, Crandall PH: Endogenous potentials generated in the human hippocampal formation and amygdala by infrequent events. Science. 1980, 210 (4471): 803-805. 10.1126/science.7434000.
Halgren E, Baudena P, Clarke JM, Heit G, Liegeois C, Chauvel P, Musolino A: Intracerebral potentials to rare target and distractor auditory and visual stimuli. I. Superior temporal plane and parietal lobe. Electroencephalography and clinical neurophysiology. 1995, 94 (3): 191-220. 10.1016/0013-4694(94)00259-N.
McCarthy G, Wood CC, Williamson PD, Spencer DD: Task-dependent field potentials in human hippocampal formation. J Neurosci. 1989, 9 (12): 4253-4268.
Paller KA, McCarthy G, Roessler E, Allison T, Wood CC: Potentials evoked in human and monkey medial temporal lobe during auditory and visual oddball paradigms. Electroencephalography and clinical neurophysiology. 1992, 84 (3): 269-279. 10.1016/0168-5597(92)90008-Y.
Inui K, Tran TD, Hoshiyama M, Kakigi R: Preferential stimulation of Adelta fibers by intra-epidermal needle electrode in humans. Pain. 2002, 96 (3): 247-252. 10.1016/S0304-3959(01)00453-5.
Hari R, Joutsiniemi SL, Sarvas J: Spatial resolution of neuromagnetic records: theoretical calculations in a spherical model. Electroencephalography and clinical neurophysiology. 1988, 71 (1): 64-72. 10.1016/0168-5597(88)90020-2.
Supek S, Aine CJ: Simulation studies of multiple dipole neuromagnetic source localization: model order and limits of source resolution. IEEE Trans Biomed Eng. 1993, 40 (6): 529-540. 10.1109/10.237672.
Tomberg C, Desmedt JE, Ozaki I, Nguyen TH, Chalklin V: Mapping somatosensory evoked potentials to finger stimulation at intervals of 450 to 4000 msec and the issue of habituation when assessing early cognitive components. Electroencephalography and clinical neurophysiology. 1989, 74 (5): 347-358. 10.1016/0168-5597(89)90002-6.
Kida T, Nishihira Y, Wasaka T, Nakata H, Sakamoto M: Differential modulation of temporal and frontal components of the somatosensory N140 and the effect of interstimulus interval in a selective attention task. Brain Res Cogn Brain Res. 2004, 19 (1): 33-39. 10.1016/j.cogbrainres.2003.10.016.
Griffiths TD, Warren JD: The planum temporale as a computational hub. Trends in neurosciences. 2002, 25 (7): 348-353. 10.1016/S0166-2236(02)02191-4.
Lenz FA, Rios M, Zirh A, Chau D, Krauss G, Lesser RP: Painful stimuli evoke potentials recorded over the human anterior cingulate gyrus. Journal of neurophysiology. 1998, 79 (4): 2231-2234.
Zaslansky R, Sprecher E, Katz Y, Rozenberg B, Hemli JA, Yarnitsky D: Pain-evoked potentials: what do they really measure?. Electroencephalogr Clin Neurophysiol. 1996, 100 (5): 384-391.
Legrain V, Bruyer R, Guerit JM, Plaghki L: Nociceptive processing in the human brain of infrequent task-relevant and task-irrelevant noxious stimuli. A study with event-related potentials evoked by CO2 laser radiant heat stimuli. Pain. 2003, 103 (3): 237-248. 10.1016/S0304-3959(02)00451-7.
Lorenz J, Garcia-Larrea L: Contribution of attentional and cognitive factors to laser evoked brain potentials. Neurophysiologie clinique = Clinical neurophysiology. 2003, 33 (6): 293-301. 10.1016/j.neucli.2003.10.004.
Allison T, McCarthy G, Wood CC: The relationship between human long-latency somatosensory evoked potentials recorded from the cortical surface and from the scalp. Electroencephalography and clinical neurophysiology. 1992, 84 (4): 301-314. 10.1016/0168-5597(92)90082-M.
Vogt BA, Finch DM, Olson CR: Functional heterogeneity in cingulate cortex: the anterior executive and posterior evaluative regions. Cereb Cortex. 1992, 2 (6): 435-443.
Picard N, Strick PL: Motor areas of the medial wall: a review of their location and functional activation. Cereb Cortex. 1996, 6 (3): 342-353. 10.1093/cercor/6.3.342.
Bush G, Luu P, Posner MI: Cognitive and emotional influences in anterior cingulate cortex. Trends in cognitive sciences. 2000, 4 (6): 215-222. 10.1016/S1364-6613(00)01483-2.
Vogt BA: Pain and emotion interactions in subregions of the cingulate gyrus. Nature reviews. 2005, 6 (7): 533-544. 10.1038/nrn1704.
Baudena P, Halgren E, Heit G, Clarke JM: Intracerebral potentials to rare target and distractor auditory and visual stimuli. III. Frontal cortex. Electroencephalography and clinical neurophysiology. 1995, 94 (4): 251-264. 10.1016/0013-4694(95)98476-O.
Smith ME, Halgren E, Sokolik M, Baudena P, Musolino A, Liegeois-Chauvel C, Chauvel P: The intracranial topography of the P3 event-related potential elicited during auditory oddball. Electroencephalography and clinical neurophysiology. 1990, 76 (3): 235-248. 10.1016/0013-4694(90)90018-F.
Wang C, Ulbert I, Schomer DL, Marinkovic K, Halgren E: Responses of human anterior cingulate cortex microdomains to error detection, conflict monitoring, stimulus-response mapping, familiarity, and orienting. J Neurosci. 2005, 25 (3): 604-613. 10.1523/JNEUROSCI.4151-04.2005.
Posner MI, Petersen SE: The attention system of the human brain. Annual review of neuroscience. 1990, 13: 25-42. 10.1146/annurev.ne.13.030190.000325.
Mesulam MM: Large-scale neurocognitive networks and distributed processing for attention, language, and memory. Annals of neurology. 1990, 28 (5): 597-613. 10.1002/ana.410280502.
Knight RT: Decreased response to novel stimuli after prefrontal lesions in man. Electroencephalography and clinical neurophysiology. 1984, 59 (1): 9-20. 10.1016/0168-5597(84)90016-9.
Yamaguchi S, Knight RT: P300 generation by novel somatosensory stimuli. Electroencephalography and clinical neurophysiology. 1991, 78 (1): 50-55. 10.1016/0013-4694(91)90018-Y.
Kekoni J, Hamalainen H, McCloud V, Reinikainen K, Naatanen R: Is the somatosensory N250 related to deviance discrimination or conscious target detection?. Electroencephalography and clinical neurophysiology. 1996, 100 (2): 115-125. 10.1016/0013-4694(95)00231-6.
Kida T, Nishihira Y, Wasaka T, Nakata H, Sakamoto M: Passive enhancement of the somatosensory P100 and N140 in an active attention task using deviant alone condition. Clin Neurophysiol. 2004, 115 (4): 871-879. 10.1016/j.clinph.2003.11.037.
Kida T, Wasaka T, Nakata H, Akatsuka K, Kakigi R: Active attention modulates passive attention-related neural responses to sudden somatosensory input against a silent background. Experimental brain research Experimentelle Hirnforschung. 2006, 175 (4): 609-617. 10.1007/s00221-006-0578-4.
Watanabe S, Kakigi R, Koyama S, Hoshiyama M, Kaneoke Y: Pain processing traced by magnetoencephalography in the human brain. Brain topography. 1998, 10 (4): 255-264. 10.1023/A:1022218906322.
Qiu Y, Inui K, Wang X, Nguyen BT, Tran TD, Kakigi R: Effects of distraction on magnetoencephalographic responses ascending through C-fibers in humans. Clin Neurophysiol. 2004, 115 (3): 636-646. 10.1016/j.clinph.2003.10.017.
Thompson RF: In search of memory traces. Annual review of psychology. 2005, 56: 1-23. 10.1146/annurev.psych.56.091103.070239.
Neves G, Cooke SF, Bliss TV: Synaptic plasticity, memory and the hippocampus: a neural network approach to causality. Nature reviews. 2008, 9 (1): 65-75. 10.1038/nrn2303.
Squire LR, Wixted JT, Clark RE: Recognition memory and the medial temporal lobe: a new perspective. Nature reviews. 2007, 8 (11): 872-883. 10.1038/nrn2154.
Rowland DC, Kentros CG: Potential anatomical basis for attentional modulation of hippocampal neurons. Annals of the New York Academy of Sciences. 2008, 1129: 213-224. 10.1196/annals.1417.014.
Donchin E, Coles MGH: Is the P300 component a manifestation of context updating?. Brain Behav Science. 1988, 100: 357-374.
Sutton S, Braren M, Zubin J, John ER: Evoked-potential correlates of stimulus uncertainty. Science. 1965, 150 (700): 1187-1188. 10.1126/science.150.3700.1187.
Picton TW, Hillyard SA: Endogeneous event-related potentials. 1988, New York: Oxford
Soltani M, Knight RT: Neural origins of the P300. Critical reviews in neurobiology. 2000, 14 (3–4): 199-224.
Linden DE: The p300: where in the brain is it produced and what does it tell us?. Neuroscientist. 2005, 11 (6): 563-576. 10.1177/1073858405280524.
Courchesne E, Hillyard SA, Galambos R: Stimulus novelty, task relevance and the visual evoked potential in man. Electroencephalography and clinical neurophysiology. 1975, 39 (2): 131-143. 10.1016/0013-4694(75)90003-6.
Squires NK, Squires KC, Hillyard SA: Two varieties of long-latency positive waves evoked by unpredictable auditory stimuli in man. Electroencephalography and clinical neurophysiology. 1975, 38 (4): 387-401. 10.1016/0013-4694(75)90263-1.
Polich J, Eischen SE, Collins GE: P300 from a single auditory stimulus. Electroencephalography and clinical neurophysiology. 1994, 92 (3): 253-261. 10.1016/0168-5597(94)90068-X.
Tarkka IM, Stokic DS: Source localization of P300 from oddball, single stimulus, and omitted-stimulus paradigms. Brain topography. 1998, 11 (2): 141-151. 10.1023/A:1022258606418.
Naatanen R, Gaillard AW, Mantysalo S: Early selective-attention effect on evoked potential reinterpreted. Acta psychologica. 1978, 42 (4): 313-329. 10.1016/0001-6918(78)90006-9.
Yamazaki T, Kamijo K, Kenmochi A, Fukuzumi S, Kiyuna T, Takaki Y, Kuroiwa Y: Multiple equivalent current dipole source localization of visual event-related potentials during oddball paradigm with motor response. Brain topography. 2000, 12 (3): 159-175. 10.1023/A:1023467806268.
Valeriani M, Fraioli L, Ranghi F, Giaquinto S: Dipolar source modeling of the P300 event-related potential after somatosensory stimulation. Muscle & nerve. 2001, 24 (12): 1677-1686. 10.1002/mus.1203.
Okada YC, Kaufman L, Williamson SJ: The hippocampal formation as a source of the slow endogenous potentials. Electroencephalography and clinical neurophysiology. 1983, 55 (4): 417-426. 10.1016/0013-4694(83)90130-X.
Halgren E, Baudena P, Clarke JM, Heit G, Marinkovic K, Devaux B, Vignal JP, Biraben A: Intracerebral potentials to rare target and distractor auditory and visual stimuli. II. Medial, lateral and posterior temporal lobe. Electroencephalography and clinical neurophysiology. 1995, 94 (4): 229-250. 10.1016/0013-4694(95)98475-N.
Schroeder CE, Foxe J: Multisensory contributions to low-level, 'unisensory' processing. Current opinion in neurobiology. 2005, 15 (4): 454-458. 10.1016/j.conb.2005.06.008.
Barnes CL, Pandya DN: Efferent cortical connections of multimodal cortex of the superior temporal sulcus in the rhesus monkey. The Journal of comparative neurology. 1992, 318 (2): 222-244. 10.1002/cne.903180207.
Pandya DN: Anatomy of the auditory cortex. Revue neurologique. 1995, 151 (8–9): 486-494.
Baylis GC, Rolls ET, Leonard CM: Functional subdivisions of the temporal lobe neocortex. J Neurosci. 1987, 7 (2): 330-342.
Hikosaka K, Iwai E, Saito H, Tanaka K: Polysensory properties of neurons in the anterior bank of the caudal superior temporal sulcus of the macaque monkey. Journal of neurophysiology. 1988, 60 (5): 1615-1637.
Schroeder CE, Foxe JJ: The timing and laminar profile of converging inputs to multisensory areas of the macaque neocortex. Brain Res Cogn Brain Res. 2002, 14 (1): 187-198. 10.1016/S0926-6410(02)00073-3.
Schroeder CE, Lindsley RW, Specht C, Marcovici A, Smiley JF, Javitt DC: Somatosensory input to auditory association cortex in the macaque monkey. Journal of neurophysiology. 2001, 85 (3): 1322-1327.
Fu KM, Johnston TA, Shah AS, Arnold L, Smiley J, Hackett TA, Garraghty PE, Schroeder CE: Auditory cortical neurons respond to somatosensory stimulation. J Neurosci. 2003, 23 (20): 7510-7515.
Brosch M, Selezneva E, Scheich H: Nonauditory events of a behavioral procedure activate auditory cortex of highly trained monkeys. J Neurosci. 2005, 25 (29): 6797-6806. 10.1523/JNEUROSCI.1571-05.2005.
Kayser C, Petkov CI, Augath M, Logothetis NK: Integration of touch and sound in auditory cortex. Neuron. 2005, 48 (2): 373-384. 10.1016/j.neuron.2005.09.018.
Werner-Reiss U, Kelly KA, Trause AS, Underhill AM, Groh JM: Eye position affects activity in primary auditory cortex of primates. Curr Biol. 2003, 13 (7): 554-562. 10.1016/S0960-9822(03)00168-4.
Fu KM, Shah AS, O'Connell MN, McGinnis T, Eckholdt H, Lakatos P, Smiley J, Schroeder CE: Timing and laminar profile of eye-position effects on auditory responses in primate auditory cortex. Journal of neurophysiology. 2004, 92 (6): 3522-3531. 10.1152/jn.01228.2003.
Ghazanfar AA, Maier JX, Hoffman KL, Logothetis NK: Multisensory integration of dynamic faces and voices in rhesus monkey auditory cortex. J Neurosci. 2005, 25 (20): 5004-5012. 10.1523/JNEUROSCI.0799-05.2005.
Lakatos P, Chen CM, O'Connell MN, Mills A, Schroeder CE: Neuronal oscillations and multisensory interaction in primary auditory cortex. Neuron. 2007, 53 (2): 279-292. 10.1016/j.neuron.2006.12.011.
McGurk H, MacDonald J: Hearing lips and seeing voices. Nature. 1976, 264 (5588): 746-748. 10.1038/264746a0.
Sams M, Mottonen R, Sihvonen T: Seeing and hearing others and oneself talk. Brain Res Cogn Brain Res. 2005, 23 (2–3): 429-435. 10.1016/j.cogbrainres.2004.11.006.
Colin C, Radeau M, Soquet A, Demolin D, Colin F, Deltenre P: Mismatch negativity evoked by the McGurk-MacDonald effect: a phonetic representation within short-term memory. Clin Neurophysiol. 2002, 113 (4): 495-506. 10.1016/S1388-2457(02)00024-X.
Schurmann M, Caetano G, Hlushchuk Y, Jousmaki V, Hari R: Touch activates human auditory cortex. NeuroImage. 2006, 30 (4): 1325-1331. 10.1016/j.neuroimage.2005.11.020.
Schurmann M, Caetano G, Jousmaki V, Hari R: Hands help hearing: facilitatory audiotactile interaction at low sound-intensity levels. The Journal of the Acoustical Society of America. 2004, 115 (2): 830-832. 10.1121/1.1639909.
Jousmaki V, Hari R: Parchment-skin illusion: sound-biased touch. Curr Biol. 1998, 8 (6): R190-10.1016/S0960-9822(98)70120-4.
Calvert GA, Bullmore ET, Brammer MJ, Campbell R, Williams SC, McGuire PK, Woodruff PW, Iversen SD, David AS: Activation of auditory cortex during silent lipreading. Science. 1997, 276 (5312): 593-596. 10.1126/science.276.5312.593.
Schroger E, Widmann A: Speeded responses to audiovisual signal changes result from bimodal integration. Psychophysiology. 1998, 35 (6): 755-759. 10.1017/S0048577298980714.
Molholm S, Sehatpour P, Mehta AD, Shpaner M, Gomez-Ramirez M, Ortigue S, Dyke JP, Schwartz TH, Foxe JJ: Audio-visual multisensory integration in superior parietal lobule revealed by human intracranial recordings. Journal of neurophysiology. 2006, 96 (2): 721-729. 10.1152/jn.00285.2006.
Fort A, Delpuech C, Pernier J, Giard MH: Dynamics of cortico-subcortical cross-modal operations involved in audio-visual object detection in humans. Cereb Cortex. 2002, 12 (10): 1031-1039. 10.1093/cercor/12.10.1031.
Murray MM, Molholm S, Michel CM, Heslenfeld DJ, Ritter W, Javitt DC, Schroeder CE, Foxe JJ: Grabbing your ear: rapid auditory-somatosensory multisensory interactions in low-level sensory cortices are not constrained by stimulus alignment. Cereb Cortex. 2005, 15 (7): 963-974. 10.1093/cercor/bhh197.
Senkowski D, Molholm S, Gomez-Ramirez M, Foxe JJ: Oscillatory beta activity predicts response speed during a multisensory audiovisual reaction time task: a high-density electrical mapping study. Cereb Cortex. 2006, 16 (11): 1556-1565. 10.1093/cercor/bhj091.
Talsma D, Woldorff MG: Selective attention and multisensory integration: multiple phases of effects on the evoked brain activity. J Cogn Neurosci. 2005, 17 (7): 1098-1114. 10.1162/0898929054475172.
Saint-Amour D, De Sanctis P, Molholm S, Ritter W, Foxe JJ: Seeing voices: High-density electrical mapping and source-analysis of the multisensory mismatch negativity evoked during the McGurk illusion. Neuropsychologia. 2007, 45 (3): 587-597. 10.1016/j.neuropsychologia.2006.03.036.
Talsma D, Doty TJ, Woldorff MG: Selective attention and audiovisual integration: is attending to both modalities a prerequisite for early integration?. Cereb Cortex. 2007, 17 (3): 679-690. 10.1093/cercor/bhk016.
This study was carried out as a part of the "Ground-based Research Announcement for Space Utilization" program by the Japan Space Forum. This study was supported by the Magnetic Health Science Foundation.
ET contributed to data collection and analysis, and drafting and revising the paper. KI contributed to planning the study, data collection and analysis, and drafting the paper. TK and TM contributed to revising the paper. YT contributed to data collection and constructing devices. RK contributed to revising the paper.