A transition from unimodal to multimodal activations in four sensory modalities in humans: an electrophysiological study
© Tanaka et al; licensee BioMed Central Ltd. 2008
Received: 05 June 2008
Accepted: 08 December 2008
Published: 08 December 2008
To investigate the long-latency activities common to all sensory modalities, electroencephalographic responses to auditory (1000 Hz pure tone), tactile (electrical stimulation to the index finger), visual (simple figure of a star), and noxious (intra-epidermal electrical stimulation to the dorsum of the hand) stimuli were recorded from 27 scalp electrodes in 14 healthy volunteers.
Results of source modeling showed multimodal activations in the anterior part of the cingulate cortex (ACC) and hippocampal region (Hip). The activity in the ACC was biphasic. In all sensory modalities, the first component of ACC activity peaked 30–56 ms later than the peak of the major modality-specific activity, the second component of ACC activity peaked 117–145 ms later than the peak of the first component, and the activity in Hip peaked 43–77 ms later than the second component of ACC activity.
The temporal sequence of activations through modality-specific and multimodal pathways was similar among all sensory modalities.
In previous studies using magnetoencephalograms (MEGs) to monitor tactile , auditory , visual [3, 4] and pain [5, 6] systems, we found very similar mechanisms of sensory processing among these sensory modalities. In brief, several 'early' activities appear serially with a time delay of about 4 ms at each step followed by one or two 'late' activities. In general, the 'early' activity reverses polarity twice with an interval of 10 ms, which results in a characteristic triphasic waveform, while the 'late' activity is long-lasting without a polarity reversal at such a short interval . For example, following tactile stimulation, 'early' activations are elicited in area 3b, area 1 and the posterior parietal cortex in this order with a delay of 3–4 ms between each step, and then a long-lasting 'late' activity is evoked in the secondary somatosensory area. We postulate that a basic role of the 'early' activity is to receive inputs from the thalamus or convergent inputs from the thalamus and/or adjacent cortical areas and to send this information to the next point quickly, while the long-lasting 'late' activity is involved in recognition of the stimuli .
In the present study, we sought to compare mechanisms of sensory processing at latencies later than the 'late' activity among these sensory modalities (vision, audition, touch and pain). At first, we expected there to be unimodal and multimodal activities. Although there is a large and growing number of studies on multimodal interaction using electroencephalography (EEG) and MEG [8–13] as well as multimodal activation and interaction using functional magnetic resonance imaging (fMRI) [14–17], it is unclear whether or not the timing of the transition from unimodal to multimodal cortical activations is different among modalities. The present study manipulated the interstimulus interval (ISI) at three levels to examine the transition from unimodal and multimodal activations. In general, evoked potentials around and after 100 ms, like N1 and P3a/P3b, are more sensitively increased by increasing the ISI than earlier responses. If the difference in response amplitude between different ISIs shows the same scalp distribution among modalities, the difference in amplitude should originate from the same generator. In addition, a source analysis is helpful to estimate the location of the generator. In general, the manipulation of the ISI more strongly affects P3a and P3b, and the non-specific activities of N1, which are considered indices of orienting attention , compared with activities ("late" activity) within 100 ms. Because the original N1 response in any modality largely includes modality-specific activities showing a different scalp distribution depending on the modality, a comparison between the scalp distributions of the original N1 might not provide a clear-cut result. Therefore, we chose a simple manipulation of the ISI to extract more effectively the non-specific activities, to obtain a clearer result and to enable us to estimate more reliably and simply the location of the activity. Another reason for this choice is that the activities obtained by manipulating the ISI may be associated with orienting attention and later processes reflected by the non-specific N1 and P3a/P3b.
We expected the non-specific, possibly multimodal, activities obtained by manipulating the ISI are clearly found at later than 100 ms and have the same scalp distribution among modalities, but the "late" activities within 100 ms to have a different scalp distribution among modalities. The multimodal activations were expected to be in the anterior cingulate gyrus or hippocampus on the basis of a large number of previous studies performing source analyses [5, 19–25] and intracranial recordings [26–29], whereas unimodal activities ("late" activities) are estimated to be in areas specific to each modality. Of final and special interest was whether or not the timing of the transition from unimodal to multimodal activations is the same among modalities.
The experiment was performed on 14 (four females and ten males) healthy right-handed volunteers, aged 23–52 years (mean, 32 ± 8). The study was approved in advance by the Ethics Committee of the National Institute for Physiological Sciences, Okazaki, Japan, and written consent was obtained from all the subjects.
The EEG activity was recorded using 27 scalp electrodes placed on Fp1/Fp2, F3/F4, F7/F8, F9/F10, C3/C4, T7/T8 (T3/T4), T9/T10, P3/P4, P7/P8 (T5/T6), P9/P10, O1/O2, Fpz, Fz, Cz, Pz and Oz according to the 10–10 system. The reference electrode was placed on the nose. The impedance of the electrode was kept below 5 kΩ. The EEG signals were recorded with a bandpass filter of 0.1–100 Hz at a sampling rate of 1000 Hz, and then digitally filtered with a 70-Hz low-pass filter. The window of analysis was from 100 ms before to 500 ms after the stimulus onset, and the prestimulus period was used as the DC baseline. Four frontal electrodes (Fp1, Fp2, F9 and F10) were used for the rejection of trials containing artifacts due to blinks.
There were three different interstimulus interval (ISI) conditions for each modality, 0.5–0.7 s, 1.8–2.2 s and 9–11 s (2 Hz, 0.5 Hz and 0.1 Hz conditions). Therefore, there were 12 conditions (4 modalities × 3 ISIs). In each condition, 56 stimuli were presented in four separate blocks (10–17 stimuli in each block). Subjects were instructed to count the number of stimuli silently, and asked to report it in each block. If the answer was incorrect for one block out of four, the accuracy rate for the condition was 75%. In each block, the first stimulus was not included in the recording. The order of the 12 conditions was randomized among subjects.
Auditory-evoked potentials (AEPs) were elicited with a 1000 Hz tone (50 ms plateau, 5 ms rise/fall) that was presented binaurally through headphones at 60 dB SPL. For somatosensory-evoked potentials (SEPs), a square wave pulse 0.5 ms in duration was delivered to the right index finger through ring electrodes with the anode and cathode at the first and second phalangeal space, respectively. The stimulus intensity was two times the sensory threshold (1.0 ± 0.4 mA). For visual-evoked potentials (VEPs), a figure of a star (white on a black background, 5.3 × 5.3° visual angle) was presented for 48 ms at the center of a screen 1.5 m in front of the subject. Noxious stimuli-evoked potentials (pain-related SEPs, pSEPs) were elicited by intra-epidermal electrical stimulation  using a concentric bipolar needle electrode  that could stimulate cutaneous A-delta fibers selectively. The electrical stimulus was a square wave pulse of 1.0 ms applied to the dorsum of the right hand between the first and second metacarpal bones. The stimulus intensity was two times the pain threshold (0.2 ± 0.1 mA). The mean visual analogue scale score (0–100) for the painful sensation was 31 ± 20.
Conventional averaged waveforms of the 12 conditions were obtained for each subject. Then, two difference waveforms were obtained in each modality. That is, we obtained the difference waveform between the 2 Hz and 0.5 Hz conditions by subtraction of the waveform of the 2 Hz condition from the 0.5 Hz-waveform (Sub1), and the difference waveform between the 0.5 Hz and 0.1 Hz conditions by subtraction of the 0.5 Hz-waveform from the 0.1 Hz-waveform (Sub2). Therefore, the Sub1 waveform indicated the activity that was increased in the 0.5 Hz condition as compared with the 2 Hz condition, and the Sub2 waveform indicated the activity that was increased in the 0.1 Hz condition as compared with the 0.5 Hz condition. The grand-averaged waveforms of the three conditions (2 Hz, 0.5 Hz, 0.1 Hz), Sub1 and Sub2 across all subjects were obtained and used for the analysis of topography. The averaged waveforms for the 2 Hz condition, Sub1 and Sub2 were used for source modeling.
In each waveform, the root mean square (RMS) across the 27 channels was calculated, and the field distribution was examined at several RMS peaks. The similarity of the field distribution at a certain latency point between different conditions or between different modalities was examined by determining the correlation coefficient, r . In addition to the grand-averaged waveform, the similarity of the topography among modalities was also assessed using data of the 0.1 Hz condition of individual subjects.
A multi-dipole analysis was performed to separate temporally overlapping multiple sources by using the brain electric source analysis (BESA) software package (NeuroScan, Mclean, VA). Model adequacy was assessed by examining: 1) percent variance , 2) F-ratio (ratio of reduced chi-square values before and after adding a new source)  and 3) residual waveforms (that is, the difference between the recorded data and the model) as described previously . Since 27 EEG channels were used in the present study, only four dipoles were allowed to be included in a model to calculate the F-ratio (the degree of freedom of a chi-square of a model is 27-6N, N = number of dipoles). This is the main reason why we used the difference waveform (Sub1 and Sub2) in this study. Our preliminary analysis showed that more than six dipoles were necessary to explain the 0.1 Hz waveform. The subtraction procedure could reduce the number of dipoles because of the presence of common source activities with similar source strength among conditions. Therefore, source modeling was applied to the 2 Hz, Sub1 and Sub2 waveforms though the goal of the analysis was to clarify the temporal sequence of each source activity in the 0.1 Hz condition, where both early and late activities were expected to be present. It is well known that the middle-long latency components of evoked potentials are sensitive to ISI (e.g. [33, 34]). Only when the fourth and fifth dipoles contributed almost equally to explain the recorded data (for example, two sources in the bilateral fusiform gyrus in the 2 Hz-VEPs), was a fifth dipole included in the model. BESA uses a spherical four-shell model (the brain, cerebrospinal fluid, bone and skin). The location of each cortical source was expressed in Talairach coordinates. To confirm the reliability of results of BESA using grand-averaged waveforms, data of each subject obtained in the 0.1 Hz condition were also subjected to the source modeling. The method was identical to that for the averaged waveform. It was sometimes difficult to analyze individual data using a multi-dipole method because of the low S/N ratio. This was the main reason that we used grand-averaged data in this study. However, the evoked responses in the 0.1 Hz condition were usually enough large for this analysis at least for detecting the late activities that the present study targeted. Therefore, data of individual subjects for the 0.1 Hz condition were used for the BESA and topographical analyses, and the results were compared to those for the grand-averaged data.
The counting task was very easy for the subjects. The mean correct answer rate was 98.2%. A two-way analysis of variance showed that the ISI (F = 3.42, P = 0.04) but not modality (F = 1.11, P = 0.35) was a significant factor modulating the correct answer rate. Bonferroni/Dunn's post hoc test indicated that the correct answer rate was significantly (P < 0.05) higher for the 0.1 Hz condition (99.4%) than the 2 Hz condition (96.4%).
Waveform and topography
In the 2 Hz-SEPs, a weak positivity at around 225 ms and weak activities at earlier latencies were evoked (Fig. 1). The positivity with the maximal amplitude at Cz was enhanced in the 0.5 Hz condition. In the 0.5 Hz condition, an additional negativity (maximal at T7) and concomitant positivity (Fz and Pz) at around 86 ms appeared. In the 0.1 Hz condition, a large negativity (140 ms) and positivity (200–350 ms) appeared in addition. Like waveforms of the 0.1 Hz-AEPs, the large positivity in this condition had a maximal amplitude at Cz at around 268 ms but the location of the positive peak shifted more posteriorly with an increase in latency. The field distribution pattern at the first RMS peak of the large positivity (268 ms) was correlated with that in the 0.1 Hz-AEPs (255 ms, r = 0.97 for the grand-averaged waveform and 0.9 ± 0.09 for the individual data), and that of the second RMS peak of the 0.1 Hz-SEPs (299 ms) was also highly correlated with that in the 0.1 Hz-AEPs (288 ms, r = 0.99 and 0.91 ± 0.07). Likewise, both the negativity (142 ms) and positivity (295 ms) of the Sub2-SEP waveform were significantly correlated with those of the Sub2-AEP waveform (125 and 267 ms)(r = 0.91 and 0.97, respectively), suggesting that similar cortical activities contributed to shape the waveform of the 0.1 Hz condition for AEPs and SEPs.
In the 2 Hz-pSEPs, no clear component was evoked. In the 0.5 Hz condition, a positivity peaking at 298 ms with a maximal amplitude at Cz was evoked (Fig. 2). In the 0.1 Hz condition, a large positivity peaking at Cz additionally appeared as in the other modalities. The field distribution pattern at the peak RMS of the positivity (354 ms) was similar to those of the 0.1 Hz-AEPs (288 ms, r = 0.96 for the grand-averaged waveform and 0.92 ± 0.05 for individual waveforms), 0.1 Hz-SEPs (299 ms, r = 0.98 and 0.93 ± 0.03) and 0.1 Hz-VEPs (355 ms, r = 0.91 and 0.81 ± 0.13). There was also an additional component at around 150 ms in the 0.1 Hz-pSEP as compared with the 0.5 Hz condition. At the peak RMS (146 ms), the negativity was maximal at T7 (and T8 in the ipsilateral hemisphere) and the positivity was maximal at the midline electrodes (Fz and Pz). The field distribution pattern at this latency was very similar to that at 86 ms of the 0.1 Hz-SEPs (r = 0.97 and 0.88 ± 0.07). Such a high correlation of the complicated distribution pattern between different modalities was noteworthy. The Sub2 waveform consisted of a large negativity and positivity with similar field distribution patterns in other sensory modalities (r = 0.61–0.91 for the negativity, r = 0.93–0.97 for the positivity). The relatively low correlation coefficient for the negativity was due to the concomitant existence of modality-specific activity at the latency of the negativity that was enhanced in the 0.1 Hz condition, especially in VEPs.
Procedures of source modeling
We tried to seek the source solution responsible for the prominent component of the potentials whose topography showed high correlation among different conditions or different modalities. This suggests that there are similar generators in different conditions or different modalities. We repeated source estimation at around the peak latency of the potential component to select a robust solution having the highest GOF (or highest improvement of GOF). Once the best first source was determined, we tried to find the second source at around the peak latency of the potentials that were remained to be explained by the first source. Usually, the peak of the residual waveform was similar to that for the original waveform where the topography showed high correlation among conditions or modalities.
Multimodal activations and modality-specific activations
Cortical activities in response to auditory, tactile, visual and noxious stimuli.
Peak latency (ms)
R superior temporal gyrus
83 ± 8
L superior temporal gyrus
87 ± 8
117 ± 13/239 ± 42
R parahippocampal gyrus
311 ± 25
L parahippocampal gyrus
323 ± 14
L opercular area
98 ± 12
135 ± 7/255 ± 30
R parahippocampal gyrus
351 ± 31
L parahippocampal gyrus
315 ± 31
R middle occipital gyrus
144 ± 10
L middle occipital gyrus
146 ± 10
189 ± 23/308 ± 43
L parahippocampal gyrus
367 ± 10
L opercular area
154 ± 21
210 ± 22/337 ± 25
R parahippocampal gyrus
386 ± 50
L parahippocampal gyrus
386 ± 28
Several modality-specific activations were estimated to occur in each sensory modality. Among them, sources in the STG (AEPs), left opercular region (OP, SEPs), bilateral middle occipital gyrus (MOG, VEPs) and left OP (pSEPs) appeared to be sensitive to the ISI, that is, they were more strongly activated in longer ISI conditions.
Time course of each source activity in the 0.1 Hz condition
Since all sources detected in the analysis described above were considered to be active in the 0.1 Hz condition, they were applied to the waveform of the 0.1 Hz condition to determine the actual time course of each cortical source activity. Figure 5A shows the source strength as a function of time of the main modality-specific source activity and multimodal activity for the waveform of the 0.1 Hz condition. The peak latency of each activity is shown in Table 1. The ACC activity was biphasic and the difference in latency between the first and second peaks was similar among modalities (117–145 ms). The activity in Hip always peaked later than the second peak of the ACC activity, and the delay (48–77 ms) was similar among modalities. In addition, the temporal sequence between the main modality-specific activation and ACC activation was similar among modalities, that is, the first component of ACC activity peaked later than the modality-specific activation by 30–56 ms. These results suggested that there were similar time courses of the multimodal activations in the ACC and Hip as well as a similar timing of the flow from the modality-specific area to the multimodal circuit among all the sensory modalities.
Results of source modeling for the 0.1 Hz waveform of individual subjects are shown in Fig. 5D. The time course of each cortical activity and the sequential pattern of activation through the sensory-specific areas, ACC and Hip were very similar to those for the grand-averaged data (Table 1). In Fig. 5D, only the main modality-specific activity and multimodal activity are shown. In addition to these sources, a significant dipole was estimated to be located in right OP (n = 5) and left S1 (n = 2) for SEP, right OP (n = 6) for pSEP, and V1 (n = 5) and the fusiform gyrus (n = 3) for VEP.
In our previous studies using MEG to monitor auditory , tactile , pain [5, 6] and visual [3, 7] systems, we showed that there were similar sequential activations through 'early' and 'late' sensory cortical areas among these sensory modalities. The results of the present study demonstrated similar time courses of activation through modality-specific areas and multimodal areas. Since the main modality-specific activations in the present study correspond to the 'late' activity in nature, these findings suggest a common temporal sequence of activations: modality-specific 'early' activity to modality-specific 'late' activity-ACC-Hip (Fig. 5C).
In the present study, we used a three-step multiple source analysis to find cortical sources responsible for evoked potentials in the 0.1 Hz condition: 2 Hz, Sub1 (0.5 Hz-2 Hz) and Sub2 (0.1 Hz-0.5 Hz). To study the timing of sequential activations among several cortical areas, it is apparent that results would be more convincing if the subtraction procedures were not necessary. However, the present results showed that this method made it easy to find several major sources of activities responsible for evoked potentials of each ISI condition. For example, the very similar Sub2 waveforms and their topographies of all modalities clearly showed the usefulness of this method. On the other hand, however, there is a possibility that we missed minor contributors during our three-step analysis, especially weak activities that contributed to all three ISI conditions equally.
Multimodal activation in the ACC and hippocampal region
The present results demonstrated that the main common components of the evoked potentials were the negative-positive vertex potentials that arise mainly from the ACC. It is well known that noxious stimuli evoke negative/positive vertex potentials with a scalp distribution similar to those in the present study. Source modeling of scalp potentials evoked by laser stimuli [21–23] showed that the ACC is the main generator of the vertex potentials, which was confirmed later in many studies, including intracranial recordings by Lenz et al. (1998) . Several studies have demonstrated that at least some of the vertex potentials reflect sensory non-specific events [37–39]. Very similar biphasic potentials were also evoked in the ACC in response to auditory and visual stimuli in an intracranial recording study . As for the negative/positive vertex potentials following tactile stimulation, the field distribution of the negative/positive potentials of the 0.1 Hz-SEPs was highly correlated with those in other modalities (e.g. r = 0.96 between SEP-N140 and AEP-N115; r = 0.99 between SEP-P299 and AEP-P288) indicating that the main generator of these potentials is not in the modality-specific area. An intracranial recording study by Allison et al. (1992)  supported this view by showing that the scalp negative/positive vertex potentials are not generated in the sensorimotor cortex. There have been a few studies whose results were consistent with the present findings that the main generator of the negative/positive vertex potentials in response to tactile stimuli is the ACC. An SEP study by Waberski et al. (2002)  reported the possible contribution of the ACC to N140, and an MEG study by Inui et al. (2003a)  reported the ACC activity following tactile stimulation with a peak at 128–150 ms that was coexistent with the peak (134 ms on average) of the simultaneously recorded scalp (Cz) negativity.
The cingulate cortex is an anatomically and functionally heterogeneous area, and is considered to serve cognitive, emotional, motor, nociceptive and visuospatial functions [41–44]. According to the traditional dichotomy of the cingulate cortex (anterior and posterior), the activation in the present study (BA 24/32) is located in the posterior part of the ACC. Functionally, this area in the cingulate cortex is coexistent with the cognitive subdivision of the ACC, which is activated by numerous cognitive/attentional tasks (for review, see Bush et al. 2000 ). The posterior part of the ACC is also a part of diffuse cortical networks sensitive to stimulus salience , stimulus changes  and oddball paradigms [20, 45–47], and is thought to be the major structure in the anterior attentional system proposed by Posner and Peterson (1990)  and a key site in Mesulam's (1990)  interconnected network for directed attention. Therefore, the similar location and time course of the ACC activity among all sensory modalities in this study suggests that the ACC activity is related to modality non-specific functions. Since only a very simple counting task was used in the present study and the ACC activation was robustly obtained in the long ISI condition, the ACC activation may be related to involuntary shifts of attention to the stimulus presented against the 'silent' background [50–54]. In a classical review analyzing the component structure of auditory N1, Näätänen and Picton (1987)  described that the N1 consists of the modality-specific and non-specific components. One of the non-specific N1 components was predicted to be generated in the frontal lobe or deeper structure and to be associated with attentional triggering mechanisms.
Activation in the Hip has usually been studied under oddball paradigms as will be described below, and reports describing Hip activity in response to simple sensory stimuli are rare. There are several EEG and MEG studies showing Hip activity following noxious stimulation [5, 23, 55, 56] and following tactile stimulation . In these studies, the Hip activity appeared later than the modality-specific and ACC activities, which is in agreement with the present results. The important roles of the hippocampus are regarded as the memory function like the memory storage, retrieval and consolidation [57–59] or attention . As described below, the Hip activity seems to be one of the major sources generating the scalp-recorded P300 (P3b), both of which we measured in this study. A series of classical psychophysical studies by Donchin and colleagues have suggested that the P3b reflects the update of context in the working memory store accompanied by the allocation of attentional resources . These imply that the Hip activity we observed is associated with the memory or attentional function. It could be at least related to the more voluntary process at the later stage than the modality non-specific process that would be reflected by the ACC activity.
Possible involvement of the ACC and Hip activities in generating the P300
In ERPs recorded under oddball paradigms, in which an infrequent target stimulus and a frequent non-target stimulus are presented in a random order, a large positive component peaking at 300 ms or more after the stimulus (P300) is elicited in response to the target stimulus . The P300 component is considered to reflect fundamental cognitive processes (for reviews, see [61, 63–65]). While task-relevant deviant stimuli elicit the parietocentral P300 or P3b, task-irrelevant salient stimuli inserted in the repeated target and non-target stimuli under three-stimulus paradigms  elicit an earlier positive deflection, the frontocentral P3a or novelty P300 [50, 51, 67]. The P3a has been interpreted as a neural correlate of the orienting response .
Both the temporal sequence and the topography of the two distinct positive components in the present study therefore resemble those of the P3a and P3b. Although the present study did not employ discrimination tasks such as those in the oddball paradigm, subjects had to count the stimuli presented against a silent background at a random ISI, and thus the 'odd' or 'infrequent' aspect of the stimulus discrimination was maintained to a certain degree and the P300 component might have been elicited. Supporting this notion, Polich et al. (1994)  compared the P300 elicited with auditory stimuli using a typical oddball paradigm with that elicited from a single stimulus procedure and concluded that the single-stimulus task produces the P300 in the same fashion as those elicited with the oddball paradigm. A source modeling study using ERPs by Tarkka and Stokic (1998)  produced similar findings.
The notion that the biphasic ACC activity in the present study might have contributed to both the negativity (fronto-central part of processing negativity or non-specific N1 component ; ) and the subsequent P3a in oddball paradigms is consistent with a recent source modeling study of scalp potentials , in which the main contributor to the N2/P3a components was the ACC. A source modeling study by Dien et al. (2003)  also found that the P3a had a source in the ACC. The hippocampal region being one of the neural origins of the P300 is consistent with previous studies using scalp potentials [69, 71, 72], MEG  and intracranial recordings [26–29]. The finding in the intracranial recording studies that the peak of the focal activity recorded from the medial temporal lobe occurred 35–100 ms later than the positive peak recorded from the scalp [28, 29, 74] is noteworthy, indicating that the activity in this area is mainly responsible for the late part of the P300.
Temporal sequence of modality-specific and multimodal activations
Several modality-specific source activities were identified in each sensory modality. In general, these activities were less sensitive to the ISI than those from the ACC and Hip. However, some of these sources were activated more strongly in longer ISI conditions. These sources included STG (auditory), OP (tactile and pain) and MOG (visual). Although the significance of the sensitivity of these sources to the ISI was not clear in this study, they appear to be involved in attentional/cognitive aspects of sensory processing to a certain degree rather than the projecting function. These activities show several common features: 1) they peaked 30–56 ms earlier than the ACC source activity; 2) they lasted about 100 ms; and 3) their response latency was too long for the early (primary) activity. According to our previous findings in MEG studies, these activities correspond to the 'late' activity that follows several 'early' activities with a characteristic triphasic time course. Therefore, the present results together with our previous findings suggest a common temporal sequence of activation across all the sensory modalities: 1) the 'early' sensory cortex, 2) the 'late' sensory cortex, 3) the ACC, and then 4) the hippocampal region, each step of which roughly corresponds to 1) the quick projection of information to the next, 2) receiving, perception and integration of sensory information that has been refined at earlier stages, 3) involuntary shift of attention to perceived stimuli, and 4) voluntary aspect of cognition, memory and execution.
Comparison with other studies on multimodal interaction
There is growing evidence for multimodal activation and interaction in humans (see review, [15, 75]. Multimodal convergence has been reported in animal studies in the posterior parietal cortex, temporo-parietal junction and cingulate cortex [76–80]. It has been also reported that the cortical areas which have been considered to be exclusively modality-specific also respond to stimuli from different sensory modalities [80–84]. Most interestingly, neuronal activity can be modulated by non-auditory influences even at the primary cortical level in the primary auditory cortex [85–88]. This multimodal interaction was expected to be neural correlates of multisensory behavioral interactions observed in humans. In behavioral studies, an accelerated reaction time and an illusion of perception have been observed as a result of multisensory interaction, such as the McGurk illusion [89–91], the hearing hand illusion [92, 93], and the parchment-skin illusion . MEG and EEG studies have successfully demonstrated neural correlates of such multisensory interactions in humans [8–12, 95–103].
The present study was planned to reveal the basic time course of unimodal and multimodal activations in a different framework from these previous studies. First of all, we attempted to know whether a similar cortical area is activated in response to sensory inputs coming individually from different modalities. Therefore, stimuli were presented individually from each modality, not the simultaneous presentation from different modalities often used in studies of multisensory interaction. In this regard, Downar et al. (2000) reported similar findings to the present study using fMRI [16, 17], though the technique used and stimulus environment were different from the present ones. Then, we analyzed the difference in timing between unimodal and multimodal activations, and found a similar time course from unimodal to multimodal activations across the different modalities including vision, audition, touch and pain. This unimodal-multimodal transition expressed as a similar time course is associated with functions produced by manipulation of the ISI. Considering that the activities tested contribute to N1 and P3a/b, the unimodal-multimodal transition implies that the orienting attention and the context update are represented by electrophysiological correlates with a similar delay among those modalities tested.
The present study revealed the temporal sequence of activations through modality-specific and multimodal pathways among all sensory modalities including vision, audition, touch and pain. The timing of the transition from unimodal to multimodal activations was similar among all the modalities. Take together with our previous studies investigating early cortical activities, there is thus the similar temporal sequence of activation among all sensory modalities.
anterior cingulate cortex
auditory evoked potential
brain electric source analysis
middle occipital gyrus
root mean square
somatosensory evoked potentials
visual evoked potential.
This study was carried out as a part of the "Ground-based Research Announcement for Space Utilization" program by the Japan Space Forum. This study was supported by the Magnetic Health Science Foundation.
- Inui K, Wang X, Tamura Y, Kaneoke Y, Kakigi R: Serial processing in the human somatosensory system. Cereb Cortex. 2004, 14 (8): 851-857. 10.1093/cercor/bhh043.PubMedGoogle Scholar
- Inui K, Okamoto H, Miki K, Gunji A, Kakigi R: Serial and parallel processing in the human auditory cortex: a magnetoencephalographic study. Cereb Cortex. 2006, 16 (1): 18-30. 10.1093/cercor/bhi080.PubMedGoogle Scholar
- Inui K, Sannan H, Miki K, Kaneoke Y, Kakigi R: Timing of early activity in the visual cortex as revealed by simultaneous MEG and ERG recordings. NeuroImage. 2006, 30 (1): 239-244. 10.1016/j.neuroimage.2005.09.003.PubMedGoogle Scholar
- Inui K, Tsuji T, Kakigi R: Temporal analysis of cortical mechanisms for pain relief by tactile stimuli in humans. Cereb Cortex. 2006, 16 (3): 355-365. 10.1093/cercor/bhi114.PubMedGoogle Scholar
- Inui K, Tran TD, Qiu Y, Wang X, Hoshiyama M, Kakigi R: A comparative magnetoencephalographic study of cortical activations evoked by noxious and innocuous somatosensory stimulations. Neuroscience. 2003, 120 (1): 235-248. 10.1016/S0306-4522(03)00261-6.PubMedGoogle Scholar
- Inui K, Wang X, Qiu Y, Nguyen BT, Ojima S, Tamura Y, Nakata H, Wasaka T, Tran TD, Kakigi R: Pain processing within the primary somatosensory cortex in humans. Eur J Neurosci. 2003, 18 (10): 2859-2866. 10.1111/j.1460-9568.2003.02995.x.PubMedGoogle Scholar
- Inui K, Kakigi R: Temporal analysis of the flow from V1 to the extrastriate cortex in humans. Journal of neurophysiology. 2006, 96 (2): 775-784.PubMedGoogle Scholar
- Molholm S, Ritter W, Javitt DC, Foxe JJ: Multisensory visual-auditory object recognition in humans: a high-density electrical mapping study. Cereb Cortex. 2004, 14 (4): 452-465. 10.1093/cercor/bhh007.PubMedGoogle Scholar
- Molholm S, Ritter W, Murray MM, Javitt DC, Schroeder CE, Foxe JJ: Multisensory auditory-visual interactions during early sensory processing in humans: a high-density electrical mapping study. Brain Res Cogn Brain Res. 2002, 14 (1): 115-128. 10.1016/S0926-6410(02)00066-6.PubMedGoogle Scholar
- Lutkenhoner B, Lammertmann C, Simoes C, Hari R: Magnetoencephalographic correlates of audiotactile interaction. NeuroImage. 2002, 15 (3): 509-522. 10.1006/nimg.2001.0991.PubMedGoogle Scholar
- Giard MH, Peronnet F: Auditory-visual integration during multimodal object recognition in humans: a behavioral and electrophysiological study. J Cogn Neurosci. 1999, 11 (5): 473-490. 10.1162/089892999563544.PubMedGoogle Scholar
- Teder-Salejarvi WA, Di Russo F, McDonald JJ, Hillyard SA: Effects of spatial congruity on audio-visual multimodal integration. J Cogn Neurosci. 2005, 17 (9): 1396-1409. 10.1162/0898929054985383.PubMedGoogle Scholar
- Teder-Salejarvi WA, McDonald JJ, Di Russo F, Hillyard SA: An analysis of audio-visual crossmodal integration by means of event-related potential (ERP) recordings. Brain Res Cogn Brain Res. 2002, 14 (1): 106-114. 10.1016/S0926-6410(02)00065-4.PubMedGoogle Scholar
- Beauchamp MS, Argall BD, Bodurka J, Duyn JH, Martin A: Unraveling multisensory integration: patchy organization within human STS multisensory cortex. Nature neuroscience. 2004, 7 (11): 1190-1192. 10.1038/nn1333.PubMedGoogle Scholar
- Calvert GA: Crossmodal processing in the human brain: insights from functional neuroimaging studies. Cereb Cortex. 2001, 11 (12): 1110-1123. 10.1093/cercor/11.12.1110.PubMedGoogle Scholar
- Downar J, Crawley AP, Mikulis DJ, Davis KD: A multimodal cortical network for the detection of changes in the sensory environment. Nature neuroscience. 2000, 3 (3): 277-283. 10.1038/72991.PubMedGoogle Scholar
- Downar J, Crawley AP, Mikulis DJ, Davis KD: A cortical network sensitive to stimulus salience in a neutral behavioral context across multiple sensory modalities. Journal of neurophysiology. 2002, 87 (1): 615-620.PubMedGoogle Scholar
- Naatanen R, Picton T: The N1 wave of the human electric and magnetic response to sound: a review and an analysis of the component structure. Psychophysiology. 1987, 24 (4): 375-425. 10.1111/j.1469-8986.1987.tb00311.x.PubMedGoogle Scholar
- Dien J, Spencer KM, Donchin E: Localization of the event-related potential novelty response as defined by principal components analysis. Brain Res Cogn Brain Res. 2003, 17 (3): 637-650. 10.1016/S0926-6410(03)00188-5.PubMedGoogle Scholar
- Crottaz-Herbette S, Menon V: Where and when the anterior cingulate cortex modulates attentional response: combined fMRI and ERP evidence. J Cogn Neurosci. 2006, 18 (5): 766-780. 10.1162/jocn.2006.18.5.766.PubMedGoogle Scholar
- Tarkka IM, Treede RD: Equivalent electrical source analysis of pain-related somatosensory evoked potentials elicited by a CO2 laser. J Clin Neurophysiol. 1993, 10 (4): 513-519. 10.1097/00004691-199310000-00009.PubMedGoogle Scholar
- Bromm B, Chen AC: Brain electrical source analysis of laser evoked potentials in response to painful trigeminal nerve stimulation. Electroencephalography and clinical neurophysiology. 1995, 95 (1): 14-26. 10.1016/0013-4694(95)00032-T.PubMedGoogle Scholar
- Valeriani M, Rambaud L, Mauguiere F: Scalp topography and dipolar source modelling of potentials evoked by CO2 laser stimulation of the hand. Electroencephalography and clinical neurophysiology. 1996, 100 (4): 343-353. 10.1016/0168-5597(96)95625-7.PubMedGoogle Scholar
- Turak B, Louvel J, Buser P, Lamarche M: Event-related potentials recorded from the cingulate gyrus during attentional tasks: a study in patients with implanted electrodes. Neuropsychologia. 2002, 40 (1): 99-107. 10.1016/S0028-3932(01)00071-9.PubMedGoogle Scholar
- Waberski TD, Gobbele R, Darvas F, Schmitz S, Buchner H: Spatiotemporal imaging of electrical activity related to attention to somatosensory stimulation. NeuroImage. 2002, 17 (3): 1347-1357. 10.1006/nimg.2002.1222.PubMedGoogle Scholar
- Halgren E, Squires NK, Wilson CL, Rohrbaugh JW, Babb TL, Crandall PH: Endogenous potentials generated in the human hippocampal formation and amygdala by infrequent events. Science. 1980, 210 (4471): 803-805. 10.1126/science.7434000.PubMedGoogle Scholar
- Halgren E, Baudena P, Clarke JM, Heit G, Liegeois C, Chauvel P, Musolino A: Intracerebral potentials to rare target and distractor auditory and visual stimuli. I. Superior temporal plane and parietal lobe. Electroencephalography and clinical neurophysiology. 1995, 94 (3): 191-220. 10.1016/0013-4694(94)00259-N.PubMedGoogle Scholar
- McCarthy G, Wood CC, Williamson PD, Spencer DD: Task-dependent field potentials in human hippocampal formation. J Neurosci. 1989, 9 (12): 4253-4268.PubMedGoogle Scholar
- Paller KA, McCarthy G, Roessler E, Allison T, Wood CC: Potentials evoked in human and monkey medial temporal lobe during auditory and visual oddball paradigms. Electroencephalography and clinical neurophysiology. 1992, 84 (3): 269-279. 10.1016/0168-5597(92)90008-Y.PubMedGoogle Scholar
- Inui K, Tran TD, Hoshiyama M, Kakigi R: Preferential stimulation of Adelta fibers by intra-epidermal needle electrode in humans. Pain. 2002, 96 (3): 247-252. 10.1016/S0304-3959(01)00453-5.PubMedGoogle Scholar
- Hari R, Joutsiniemi SL, Sarvas J: Spatial resolution of neuromagnetic records: theoretical calculations in a spherical model. Electroencephalography and clinical neurophysiology. 1988, 71 (1): 64-72. 10.1016/0168-5597(88)90020-2.PubMedGoogle Scholar
- Supek S, Aine CJ: Simulation studies of multiple dipole neuromagnetic source localization: model order and limits of source resolution. IEEE Trans Biomed Eng. 1993, 40 (6): 529-540. 10.1109/10.237672.PubMedGoogle Scholar
- Tomberg C, Desmedt JE, Ozaki I, Nguyen TH, Chalklin V: Mapping somatosensory evoked potentials to finger stimulation at intervals of 450 to 4000 msec and the issue of habituation when assessing early cognitive components. Electroencephalography and clinical neurophysiology. 1989, 74 (5): 347-358. 10.1016/0168-5597(89)90002-6.PubMedGoogle Scholar
- Kida T, Nishihira Y, Wasaka T, Nakata H, Sakamoto M: Differential modulation of temporal and frontal components of the somatosensory N140 and the effect of interstimulus interval in a selective attention task. Brain Res Cogn Brain Res. 2004, 19 (1): 33-39. 10.1016/j.cogbrainres.2003.10.016.PubMedGoogle Scholar
- Griffiths TD, Warren JD: The planum temporale as a computational hub. Trends in neurosciences. 2002, 25 (7): 348-353. 10.1016/S0166-2236(02)02191-4.PubMedGoogle Scholar
- Lenz FA, Rios M, Zirh A, Chau D, Krauss G, Lesser RP: Painful stimuli evoke potentials recorded over the human anterior cingulate gyrus. Journal of neurophysiology. 1998, 79 (4): 2231-2234.PubMedGoogle Scholar
- Zaslansky R, Sprecher E, Katz Y, Rozenberg B, Hemli JA, Yarnitsky D: Pain-evoked potentials: what do they really measure?. Electroencephalogr Clin Neurophysiol. 1996, 100 (5): 384-391.PubMedGoogle Scholar
- Legrain V, Bruyer R, Guerit JM, Plaghki L: Nociceptive processing in the human brain of infrequent task-relevant and task-irrelevant noxious stimuli. A study with event-related potentials evoked by CO2 laser radiant heat stimuli. Pain. 2003, 103 (3): 237-248. 10.1016/S0304-3959(02)00451-7.PubMedGoogle Scholar
- Lorenz J, Garcia-Larrea L: Contribution of attentional and cognitive factors to laser evoked brain potentials. Neurophysiologie clinique = Clinical neurophysiology. 2003, 33 (6): 293-301. 10.1016/j.neucli.2003.10.004.PubMedGoogle Scholar
- Allison T, McCarthy G, Wood CC: The relationship between human long-latency somatosensory evoked potentials recorded from the cortical surface and from the scalp. Electroencephalography and clinical neurophysiology. 1992, 84 (4): 301-314. 10.1016/0168-5597(92)90082-M.PubMedGoogle Scholar
- Vogt BA, Finch DM, Olson CR: Functional heterogeneity in cingulate cortex: the anterior executive and posterior evaluative regions. Cereb Cortex. 1992, 2 (6): 435-443.PubMedGoogle Scholar
- Picard N, Strick PL: Motor areas of the medial wall: a review of their location and functional activation. Cereb Cortex. 1996, 6 (3): 342-353. 10.1093/cercor/6.3.342.PubMedGoogle Scholar
- Bush G, Luu P, Posner MI: Cognitive and emotional influences in anterior cingulate cortex. Trends in cognitive sciences. 2000, 4 (6): 215-222. 10.1016/S1364-6613(00)01483-2.PubMedGoogle Scholar
- Vogt BA: Pain and emotion interactions in subregions of the cingulate gyrus. Nature reviews. 2005, 6 (7): 533-544. 10.1038/nrn1704.PubMed CentralPubMedGoogle Scholar
- Baudena P, Halgren E, Heit G, Clarke JM: Intracerebral potentials to rare target and distractor auditory and visual stimuli. III. Frontal cortex. Electroencephalography and clinical neurophysiology. 1995, 94 (4): 251-264. 10.1016/0013-4694(95)98476-O.PubMedGoogle Scholar
- Smith ME, Halgren E, Sokolik M, Baudena P, Musolino A, Liegeois-Chauvel C, Chauvel P: The intracranial topography of the P3 event-related potential elicited during auditory oddball. Electroencephalography and clinical neurophysiology. 1990, 76 (3): 235-248. 10.1016/0013-4694(90)90018-F.PubMedGoogle Scholar
- Wang C, Ulbert I, Schomer DL, Marinkovic K, Halgren E: Responses of human anterior cingulate cortex microdomains to error detection, conflict monitoring, stimulus-response mapping, familiarity, and orienting. J Neurosci. 2005, 25 (3): 604-613. 10.1523/JNEUROSCI.4151-04.2005.PubMedGoogle Scholar
- Posner MI, Petersen SE: The attention system of the human brain. Annual review of neuroscience. 1990, 13: 25-42. 10.1146/annurev.ne.13.030190.000325.PubMedGoogle Scholar
- Mesulam MM: Large-scale neurocognitive networks and distributed processing for attention, language, and memory. Annals of neurology. 1990, 28 (5): 597-613. 10.1002/ana.410280502.PubMedGoogle Scholar
- Knight RT: Decreased response to novel stimuli after prefrontal lesions in man. Electroencephalography and clinical neurophysiology. 1984, 59 (1): 9-20. 10.1016/0168-5597(84)90016-9.PubMedGoogle Scholar
- Yamaguchi S, Knight RT: P300 generation by novel somatosensory stimuli. Electroencephalography and clinical neurophysiology. 1991, 78 (1): 50-55. 10.1016/0013-4694(91)90018-Y.PubMedGoogle Scholar
- Kekoni J, Hamalainen H, McCloud V, Reinikainen K, Naatanen R: Is the somatosensory N250 related to deviance discrimination or conscious target detection?. Electroencephalography and clinical neurophysiology. 1996, 100 (2): 115-125. 10.1016/0013-4694(95)00231-6.PubMedGoogle Scholar
- Kida T, Nishihira Y, Wasaka T, Nakata H, Sakamoto M: Passive enhancement of the somatosensory P100 and N140 in an active attention task using deviant alone condition. Clin Neurophysiol. 2004, 115 (4): 871-879. 10.1016/j.clinph.2003.11.037.PubMedGoogle Scholar
- Kida T, Wasaka T, Nakata H, Akatsuka K, Kakigi R: Active attention modulates passive attention-related neural responses to sudden somatosensory input against a silent background. Experimental brain research Experimentelle Hirnforschung. 2006, 175 (4): 609-617. 10.1007/s00221-006-0578-4.PubMedGoogle Scholar
- Watanabe S, Kakigi R, Koyama S, Hoshiyama M, Kaneoke Y: Pain processing traced by magnetoencephalography in the human brain. Brain topography. 1998, 10 (4): 255-264. 10.1023/A:1022218906322.PubMedGoogle Scholar
- Qiu Y, Inui K, Wang X, Nguyen BT, Tran TD, Kakigi R: Effects of distraction on magnetoencephalographic responses ascending through C-fibers in humans. Clin Neurophysiol. 2004, 115 (3): 636-646. 10.1016/j.clinph.2003.10.017.PubMedGoogle Scholar
- Thompson RF: In search of memory traces. Annual review of psychology. 2005, 56: 1-23. 10.1146/annurev.psych.56.091103.070239.PubMedGoogle Scholar
- Neves G, Cooke SF, Bliss TV: Synaptic plasticity, memory and the hippocampus: a neural network approach to causality. Nature reviews. 2008, 9 (1): 65-75. 10.1038/nrn2303.PubMedGoogle Scholar
- Squire LR, Wixted JT, Clark RE: Recognition memory and the medial temporal lobe: a new perspective. Nature reviews. 2007, 8 (11): 872-883. 10.1038/nrn2154.PubMed CentralPubMedGoogle Scholar
- Rowland DC, Kentros CG: Potential anatomical basis for attentional modulation of hippocampal neurons. Annals of the New York Academy of Sciences. 2008, 1129: 213-224. 10.1196/annals.1417.014.PubMedGoogle Scholar
- Donchin E, Coles MGH: Is the P300 component a manifestation of context updating?. Brain Behav Science. 1988, 100: 357-374.Google Scholar
- Sutton S, Braren M, Zubin J, John ER: Evoked-potential correlates of stimulus uncertainty. Science. 1965, 150 (700): 1187-1188. 10.1126/science.150.3700.1187.PubMedGoogle Scholar
- Picton TW, Hillyard SA: Endogeneous event-related potentials. 1988, New York: OxfordGoogle Scholar
- Soltani M, Knight RT: Neural origins of the P300. Critical reviews in neurobiology. 2000, 14 (3–4): 199-224.PubMedGoogle Scholar
- Linden DE: The p300: where in the brain is it produced and what does it tell us?. Neuroscientist. 2005, 11 (6): 563-576. 10.1177/1073858405280524.PubMedGoogle Scholar
- Courchesne E, Hillyard SA, Galambos R: Stimulus novelty, task relevance and the visual evoked potential in man. Electroencephalography and clinical neurophysiology. 1975, 39 (2): 131-143. 10.1016/0013-4694(75)90003-6.PubMedGoogle Scholar
- Squires NK, Squires KC, Hillyard SA: Two varieties of long-latency positive waves evoked by unpredictable auditory stimuli in man. Electroencephalography and clinical neurophysiology. 1975, 38 (4): 387-401. 10.1016/0013-4694(75)90263-1.PubMedGoogle Scholar
- Polich J, Eischen SE, Collins GE: P300 from a single auditory stimulus. Electroencephalography and clinical neurophysiology. 1994, 92 (3): 253-261. 10.1016/0168-5597(94)90068-X.PubMedGoogle Scholar
- Tarkka IM, Stokic DS: Source localization of P300 from oddball, single stimulus, and omitted-stimulus paradigms. Brain topography. 1998, 11 (2): 141-151. 10.1023/A:1022258606418.PubMedGoogle Scholar
- Naatanen R, Gaillard AW, Mantysalo S: Early selective-attention effect on evoked potential reinterpreted. Acta psychologica. 1978, 42 (4): 313-329. 10.1016/0001-6918(78)90006-9.PubMedGoogle Scholar
- Yamazaki T, Kamijo K, Kenmochi A, Fukuzumi S, Kiyuna T, Takaki Y, Kuroiwa Y: Multiple equivalent current dipole source localization of visual event-related potentials during oddball paradigm with motor response. Brain topography. 2000, 12 (3): 159-175. 10.1023/A:1023467806268.PubMedGoogle Scholar
- Valeriani M, Fraioli L, Ranghi F, Giaquinto S: Dipolar source modeling of the P300 event-related potential after somatosensory stimulation. Muscle & nerve. 2001, 24 (12): 1677-1686. 10.1002/mus.1203.Google Scholar
- Okada YC, Kaufman L, Williamson SJ: The hippocampal formation as a source of the slow endogenous potentials. Electroencephalography and clinical neurophysiology. 1983, 55 (4): 417-426. 10.1016/0013-4694(83)90130-X.PubMedGoogle Scholar
- Halgren E, Baudena P, Clarke JM, Heit G, Marinkovic K, Devaux B, Vignal JP, Biraben A: Intracerebral potentials to rare target and distractor auditory and visual stimuli. II. Medial, lateral and posterior temporal lobe. Electroencephalography and clinical neurophysiology. 1995, 94 (4): 229-250. 10.1016/0013-4694(95)98475-N.PubMedGoogle Scholar
- Schroeder CE, Foxe J: Multisensory contributions to low-level, 'unisensory' processing. Current opinion in neurobiology. 2005, 15 (4): 454-458. 10.1016/j.conb.2005.06.008.PubMedGoogle Scholar
- Barnes CL, Pandya DN: Efferent cortical connections of multimodal cortex of the superior temporal sulcus in the rhesus monkey. The Journal of comparative neurology. 1992, 318 (2): 222-244. 10.1002/cne.903180207.PubMedGoogle Scholar
- Pandya DN: Anatomy of the auditory cortex. Revue neurologique. 1995, 151 (8–9): 486-494.PubMedGoogle Scholar
- Baylis GC, Rolls ET, Leonard CM: Functional subdivisions of the temporal lobe neocortex. J Neurosci. 1987, 7 (2): 330-342.PubMedGoogle Scholar
- Hikosaka K, Iwai E, Saito H, Tanaka K: Polysensory properties of neurons in the anterior bank of the caudal superior temporal sulcus of the macaque monkey. Journal of neurophysiology. 1988, 60 (5): 1615-1637.PubMedGoogle Scholar
- Schroeder CE, Foxe JJ: The timing and laminar profile of converging inputs to multisensory areas of the macaque neocortex. Brain Res Cogn Brain Res. 2002, 14 (1): 187-198. 10.1016/S0926-6410(02)00073-3.PubMedGoogle Scholar
- Schroeder CE, Lindsley RW, Specht C, Marcovici A, Smiley JF, Javitt DC: Somatosensory input to auditory association cortex in the macaque monkey. Journal of neurophysiology. 2001, 85 (3): 1322-1327.PubMedGoogle Scholar
- Fu KM, Johnston TA, Shah AS, Arnold L, Smiley J, Hackett TA, Garraghty PE, Schroeder CE: Auditory cortical neurons respond to somatosensory stimulation. J Neurosci. 2003, 23 (20): 7510-7515.PubMedGoogle Scholar
- Brosch M, Selezneva E, Scheich H: Nonauditory events of a behavioral procedure activate auditory cortex of highly trained monkeys. J Neurosci. 2005, 25 (29): 6797-6806. 10.1523/JNEUROSCI.1571-05.2005.PubMedGoogle Scholar
- Kayser C, Petkov CI, Augath M, Logothetis NK: Integration of touch and sound in auditory cortex. Neuron. 2005, 48 (2): 373-384. 10.1016/j.neuron.2005.09.018.PubMedGoogle Scholar
- Werner-Reiss U, Kelly KA, Trause AS, Underhill AM, Groh JM: Eye position affects activity in primary auditory cortex of primates. Curr Biol. 2003, 13 (7): 554-562. 10.1016/S0960-9822(03)00168-4.PubMedGoogle Scholar
- Fu KM, Shah AS, O'Connell MN, McGinnis T, Eckholdt H, Lakatos P, Smiley J, Schroeder CE: Timing and laminar profile of eye-position effects on auditory responses in primate auditory cortex. Journal of neurophysiology. 2004, 92 (6): 3522-3531. 10.1152/jn.01228.2003.PubMedGoogle Scholar
- Ghazanfar AA, Maier JX, Hoffman KL, Logothetis NK: Multisensory integration of dynamic faces and voices in rhesus monkey auditory cortex. J Neurosci. 2005, 25 (20): 5004-5012. 10.1523/JNEUROSCI.0799-05.2005.PubMedGoogle Scholar
- Lakatos P, Chen CM, O'Connell MN, Mills A, Schroeder CE: Neuronal oscillations and multisensory interaction in primary auditory cortex. Neuron. 2007, 53 (2): 279-292. 10.1016/j.neuron.2006.12.011.PubMed CentralPubMedGoogle Scholar
- McGurk H, MacDonald J: Hearing lips and seeing voices. Nature. 1976, 264 (5588): 746-748. 10.1038/264746a0.PubMedGoogle Scholar
- Sams M, Mottonen R, Sihvonen T: Seeing and hearing others and oneself talk. Brain Res Cogn Brain Res. 2005, 23 (2–3): 429-435. 10.1016/j.cogbrainres.2004.11.006.PubMedGoogle Scholar
- Colin C, Radeau M, Soquet A, Demolin D, Colin F, Deltenre P: Mismatch negativity evoked by the McGurk-MacDonald effect: a phonetic representation within short-term memory. Clin Neurophysiol. 2002, 113 (4): 495-506. 10.1016/S1388-2457(02)00024-X.PubMedGoogle Scholar
- Schurmann M, Caetano G, Hlushchuk Y, Jousmaki V, Hari R: Touch activates human auditory cortex. NeuroImage. 2006, 30 (4): 1325-1331. 10.1016/j.neuroimage.2005.11.020.PubMedGoogle Scholar
- Schurmann M, Caetano G, Jousmaki V, Hari R: Hands help hearing: facilitatory audiotactile interaction at low sound-intensity levels. The Journal of the Acoustical Society of America. 2004, 115 (2): 830-832. 10.1121/1.1639909.PubMedGoogle Scholar
- Jousmaki V, Hari R: Parchment-skin illusion: sound-biased touch. Curr Biol. 1998, 8 (6): R190-10.1016/S0960-9822(98)70120-4.PubMedGoogle Scholar
- Calvert GA, Bullmore ET, Brammer MJ, Campbell R, Williams SC, McGuire PK, Woodruff PW, Iversen SD, David AS: Activation of auditory cortex during silent lipreading. Science. 1997, 276 (5312): 593-596. 10.1126/science.276.5312.593.PubMedGoogle Scholar
- Schroger E, Widmann A: Speeded responses to audiovisual signal changes result from bimodal integration. Psychophysiology. 1998, 35 (6): 755-759. 10.1017/S0048577298980714.PubMedGoogle Scholar
- Molholm S, Sehatpour P, Mehta AD, Shpaner M, Gomez-Ramirez M, Ortigue S, Dyke JP, Schwartz TH, Foxe JJ: Audio-visual multisensory integration in superior parietal lobule revealed by human intracranial recordings. Journal of neurophysiology. 2006, 96 (2): 721-729. 10.1152/jn.00285.2006.PubMedGoogle Scholar
- Fort A, Delpuech C, Pernier J, Giard MH: Dynamics of cortico-subcortical cross-modal operations involved in audio-visual object detection in humans. Cereb Cortex. 2002, 12 (10): 1031-1039. 10.1093/cercor/12.10.1031.PubMedGoogle Scholar
- Murray MM, Molholm S, Michel CM, Heslenfeld DJ, Ritter W, Javitt DC, Schroeder CE, Foxe JJ: Grabbing your ear: rapid auditory-somatosensory multisensory interactions in low-level sensory cortices are not constrained by stimulus alignment. Cereb Cortex. 2005, 15 (7): 963-974. 10.1093/cercor/bhh197.PubMedGoogle Scholar
- Senkowski D, Molholm S, Gomez-Ramirez M, Foxe JJ: Oscillatory beta activity predicts response speed during a multisensory audiovisual reaction time task: a high-density electrical mapping study. Cereb Cortex. 2006, 16 (11): 1556-1565. 10.1093/cercor/bhj091.PubMedGoogle Scholar
- Talsma D, Woldorff MG: Selective attention and multisensory integration: multiple phases of effects on the evoked brain activity. J Cogn Neurosci. 2005, 17 (7): 1098-1114. 10.1162/0898929054475172.PubMedGoogle Scholar
- Saint-Amour D, De Sanctis P, Molholm S, Ritter W, Foxe JJ: Seeing voices: High-density electrical mapping and source-analysis of the multisensory mismatch negativity evoked during the McGurk illusion. Neuropsychologia. 2007, 45 (3): 587-597. 10.1016/j.neuropsychologia.2006.03.036.PubMed CentralPubMedGoogle Scholar
- Talsma D, Doty TJ, Woldorff MG: Selective attention and audiovisual integration: is attending to both modalities a prerequisite for early integration?. Cereb Cortex. 2007, 17 (3): 679-690. 10.1093/cercor/bhk016.PubMedGoogle Scholar