Skip to main content

Self-organizing neural maps for multi-modal associations

To move in a complex and dynamic environment, according to the active perception theory, an agent needs to learn the multi modal correlations between its actions and the changes they induce in the environment [1]. Human beings interact with their environment through several distant organs, whose sensory flow processing are influencing each others as, for example, in the Mc Gurk effect [2]. In the functional view of the cortex, each sensory area processes a specific sensory flow and associative areas integrate these flows in a consistent representation of the world that influences in return uni-modal perceptions as for the ventriloquist effect [3]. At a mesoscopic level, the cortex shows a generic structure composed of multi-layer cortical columns. We propose a bio inspired model of perceptive map which, using a continuous and unsupervised learning, self-organizes to map a sensory data flow. Thus, using a spatial competition mechanism, a perceptive map provides an activity bump representing the current perception. In a multi modal architecture, by connecting multiple perceptive maps to an associative map with reciprocal spatially constrained connectivity, the spatial organization of all perceptions have to relax the multi modal constraints (see figure 1A). Thus, this multimodal architecture provides a way to learn multi modal correlations with generalization.

Figure 1
figure 1

A Example of use of our perceptive map model in a multi modal architecture with two sensors and one actuator. B Generic structure of one cortical column in a perceptive map.

More precisely, each perceptive map consists of generic cortical columns with computing and learning which are local and decentralized (see figure1 B). Sensitive layer receives the feed-forward flow coming from a sensor and uses the BCM learning rule [4]. This synaptic learning rule is based on the hebbian one and is able to autonomously raise a selectivity to one stimulus of the upcoming flow. From a computational point of view, the BCM rule uses a sliding threshold between long term potentiation and long term depression, which has been confirmed by biological evidences [5]. The cortical layer receives feedback influence coming from the associative map, whose activity represents the multi modal perception. The perceptive layer is based on the neural field theory [6] and it receives the sensitive activity modulated by the cortical one. Thanks to the competitive mechanism introduced by the lateral connectivity with a difference of Gaussian shape, an activity bump emerges at the map level, where the activity is the most spatially consistent, representing a consensus between the local sensation and the multi modal constraints.

To obtain a self-organization of the sensitive layer at the map level, the perceptive layer modulates the BCM activity, so that the spatial consistency of the activity bump is propagated to the selectivity organization. We have also introduced an unlearning term in the BCM learning rule in order to forget the current selectivity if it is not consistent with the received modulation. This unlearning mechanism provides plasticity to the self-organization in order to adapt to the multi modal constraints.


  1. O’Regan JK, Noë A: A sensorimotor account of vision and visual consciousness. Behavioral and brain sciences. 2001, 24 (05): 939-973. 10.1017/S0140525X01000115.

    Article  PubMed  Google Scholar 

  2. Mcgurk H, Macdonald J: Hearing lips and seeing voices. Nature. 1976, 264 (5588): 746-748. 10.1038/264746a0.

    Article  CAS  PubMed  Google Scholar 

  3. Bonath B, Noesselt T, Martinez A, Mishra J, Schwiecker K, Heinze HJ, Hillyard SA: Neural basis of the ventriloquist illusion. Current Biology. 2007, 17 (19): 1697-1703. 10.1016/j.cub.2007.08.050.

    Article  CAS  PubMed  Google Scholar 

  4. Bienenstock EL, Cooper LN, Munro PW: Theory for the development of neuron selectivity: orientation specificity and binocular interaction in visual cortex. Journal of Neuroscience. 1982, 2 (1): 32.

    CAS  PubMed  Google Scholar 

  5. Bear MF: Mechanism for a sliding synaptic modification threshold. Neuron. 1995, 15 (1): 1-4. 10.1016/0896-6273(95)90056-X.

    Article  CAS  PubMed  Google Scholar 

  6. Amari S: Dynamics of pattern formation in lateral-inhibition type neural fields. Biological Cybernetics. 1977, 27 (2): 77-87. 10.1007/BF00337259.

    Article  CAS  PubMed  Google Scholar 

Download references

Author information

Authors and Affiliations


Corresponding author

Correspondence to Mathieu Lefort.

Rights and permissions

This article is published under license to BioMed Central Ltd. This is an open access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and Permissions

About this article

Cite this article

Lefort, M., Boniface, Y. & Girau, B. Self-organizing neural maps for multi-modal associations. BMC Neurosci 12 (Suppl 1), P125 (2011).

Download citation

  • Published:

  • DOI: