Skip to main content

Learning modular representations from global sparse coding networks

The introduction of the efficient coding hypothesis by Barlow in 1961 marked the beginning of a detailed investigation into how neurons could adapt their receptive field (RF) profiles to represent their external sensory environment more efficiently. The efficiency of the neural code was originally studied in the context of a single neuron, primarily with respect to the amount of information that could be sent from a single cell across an axonal communication channel. With the development of techniques to examine the activity of multiple neurons in intact neural circuits, our view of information processing in sensory systems has shifted from a neuron-centric view to a network-centric one. The sparse coding hypothesis attempts to describe how efficient coding can be performed at the network level. In sparse coding models, a network’s representation is deemed efficient if the number of neurons that are active within a population is small relative to the size of the population.

Mounting evidence shows that sensory systems employ sparse population coding, both in experimental [1] and computational modeling [2] studies. While these studies have focused on testing the sparse coding hypothesis with respect to the development of RF profiles in different sensory environments, predictions of information processing in mature neural sparse coding circuits have yet to be studied in detail, let alone tested experimentally. To determine relevant predictions of sparse coding models for mature neural circuits, we began by studying current neurally plausible models for sparse coding [3],4 that employ a number of processes known to take place within cortical networks, including recurrent inhibition, stimulus-driven excitation, and thresholding of each cell’s membrane potential. These models also predict that in order to perform sparse approximation across a network of neurons, the strength of inhibition amongst any pair of neurons is given by the coherence between the RF profiles of the pre- and post-synaptic cell. This implies that as a cell’s RF profile changes over time, the cell would need to alert all interneurons that synapse onto excitatory cells with overlapping receptive fields of the precise changes in its RF profile. Furthermore, predictions made about the spatial arrangement of cells differ from observations of the spatial organization of cells in the cortex.

To move towards a more biologically accurate model for sparse coding in sensory systems, we incorporated a simple Hebbian learning rule into the locally competitive sparse coding model described in [3]. We found that as connections are strengthened amongst all the cells that become active in response to a given stimuli, the networks that emerge exhibit modularity (akin to columnar architectures) and small-world topologies (high clustering with small average path length), both of which have been observed in networks in the cortex. Using this model, we go on to show that under certain assumptions, orientation maps emerge. In addition to suggesting that sparse coding must be refined to incorporate stimulus-dependent plasticity, our results suggest that analyzing the structure of the coherence amongst the RFs of neighboring neurons should enable a more principled investigation of the sparse coding hypothesis in intact mature neural circuits.


  1. Hsu AS, Dayan P: An unsupervised learning model of neural plasticity: Orientation selectivity in goggle-reared kittens. Vision Research. 2007, 47 (22): 2868-2877. 10.1016/j.visres.2007.07.023.

    Article  PubMed  Google Scholar 

  2. Olshausen BA, Field DJ: Emergence of Simple-Cell Receptive Field Properties by Learning a Sparse Code for Natural Images. Nature. 1996, 381: 607-609. 10.1038/381607a0.

    Article  CAS  PubMed  Google Scholar 

  3. Rozell CJ, Johnson DH, Baraniuk RG, Olshausen BA: Sparse Coding via Thresholding and Local Competition in Neural Circuits. Neural Computation. 20: 2526-2563. 10.1162/neco.2008.03-07-486.

  4. Rehn M, Sommer FT: A network that uses few active neurones to code visual input predicts the diverse shapes of cortical receptive fields. J Comput Neurosci. 2007, 22 (2): 135-146. 10.1007/s10827-006-0003-9.

    Article  PubMed  Google Scholar 

Download references

Author information

Authors and Affiliations


Corresponding author

Correspondence to Eva L Dyer.

Rights and permissions

Open Access This article is published under license to BioMed Central Ltd. This is an Open Access article is distributed under the terms of the Creative Commons Attribution 2.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and Permissions

About this article

Cite this article

Dyer, E.L., Johnson, D.H. & Baraniuk, R.G. Learning modular representations from global sparse coding networks. BMC Neurosci 11 (Suppl 1), P131 (2010).

Download citation

  • Published:

  • DOI:


  • Code Network
  • Sparse Code
  • Average Path Length
  • Sensory Environment
  • Hebbian Learning