Learning modular representations from global sparse coding networks

The introduction of the efficient coding hypothesis by Barlow in 1961 marked the beginning of a detailed investigation into how neurons could adapt their receptive field (RF) profiles to represent their external sensory environment more efficiently. The efficiency of the neural code was originally studied in the context of a single neuron, primarily with respect to the amount of information that could be sent from a single cell across an axonal communication channel. With the development of techniques to examine the activity of multiple neurons in intact neural circuits, our view of information processing in sensory systems has shifted from a neuron-centric view to a network-centric one. The sparse coding hypothesis attempts to describe how efficient coding can be performed at the network level. In sparse coding models, a network’s representation is deemed efficient if the number of neurons that are active within a population is small relative to the size of the population.

Mounting evidence shows that sensory systems employ sparse population coding, both in experimental [1] and computational modeling [2] studies. While these studies have focused on testing the sparse coding hypothesis with respect to the development of RF profiles in different sensory environments, predictions of information processing in mature neural sparse coding circuits have yet to be studied in detail, let alone tested experimentally. To determine relevant predictions of sparse coding models for mature neural circuits, we began by studying current neurally plausible models for sparse coding [3],4 that employ a number of processes known to take place within cortical networks, including recurrent inhibition, stimulus-driven excitation, and thresholding of each cell’s membrane potential. These models also predict that in order to perform sparse approximation across a network of neurons, the strength of inhibition amongst any pair of neurons is given by the coherence between the RF profiles of the pre- and post-synaptic cell. This implies that as a cell’s RF profile changes over time, the cell would need to alert all interneurons that synapse onto excitatory cells with overlapping receptive fields of the precise changes in its RF profile. Furthermore, predictions made about the spatial arrangement of cells differ from observations of the spatial organization of cells in the cortex.

To move towards a more biologically accurate model for sparse coding in sensory systems, we incorporated a simple Hebbian learning rule into the locally competitive sparse coding model described in [3]. We found that as connections are strengthened amongst all the cells that become active in response to a given stimuli, the networks that emerge exhibit modularity (akin to columnar architectures) and small-world topologies (high clustering with small average path length), both of which have been observed in networks in the cortex. Using this model, we go on to show that under certain assumptions, orientation maps emerge. In addition to suggesting that sparse coding must be refined to incorporate stimulus-dependent plasticity, our results suggest that analyzing the structure of the coherence amongst the RFs of neighboring neurons should enable a more principled investigation of the sparse coding hypothesis in intact mature neural circuits.