Skip to main content
  • Poster presentation
  • Open access
  • Published:

Spiking neural network model of reinforcement learning in the honeybee implemented on the GPU

Honeybees can learn and perform complex behavioral tasks despite their small brains that contain less than a million neurons. At the same time they are accessible to physiological experiments and the relatively small number of neurons in their brain lends itself to quite detailed numerical simulations. Bees therefore are a good model system for studying sensory cognition and reinforcement learning.

We have shown in earlier work [1] that the anatomy and known electrophysiological properties of the olfactory pathway of insects in combination with spike-timing dependent plasticity (STDP) and lateral inhibition lend themselves to an unsupervised self-organization of synaptic connections for the recognition of odors. Here we extend this model by adding mechanisms of reinforcement learning, as suggested by [2] (see Figure 1). We employ a three factor learning rule where plasticity is governed by pre-synaptic and post-synaptic activity and a global octopaminergic/dopaminergic reinforcement signal, triggered by a reward. We investigated the role of feed-forward and feedback mechanisms, as well as the role of the connectivity initially achieved by unsupervised STDP.

Figure 1
figure 1

Network diagram for the hypothesized model of reinforcement learning in the honeybee olfactory system. Excitatory connections are shown in black, inhibitory connections in blue and learning synapses in red. Grey arrows represent the abstractions modeled by implicit mechanisms. The model consists of the antennal lobe, lateral horn interneurons, mushroom body Kenyon cells and lobe neurons, and an octopaminergic/dopaminergic pathway for reinforcement, classically considered to be the VUMmx1 neuron. A conditioned stimulus is paired with an unconditioned stimulus (sugar to the antenna) to elicit the behavioral response (proboscis extension) in the training phase, which can then be rewarded by letting the bees drink. The association with reward facilitates plasticity in the synapses between the Kenyon cells and lobe neurons and between lobe neurons and pre-motor neurons. The size of weight changes is determined by an eligibility trace as a function of the delay between stimulus and reward.

Our model is implemented in the GeNN [3] framework, which facilitates the use of GPUs for spiking neural network simulations using a code generation framework. Because of the massive parallelism provided by GPUs, we can simulate tens of thousands of neurons in real time in the sparse firing regime relevant here. We investigated optimization strategies and neuron and synapse model choices for a better performance on the GPU. The model presented here is a stepping-stone to more sophisticated learning models and multi-sensory integration in the Green Brain Project [4], in which we aim to control a flying robot with a simulation of learning and decision making mechanisms in the honeybee related both to the olfactory and visual pathways.

References

  1. Nowotny T, Huerta R, Abarbanel HD, Rabinovich MI: Self-organization in the olfactory system: one shot odor recognition in insects. Biological Cybernetics. 2005, 93 (6): 436-446.

    Article  PubMed  Google Scholar 

  2. Smith D, Wessnitzer J, Webb B: A model of associative learning in the mushroom body. Biological Cybernetics. 2008, 99 (2): 89-103.

    Article  PubMed  Google Scholar 

  3. Yavuz E, Turner J, Nowotny T: Simulating spiking neural networks on massively parallel graphical processing units using a code generation approach with GeNN. BMC Neuroscience. 2014, 15 (Suppl 1): O1-

    Article  PubMed Central  Google Scholar 

  4. The Green Brain project. accessed 20-02-2015, [http://www.greenbrainproject.co.uk/]

Download references

Acknowledgements

This project is supported by the EPSRC (Green Brain Project, grant number EP/J019690/1).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Esin Yavuz.

Rights and permissions

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yavuz, E., Maul, P. & Nowotny, T. Spiking neural network model of reinforcement learning in the honeybee implemented on the GPU. BMC Neurosci 16 (Suppl 1), P181 (2015). https://doi.org/10.1186/1471-2202-16-S1-P181

Download citation

  • Published:

  • DOI: https://doi.org/10.1186/1471-2202-16-S1-P181

Keywords