Volume 15 Supplement 1

Abstracts from the Twenty Third Annual Computational Neuroscience Meeting: CNS*2014

Open Access

Neuromodulation by surprise: a biologically plausible model of the learning rate dynamics

  • Mohammad Javad Faraji1Email author,
  • Kerstin Preuschoff1 and
  • Wulfram Gerstner1
BMC Neuroscience201415(Suppl 1):P159

DOI: 10.1186/1471-2202-15-S1-P159

Published: 21 July 2014

Surprise is a central concept in learning, attention and the study of the neural basis of behaviour. However, how surprise affects learning and more specifically, how surprise affects synaptic learning rules in neural networks is largely undetermined. Here we study how surprise facilitates learning in different environments and how surprise can potentially modulate Hebbian learning in the form of a global factor in multi-factor learning rules.

Learning rate is a crucial factor in determining to what extent the learning agent should rely on the newly acquired information rather than the old information in building its own internal model of the external world. Both theory and empirical evidences suggest that the learning rate should be adjusted under different circumstances for having an optimal and effective learning strategy. We propose a simple and biologically plausible model that describes the dynamics of the learning rate in terms of surprise and uncertainty measures. We apply our model to three different tasks: a reversal task (Fig. 1), a dynamic decision making task, and a dynamic clustering task.
https://static-content.springer.com/image/art%3A10.1186%2F1471-2202-15-S1-P159/MediaObjects/12868_2014_Article_3708_Fig1_HTML.jpg
Figure 1

Estimation of the probability of reward delivery in a reversal task. A. Estimated reward rate. B. Reward prediction error. C. Surprise measure. D. Learning rate. E. Uncertainty measure. F. Optimal Kalman learner.

Our proposed model explains how the agent should effectively control the speed of learning in different environments such that it matches both theory and empirical evidences from human and animal subjects. This model explains why surprising events provoke humans and animals to learn faster and why they rapidly adapt to changing environments. It also addresses the question of what the effective learning rate should be in both stable (either low-risky or high-risky) and volatile environments. Here effectiveness is defined as having a higher accuracy in learning a task, for instance the estimation of the mean reward in classic reinforcement learning, for a given time and computational complexity as well as the available memory as our constraints. This study also suggests a functional connectivity pattern for the neurochemical systems that are related to contextual modulation of learning rate. Further, it explains why we need different neuromodulators with distinct functional roles to act in parallel in a broad range of distribution and proposes suitable candidates responsible for measuring different quantities we need in the model.

Declarations

Acknowledgements

This research was supported by the European Research Council (grant agreement no. 268 689, H.S.).

Authors’ Affiliations

(1)
School of Life Sciences, Brain Mind Institute and School of Computer and Communication Sciences, Ecole Polytechnique Federal de Lausanne

Copyright

© Faraji et al; licensee BioMed Central Ltd. 2014

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Advertisement