Properties of synaptic plasticity rules implementing actor-critic temporal-difference learning
© Potjans et al; licensee BioMed Central Ltd. 2008
Published: 11 July 2008
There is considerable interest in establishing a link between system-level learning and synaptic plasticity [1–3]. In a previous study  we presented a specific set of biologically plausible synaptic plasticity rules implementing temporal-difference (TD) learning in a spiking neuronal network inspired by the actor-critic architecture . We showed the equivalence between the plasticity rules and the traditional discrete-time TD(0) algorithm and demonstrated that the network learns a complex task with a similar speed to its discrete time counterpart and attains the same equilibrium performance. However, the set of learning rules represents only one possible way in which actor-critic TD learning could be implemented in the brain, and so the model has only limited predictive power for experimental work.
Here, we extract properties of synaptic plasticity rules that suffice to implement actor-critic TD(0) learning, under the assumption that states are represented by elevated rates in disjunct sets of neurons. On this basis we define generalized classes of continuous time synaptic plasticity rules that implement value function and policy updates. The main property is that the amount and sign of the weight update depends on a characteristic change in the activity of the critic module combined with a global reward signal. We present concrete examples belonging to the defined class and demonstrate that they are able to solve a non-trivial task. We further analyze to what extent the defined class of plasticity rules are compatible with experimental findings of synaptic plasticity [6, 7].
Partially funded by DIP F1.2, BMBF Grant 01GQ0420 to the Bernstein Center for Computational Neuroscience Freiburg, and EU Grant 15879 (FACETS).
- Izhikevich EM: Solving the distal reward problem through linkage of STDP and dopamine signaling. Cerebral Cortex. 2007, 17 (10): 2443-2452. 10.1093/cercor/bhl152.View ArticlePubMedGoogle Scholar
- Baras D, Meir R: Reinforcement Learning, Spike-Time-Dependent Plasticity, and the BCM Rule. Neural Computation. 2007, 19: 2245-2279. 10.1162/neco.2007.19.8.2245.View ArticlePubMedGoogle Scholar
- Florian RV: Reinforcement learning Through Modulation of Spike-Timing – Dependent Synaptic Plasticity. Neural Computation. 2007, 19: 1468-1502. 10.1162/neco.2007.19.6.1468.View ArticlePubMedGoogle Scholar
- Potjans W, Morrison A, Diesmann M: A spiking neural network model for the actor-critic temporal-difference learning algorithm. 342.6. 37th SFN meeting, San Diego, USA.Google Scholar
- Sutton RS, Barto AG: Reinforcement learning, An Introduction. 1998, The MIT pressGoogle Scholar
- Kirkwood A, Rioult MG, Bear MF: Experience-dependent modification of synaptic plasticity in visual cortex. Nature. 1996, 381: 526-528. 10.1038/381526a0.View ArticlePubMedGoogle Scholar
- Reynolds JNJ, Wickens JR: Dopamine-dependent plasticity of corticostriatal synapses. Neural Networks. 2002, 15: 507-521. 10.1016/S0893-6080(02)00045-X.View ArticlePubMedGoogle Scholar
This article is published under license to BioMed Central Ltd.