Properties of synaptic plasticity rules implementing actor-critic temporal-difference learning
BMC Neuroscience volume 9, Article number: P69 (2008)
There is considerable interest in establishing a link between system-level learning and synaptic plasticity [1–3]. In a previous study  we presented a specific set of biologically plausible synaptic plasticity rules implementing temporal-difference (TD) learning in a spiking neuronal network inspired by the actor-critic architecture . We showed the equivalence between the plasticity rules and the traditional discrete-time TD(0) algorithm and demonstrated that the network learns a complex task with a similar speed to its discrete time counterpart and attains the same equilibrium performance. However, the set of learning rules represents only one possible way in which actor-critic TD learning could be implemented in the brain, and so the model has only limited predictive power for experimental work.
Here, we extract properties of synaptic plasticity rules that suffice to implement actor-critic TD(0) learning, under the assumption that states are represented by elevated rates in disjunct sets of neurons. On this basis we define generalized classes of continuous time synaptic plasticity rules that implement value function and policy updates. The main property is that the amount and sign of the weight update depends on a characteristic change in the activity of the critic module combined with a global reward signal. We present concrete examples belonging to the defined class and demonstrate that they are able to solve a non-trivial task. We further analyze to what extent the defined class of plasticity rules are compatible with experimental findings of synaptic plasticity [6, 7].
Izhikevich EM: Solving the distal reward problem through linkage of STDP and dopamine signaling. Cerebral Cortex. 2007, 17 (10): 2443-2452. 10.1093/cercor/bhl152.
Baras D, Meir R: Reinforcement Learning, Spike-Time-Dependent Plasticity, and the BCM Rule. Neural Computation. 2007, 19: 2245-2279. 10.1162/neco.2007.19.8.2245.
Florian RV: Reinforcement learning Through Modulation of Spike-Timing – Dependent Synaptic Plasticity. Neural Computation. 2007, 19: 1468-1502. 10.1162/neco.2007.19.6.1468.
Potjans W, Morrison A, Diesmann M: A spiking neural network model for the actor-critic temporal-difference learning algorithm. 342.6. 37th SFN meeting, San Diego, USA.
Sutton RS, Barto AG: Reinforcement learning, An Introduction. 1998, The MIT press
Kirkwood A, Rioult MG, Bear MF: Experience-dependent modification of synaptic plasticity in visual cortex. Nature. 1996, 381: 526-528. 10.1038/381526a0.
Reynolds JNJ, Wickens JR: Dopamine-dependent plasticity of corticostriatal synapses. Neural Networks. 2002, 15: 507-521. 10.1016/S0893-6080(02)00045-X.
Partially funded by DIP F1.2, BMBF Grant 01GQ0420 to the Bernstein Center for Computational Neuroscience Freiburg, and EU Grant 15879 (FACETS).
About this article
Cite this article
Potjans, W., Morrison, A. & Diesmann, M. Properties of synaptic plasticity rules implementing actor-critic temporal-difference learning. BMC Neurosci 9 (Suppl 1), P69 (2008). https://doi.org/10.1186/1471-2202-9-S1-P69
- Synaptic Plasticity
- Neuronal Network
- Critic Module
- Elevated Rate
- Reward Signal