Breaking the virtual barrier: real-time interactions with spiking neural models

The unrivaled complexity of the human brain has driven many researchers towards larger and more detailed models of neural processing. Often run on remote high-performance computing architectures, these simulations can be difficult to access at significant levels of detail. In general, after simulations are completed, the results are analyzed off-line. This paradigm can make development of large models exploring complex and time-consuming simulations, such as learning or persistent neural activity, very difficult. Presented here is a toolkit, dubbed NCSTools, used for real-time interactions with large-scale neural simulations run on the NeoCortical Simulator (NCS).

NCSTools is a remote monitoring package that provides a number of options for input, output, and modification of a running simulation. Input stimuli can be voltage, current, or probabilities of firing. For output there are several options for both collection and visualization. Information can be compiled as cell voltages, currents, synaptic efficacy, or spike-events within a population. The model information flow can be altered dynamically by NCSTools, as can model parameters such as spike timing dependent plasticity, long-term potentiation, and long-term depression. Additionally, NCSTools includes a simple network server interface that supports multiple connections from different client programs. These programs can be used for control of defined NCSTools actions, monitoring specific neural information, or synchronization with the simulation. This allows dynamic construction of distributed neural processing systems.

Recently, NCSTools was utilized for real-time interactions between a Hebbian-STDP enabled neural simulation, a virtual robotic interface, and a human participant. Presented in Figure 1 is a diagrammatic view of the learning scenario. Processed visual information, in this case the color of a card, was sent to NCSTools. The information was then converted and sent to the neural simulation running on a remote computing cluster. As the simulation proceeded, the competing neural areas of visual and motor processing were monitored by NCSTools. The resulting activity was correlated with a pointing action to one of two colored balls that matched the color presented. After the robot completed pointing, NCSTools would provide the human interacting with the simulation an interface to reward the robot if it pointed to the correct ball. The reward, analogous to a dopaminergic increase, resulted in an STDP dependent increase in synaptic efficacy.

Diagram of a Virtual Neuro-Robotic Interface constructed around NCSTools.

Full size image

The ability to monitor and modify simulations in real-time can be incredibly useful in large-scale spiking-network research. More importantly, this demonstrates another step towards multi-scale visualization of neural simulations in a virtual environment.