Novel local learning rule for neural adaptation fits Hopfield memory networks efficiently and optimally

In 1982, motivated by neural modeling work of [3] and the Ising spin glass model from statistical physics [2], Hopfield introduced a method for the storage and retrieval of binary patterns in an auto-associative neural-network [1]. However, existing techniques for training Little-Hopfield networks suffer either from limited pattern capacity or excessive training time, and they exhibit poor performance when trained on unlabeled, corrupted memories.

Our main theoretical contributions here are the introduction of a tractable and neurally-plausible algorithm MPF for the optimal storage of patterns in a Little-Hopfield network, a proof that the capacity of such a network is at least one pattern per neuron, and a novel local learning rule for training neural networks. Our approach is inspired by minimum probability flow [4], which fits probabilistic models without computations with a partition function.

We also present several experimental results. Compared with standard techniques for Little-Hopfield pattern storage, our method is shown to be superior in efficiency and generalization (Figure 1). Another finding is that our algorithm can store many patterns in a Little-Hopfield network from highly corrupted (unlabeled) samples of them (Figure 2). We also store 64 × 64 binary images of human fingerprints from highly corrupted versions (Figure not shown).