Conference on Cognitive Computational Neuroscience

Many memory models rely on a localist framework, using a neuron or slot for each memory. However, neuroscience research suggests that memories depend on sparse, distributed representations over neurons with sparse connectivity. Accordingly, we extend a canonical localist memory model - the modern Hopfield network - to a distributed variant called the K-winner modern Hopfield network, equating the number of synaptic parameters (weights) in the localist and K-winner variants. We study both models' sequential learning, updating the parameters of the best-matching memory neurons as each new memory comes in. We find that K-winner nets that compromise slightly on initial learning and retrieval accuracy exhibit superior retention of older memories.