In search of robust memory models for artificial intelligence.
**Caveat: While readers are welcome, the thoughts below are a work in progress.
Memory imbues everything that humans find meaningful in one way or another; our personal lives are shaped by our experiential memories, our language by the memory of the cultural and pragmatic milieu in which it was formed, our biology by the ancestral memory of our DNA, and the physical world in which we find ourselves by the memory of the physical processes that brought it into being. The motivation for this work, what may be my life's work, is simple and threefold:
I believe that the endeavor of creating new, more efficient, and flexible forms of memory to be one of the most profound things humanity has done and will do.
The intelligent machines/algorithms of the future will require new memory structures, qualitatively different than those currently dominating the landscape, and achieving this will enable profound advances in our well being through its impact on medicine, energy, food production, environmental sustainability, and more.
Human memory, the ways we relate to our own and its failing in disease processes, is inextricably linked to our well being. Thus understanding this area is very personally rewarding, and who knows, maybe someday as with other aspects of human physiology we may create useful memory prosthetics.
Machine learning algorithms embody both many of the ways in which we use our intelligence and many of the ways in which memories are encoded in our minds. For example, clustering or categorization isn't just a way to improve marketing campaigns, it's also a very efficient form of memory storage and relational structuring utilized by the brain.
Phenomena of interest include:
Recall, including cued and free recall.
Generative processes such as imagination. Such processes are currently modeled by distribution sampling using algorithms like variational auto-encoders.
Recursive processes, such as when the retrieval of one memory triggers another, related to cued recall and priming.
Categorization and hierarchy.
The Problem with Differentiable Memory
The issue with differentiable memories such as those encoded in recurrent neural nets, including LSTMs, is that the memories are susceptible to interference, aka catastrophic forgetting, from the back-propagation algorithm used in training the temporally local function approximation optima.