טכניון מכון טכנולוגי לישראל
הטכניון מכון טכנולוגי לישראל - בית הספר ללימודי מוסמכים  
Ph.D Thesis
Ph.D StudentRivkind Alexandr
SubjectSlowness and Memory in Recurrent Neural Networks
DepartmentDepartment of Medicine
Supervisor Professor Omri Barak


Abstract

Task-performing neural networks are commonly treated as a “black box”. In a recurrent neural network (RNN) setting, this black box is a dynamical system rather than an algebraic one. To handle non-trivial tasks long timescales memorizing past events are essential, and they indeed emerge over the course of RNN training. Understanding theoretical underpinnings  of  such slow dynamics may shed light on a variety of biological mechanisms which are modeled by similar networks. The latter include but are not limited to neuronal networks, ecological networks, gene regulatory networks as well as many other modalities.


Here we develop a mean field theory for reservoir computing networks trained to have multiple fixed-point attractors, which might be used as memory states. Our main result is that the dynamics of the network’s output in the vicinity of attractors is governed by a low-order linear ordinary differential equation. The stability of the resulting equation can be assessed, predicting training success or failure. As a consequence, networks of rectified linear units (ReLU) and of sigmoidal nonlinearities (e.g. tanh) are shown to have diametrically different properties when it comes to learning attractors. Furthermore, a characteristic time constant, which remains finite at the edge of chaos, offers an explanation of the network’s output robustness in the presence of variability of the internal neural dynamics. Finally, we will discuss an application of our approach to the analysis of sparse Boolean networks used as a gene regulatory network model.