טכניון מכון טכנולוגי לישראל
הטכניון מכון טכנולוגי לישראל - בית הספר ללימודי מוסמכים  
M.Sc Thesis
M.Sc StudentTurner Elia
SubjectCharting and Navigating the Space of Solutions for Recurrent
Neural Networks
DepartmentDepartment of Computer Science
Supervisor ASSOCIATE PROF. Omri Barak
Full Thesis textFull thesis text - English Version


Abstract

In recent years, trained recurrent neural networks (RNNs) were used as models for the neural activity of behaving animals. In this framework, networks are trained to perform a task that is similar to the experimental one. Features of the network's behavior and neural activity are then compared to neural recordings. Despite this increasing use, it is still unknown in which cases this approach will work. In particular, the match between model and data seems to suggest a unique solution found by both biology and the artificial network -- a puzzling conjecture.

Recent work (Maheswaranathan et al., 2019) addressed the uniqueness problem, and proposed that the solutions to various canonical tasks are, from a topological perspective, widely universal. Here, we study a slightly more complex task -- the Ready-Set-Go timing task. We find that universality no longer holds for this task, and that even identical settings can lead to qualitatively different solutions - both from behavioral and neuronal perspectives. We discover these differences by testing the trained networks ability to extrapolate, as perturbation to a system often reveals hidden structure. With the goal of understanding the solution space, we cluster the solutions into discrete sets and characterize each - showing that the neural and behavioral clusters are highly consistent. We draw a low-dimensional map of the solution space and sketch the training process as a trajectory in it. We suggest that the effects of updating parameters during training often take form as a bifurcation in the governing discrete ODE, which results in a topological change in the dynamical mechanism and in behavior.

Moreover, we explore the question of nature vs. nurture - the effect of the initial weights vs. training set over the final solution and show that, in our setting, only the former has a meaningful impact over the learned solution