Ph.D Thesis

Ph.D StudentMankowitz Jaymin
SubjectLearning Options in Hierarchical Reinforcement Learning
DepartmentDepartment of Electrical and Computer Engineering
Supervisor PROF. Shie Mannor


Reinforcement Learning is a computational framework for solving sequential decision making tasks. In many real-world domains, the task at hand is high-dimensional, complex and difficult to solve. However, many tasks can be naturally decomposed into hierarchical abstractions; sub-modules that enable a Reinforcement Learning agent to plan at a higher level of abstraction, thus simplifying the planning problem. Reinforcement Learning represents hierarchical abstractions using temporally extended actions, also know as options or skills (Sutton et. al. 1999). Designing options is non-trivial, time-consuming and requires domain expertise. A sub-optimal set of options results in model misspecification. Although many forms of model misspecification exist in Machine Learning, we collectively define it here as comprising Feature-based Model Misspecification (FMM) and model uncertainty. This thesis proposes reinforcement learning techniques that mitigate model misspecification by learning an optimal set of options from scratch or improving upon an initially sub-optimal set of options. It also provides an interesting use-case for options in a lifelong learning setting.  The goal is to move towards a truly general option learning framework.