Publication Details

SELECT * FROM publications WHERE Record_Number=11232
Reference TypeThesis
Author(s)Tateo, D.
TitleBuilding structured hierarchical agents
Journal/Conference/Book TitlePh.D. Thesis
KeywordsHierarchical Reinforcement Learning; Inverse Reinforcement Learning
AbstractThere is an increasing interest in Reinforcement Learning to solve new and more challenging problems. We are now able to solve moderately complex environments thanks to the advances in Policy Search methods and Deep Reinforcement Learning, even when using low-level data representations as images or raw sensor inputs. These advances have widened the set of application contexts in which machine learning techniques can be applied, bringing in the near future the application of these techniques in other emerging fields of research, such as robotics and unmanned autonomous vehicles. In these applications, autonomous agents are required to solve very complex tasks, using information taken from low-level sensors, in uncontrolled, dangerous, and unknown scenarios. However, many of these new methods suffer from major drawbacks: lack of theoretical results, even when based on sound theoretical frameworks, lack of interpretability of the learned behavior, instability of the learning process, domain knowledge not exploited systematically, extremely data hungry algorithms. The objective of this thesis is to address some of these problems and provide a set of tools to simplify the design of Reinforcement Learning agents, particularly when it comes to robotic systems that share some common characteristics. Most of these systems use continuous state and action variables that may need a fine-grained precision, making a good variety of deep learning approaches ineffective. They may exhibit different dynamics between different parts of the system, leading to a natural division based on different time scales, variable magnitudes, and abstraction levels. Finally, some of them are even difficult to formalize as a Reinforcement Learning task, making it difficult to define a reward function, while some human (or non-human) experts may be able to provide behavioral demonstrations. Based on these assumptions, we propose two approaches to improve the applicability of Reinforcement Learning techniques in these scenarios: hierarchical approaches to Reinforcement Learning, to exploit the structure of the problem, and Inverse Reinforcement Learning, which are a set of techniques able to extract the reward function i.e., the representation of the objective pursued by the agent, and the desired behavior from a set of experts' demonstrations. From these ideas follow the two major contributions of this work: a new Hierarchical Reinforcement Learning framework based on the Control Theory framework, which is particularly well-suited for robotic systems, and a family of Inverse Reinforcement Learning algorithms that are able to learn a suitable reward function for tasks (or subtasks) difficult to formalize as a reward function, particularly when demonstrations come from a set of different suboptimal experts. Our proposals make it possible to easily design a complex hierarchical control structure and learn the policy either by interacting directly with the environment or providing demonstrations for some subtasks or for the whole system.
Link to PDF


zum Seitenanfang