Reinforcement learning is from our perspective the automatic design of approximately optimal controllers from measurements. In traditional (optimal) control, the smart human in the loop decides how to measure and model the system. In RL, on the other hand, the optimal controller is constructed by the RL system directly from measurements; however, the way to the optimal controller can require extensive prestructuring through structured policies, value functions or models. In this page, I want to list some of the projects I am working on or have worked on but this list will always be fairly incomplete.
Our general goal in reinforcement learning is the development of methods which scale into the dimensionality of humanoid robots and can generate actions for seven or more degrees of freedom, e.g., for a human arm. Such problems are a tremendous challenge for reinforcement learning as they require a state space of 21 or more dimensions (one dimension for each joint position, velocity and acceleration) and an action space of seven dimensions.
While supervised statistical learning techniques have significant applications for model and imitation learning, they do not suffice for all motor learning problems, particularly when no expert teacher or idealized desired behavior is available. Thus, both robotics and the understanding of human motor control require reward (or cost) related self-improvement. The development of efficient reinforcement learning methods is therefore essential for the success of learning in motor control.
However, reinforcement learning in high-dimensional spaces such as manipulator and humanoid robotics is extremely difficult as a complete exploration of the underlying state-action spaces is impossible and few existing techniques scale into this domain.
Nevertheless, it is obvious that humans also never need such an extensive exploration in order to learn new motor skills and instead rely upon a combination of both watching a teacher and subsequent self-improvement. In more technical terms: first, a control policy is obtained by imitation and then improved using reinforcement learning. It is essential that only local policy search techniques, e.g., policy gradient methods, are applied as a rapid change to the policy would result into a complete unlearning of the policy and might also result into unstable control policies which can damage the robot.
In order to bring reinforcement learning to robotics and computational motor control, we have both improved existing reinforcement learning methods as well as developed a variety of novel algorithms. At this point, we can only give a short overview of these methods: