(:requiretuid:)

Vorlesung: Robot Learning

Quick Facts

Lecturer:Jan Peters
Teaching Assistants:Marco Ewerton, Daniel Tanneberg
Classes:Wednesday, 08:55-11:30 in Room S202/C120
Language:English
Office Hours:Marco Ewerton, contact via ewerton@ias.tu-darmstadt.de
 Daniel Tanneberg, contact via daniel@robot-learning.de
TU-CAN:20-00-0629-vl Lernende Roboter
Credits:6,0
Exam:12.02.18, 12-14 Uhr in C205
Klausureinsicht: 13.03.18, 10-12 Uhr in A213

Description

(:youtube qtqubguikMk :) In the 1980s, classical robotics already reached a high level of maturity and it was able to produce large factories. For example, cars factories were completely automated. Despite these impressive achievements, unlike personal computers, modern service robots still did not leave the factories and take a seat as robot companions on our side. The reason is that it is still harder for us to program robots than computers. Usually, modern companion robots learn their duties by a mixture of imitation and trial-and-error. This new way of programming robots has a crucial consequence in the field of industry: the programming cost increases, making mass production impossible.

However, in research, this approach had a great influence and over the last ten years all top universities in the world conduct research in this area. The success of these new methods has been demonstrated in a variety of sample scenarios: autonomous helicopters learning from teachers complex maneuver, walking robot learning impressive balancing skills, self-guided cars hurtling at high speed in racetracks, humanoid robots balancing a bar in their hand and anthropomorphic arms cooking pancakes.

Accordingly, this class serves as an introduction to autonomous robot learning. The class focuses on approaches from the fields of robotics, machine learning, model learning, imitation learning, reinforcement learning and motor primitives. Application scenarios and major challenges in modern robotics will be presented as well.

We pay particular attention to interactions with the participants of the lecture, asking multiple question and appreciating enthusiastic students.

We also offer a parallel project, the Robot Learning: Integrated Project. It is designed to enable participants to understand robot learning in its full depth by directly applying methods presented in this class to real or simulated robots. We suggest motivated students to attend it as well, either during or after the Robot Learning Class!



Contents

The course gives a introduction to robotics and machine learning methods. The following topics are expected to be covered throughout the semester.

  • Robotics Basics
  • Machine Learning Basics
  • Model Learning
  • Imitation Learning
  • Optimal Decision Making
  • Optimal Control
  • Reinforcement Learning
  • Policy Search
  • Inverse Reinforcement Learning

Requirements

Mathematics from the first semesters, basic programming abilities, computer science basics.

Other Information

Lecture Slides

Nr.TopicSlides
1OrganizationSlides
2RoboticsSlides
3Machine LearningSlides Part 1Slides Part 2
4Model LearningSlides
5Imitation LearningSlides
6Optimal Control with Learned Models, Part 1: Discrete State-Action Optimal ControlSlides
7Optimal Control with Learned Models, Part 2: Continuous State-Action Optimal ControlSlides
8Value Function MethodsSlidesTD Algorithms Code
9Policy Search, Part 1: Policy Gradient MethodsSlides
10Policy Search, Part 2: Probabilistic MethodsSlides1 Slides2
11Inverse Reinforcement LearningSlides

Final Exam

  • The final exam date will be announced during the semester.
  • The exam will cover all material presented in the lectures, unless specified otherwise.
  • The exam will consist of roughly 30-35 questions and will last 90 minutes.
  • Students are allowed to bring to the exam a cheat sheet consisting of a single A4 paper. The paper must be handwritten (not printed) and you can write on both faces. You have to sign and submit the cheat sheet as well, and you can get it back at the exam correction.
  • Students are allowed to use a non-electronic dictionary and a non-programmable calculator.
  • You have to identify yourself with both a personal ID and the TU-ID.
  • This pdf shows you how the exam will look like.

Homeworks

  • Four homeworks will be given during the course.
  • Homeworks are not compulsory but we strongly suggest that you complete them as they provide hands-on experience on the topics.
  • The homework should be done in groups of two. These groups should remain unchanged for the remainder of the semester.
  • Homeworks can be handed in at any time before the deadline by dropping them in the mailbox outside Room E314@S2|02.
  • Homeworks will only be graded if they are submitted in time, that is
- Handed in before the beginning of the lecture on the delivery date, OR
- Left in the mailbox no later than ten minutes before the lecture on the delivery date.
  • Only one copy per group has to be submitted.
  • Email delivery will be accepted exceptionally, only if a good reason exists.
  • We provide a Latex template for submitting your solutions (you need to use the TUD Latex template). We highly recommend you to use it!
  • If the exercise includes programming, then code is expected to be handed in (except if noted differently).
  • We use Python for programming assignments. You are allowed to use numpy and any plotting library you like (we recommend matplotlib). You are not allowed to use scikit-learn and scipy, unless explicitly stated otherwise.
  • We encourage the creation of nice plots (different symbols for different lines) as showing the results clearly is as important as obtaining them.
  • Include only important snippets of your code, not entire files. You can include snippets in the Latex template using the 'listings' package. Alternatively, if you use ipython/jupyter notebook, you can directly export your notebook (code + plots) as HTML and print it.
  • Your code must be briefly documented.
  • The assignment may be updated after its release if any issue is detected.
  • After the deadline there will be a presentation of the solutions.
  • Partial solutions (including only plots and numerical results, no code or explanation) will be published after the above presentation.
  • After the homeworks are corrected, they will be given back to you.
  • Successfully completing the homeworks will provide up to one additional bonus point for the final grade (only if you pass the exam) according to the following rule

{$ \min\left\lbrace \frac{\text{your homework points \: (including bonus)}}{\text{homework points \: (excluding bonus)}} \: , \: 1\right\rbrace $}

!!! Homework 4 is out !!!

Homework Assignments

TopicPDFLatex TemplateAdditional MaterialPartial Solutions
Robotics and Machine Learningpdfzipcode dataset-
Model and Imitation Learningpdfzipcode dataset-
Optimal Control and Reinforcement Learningpdfzipcode figure-
Policy Search and Policy Gradientpdfzipcode-

Additional Material

Here is some small tutorials on relevant topics for the lecture. These tutorials are supporting materials for the slides.

  1. Constraint Optimization
  2. Probabilistic Regression
  3. Optimal Control (from Univ. of Stuttgart / Prof. Marc Toussaint)
  4. Gaussian Identities (from Univ. of Stuttgart / Prof. Marc Toussaint)
  5. Matrix Cookbook (from MIT)
  6. Jacobians, velocity and torque

(:else:)

To see information on the teaching material, homeworks and final exam info, please login with your TU-ID!

Literature

The most important books for this class are:

  1. B. Siciliano, L. Sciavicco. Robotics: Modelling, Planning and Control, Springer
  2. C.M. Bishop. Pattern Recognition and Machine Learning, Springer free online copy
  3. R. Sutton, A. Barto. Reinforcement Learning - An Introduction, MIT Press free online copy

Additionally, the following papers are useful for specific topics:

    •       Bib
      Deisenroth, M. P.; Neumann, G.; Peters, J. (2013). A Survey on Policy Search for Robotics, Foundations and Trends in Robotics, 21, pp.388-403.
    •       Bib
      Kober, J.; Bagnell, D.; Peters, J. (2013). Reinforcement Learning in Robotics: A Survey, International Journal of Robotics Research (IJRR), 32, 11, pp.1238-1274.
    •     Bib
      Nguyen Tuong, D.; Peters, J. (2011). Model Learning in Robotics: a Survey, Cognitive Processing, 12, 4.


Teaching Staff

Lectures will be held by Jan Peters and additionally supervised by Daniel Tanneberg and Marco Ewerton.

Jan Peters heads the Intelligent Autonomous Systems Lab at the Department of Computer Science at the TU Darmstadt. Jan has studied computer science, electrical, control, mechanical and aerospace engineering. You can find Jan Peters in the Robert-Piloty building S2 | 02 in room E314. You can also contact him through mail@jan-peters.net.

Daniel Tanneberg is a Ph.D. student at the Intelligent Autonomous Systems (IAS) Group at the Technical University of Darmstadt since October 2015. He is investigating the applicability and properties of (spiking/stochastic) deep neural networks for open-ended robot learning. He is working on the GOAL-Robots project, that aims at developing goal-based open-ended autonomous learning robots; building lifelong learning robots. You can contact him by email at daniel@robot-learning.de .
Marco Ewerton is a Ph.D. student at the IAS since January 2015. He works on the BIMROB project, which investigates how humans and robots can improve their movements by interacting with each other. Before his Ph.D., Marco completed his Master Degree in Electrical Engineering at the TU Darmstadt. You can find him in the Robert-Piloty building S2 | 02 room E226. You can also contact him through ewerton@ias.tu-darmstadt.de.


For further inquiries do not hesitate to contact us immediately!