Learning Mobile Manipulation Actions From Human Demonstrations An Approach To Learning And Augmenting Action Models And Their Integration Into Task Representations PDF Download

Abstract: While incredible advancements in robotics have been achieved over the last decade, direct physical interaction with an initially unknown and dynamic environment is still a challenging problem. In order to use robots as service assistants and take over household chores in the user's home environment, they must be able to perform goal directed manipulation tasks autonomously and further, learn these task intuitively from their owners. Consider for instance the task of setting a breakfast table: Although it is a relatively simple task for a human being, it poses some serious challenges to the robot. It must physically handle the users customized household environment and the objects therein, i.e., how can the items needed to set up the table be grasped and moved, how can the kitchen cabinets be opened, etc. Additionally the personal preferences of the user on how the breakfast table should be arranged must be respected. Due to the diverse characteristics of the custom objects and the individual human needs even a standard task like setting a breakfast table is impossible to pre-program before knowing the place of use and its occurrences. Therefore, the most promising way to engage robots as domestic help is to enable them to learn the tasks they should perform directly by their owners, without requiring the owner to possess any special knowledge of robotics or programming skills. Throughout this thesis we present various contributions addressing these challenges. Although learning from demonstration is a well-established approach to teaching robots without explicit programming, most approaches in literature for learning manipulation actions use kinesthetic training as these actions require thorough knowledge of the interactions between the robot and the object which can be learned directly by kinesthetic teaching since no abstraction is needed. In addition, in most current imitation learning approaches mobile platforms are not considered. In this thesis we present a novel approach to learn joint robot base and end-effector action models from observing demonstrations carried out by a human teacher. To achieve this we adapt trajectory data obtained from RGBD recordings of the human teacher performing the action to the capabilities of the robot. We formulate a graph optimization problem that the links the observed human trajectories with robot grasping capabilities and kinematic constraints between co-occurring base and gripper poses, allowing us to generate robot suitable trajectories. In a next step, we do not just learn individual manipulation actions, but to combine several actions into one task. Challenges arise from handling ambiguous goals and generalizing the task to new settings. We present an approach to learn both representations together from the same teacher demonstrations, one for individual mobile manipulation actions as described above, and one for the representation of the overall task intent. We leverage a framework based on Monte Carlo tree search to compute sequences of feasible actions imitating the teacher intention in new settings without explicitly specifying a task goal. In this way, we can reproduce complex tasks while ensuring that all composing actions are executable in the given setting. The mobile manipulation models mentioned above are encoded as dynamic systems to facilitate interaction with objects in world coordinates. However, this poses the challenge of translating kinematic constraints of the robot to the task space and including them in the action models. In this thesis we propose to couple robot base and end-effector motions generated by arbitrary dynamical systems by modulating the base velocity, while respecting the robots kinematic design. To this end we learn an approximation of the inverse reachability in closed form and implement the coupling as an obstacle avoidance problem. Furthermore, in this work we address the challenge of imitating manipulation actions, the execution of which depends on additional non-geometric quantities as, e.g., contact forces when handing over an object or measured liquid height, while pouring water into a cup. We suggest an approach to include this additional information in form of measured features directly into the action models. These features are recorded in the demonstrations alongside the geometric route of the manipulation action and their correlation is captured in a Gaussian Mixture Model that parametrizes the dynamic system used. This enables us to also couple the motion's geometric trajectory to the perceived features in the scene during action imitation. All the above described contributions were evaluated extensively in real world robot experiments on a PR2 system and a KUKA Iiwa Robot Arm