Interactive Multi-Modal Robot Programming

Soshi Iba
doctoral dissertation, tech. report CMU-RI-TR-04-50, Robotics Institute, Carnegie Mellon University, May, 2004

  • Adobe portable document format (pdf) (3MB)
Copyright notice: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.

As robots enter the human environment and come in contact with inexperienced users, they need to be able to interact with users in a multi-modal fashion-keyboard and mouse are no longer acceptable as the only input modalities. Humans should be able to communicate with robots using methods as similar as possible to the concise, rich, and diverse means they use to communicate with one another. This thesis is an investigation of how one can improve user's ability to control and program a mobile robot. The goal is a comprehensive multi-modal human-machine interface that allows non-experts to compose robot programs conveniently. Two key characteristics of this novel programming approach are that the system infer the user's intent to support interaction, and that the user can provide feedback interactively through intuitive interface, at any time. The framework takes a three-step approach to the problem: multi-modal recognition, intention interpretation, and prioritized task execution. The multi- modal recognition module translates hand gestures and spontaneous speech into a structured symbolic data stream without abstracting away the user's intent. The intention interpretation module selects the appropriate primitives based on the user input, current state, and robot sensor data. Finally, the prioritized task execution module selects and executes primitives based on current state, sensor input, and the task given by the previous step. Depending on the mode of operation, the system can provide interactive robot control and composition of robot programs. The framework is demonstrated by interactively controlling and programming a vacuum-cleaning robot. The demonstrations are used to exemplify the interactive programming and the plan recognition aspect of the research. The key contributions of this thesis are the introduction and implementation of the novel programming approach. It is expected to improve significantly the state-of-the-art in robot programming and interactive personal robotics.

Associated Center(s) / Consortia: National Robotics Engineering Center
Associated Lab(s) / Group(s): Advanced Mechatronics Lab
Number of pages: 101

Text Reference
Soshi Iba, "Interactive Multi-Modal Robot Programming," doctoral dissertation, tech. report CMU-RI-TR-04-50, Robotics Institute, Carnegie Mellon University, May, 2004

BibTeX Reference
   author = "Soshi Iba",
   title = "Interactive Multi-Modal Robot Programming",
   booktitle = "",
   school = "Robotics Institute, Carnegie Mellon University",
   month = "May",
   year = "2004",
   number= "CMU-RI-TR-04-50",
   address= "Pittsburgh, PA",