Header logo is

18 results (View BibTeX file of all listed publications)

2009

no image
Path integral-based stochastic optimal control for rigid body dynamics

Theodorou, E. A., Buchli, J., Schaal, S.

In Adaptive Dynamic Programming and Reinforcement Learning, 2009. ADPRL ’09. IEEE Symposium on, pages: 219-225, 2009, clmc (inproceedings)

Abstract
Recent advances on path integral stochastic optimal control [1],[2] provide new insights in the optimal control of nonlinear stochastic systems which are linear in the controls, with state independent and time invariant control transition matrix. Under these assumptions, the Hamilton-Jacobi-Bellman (HJB) equation is formulated and linearized with the use of the logarithmic transformation of the optimal value function. The resulting HJB is a linear second order partial differential equation which is solved by an approximation based on the Feynman-Kac formula [3]. In this work we review the theory of path integral control and derive the linearized HJB equation for systems with state dependent control transition matrix. In addition we derive the path integral formulation for the general class of systems with state dimensionality that is higher than the dimensionality of the controls. Furthermore, by means of a modified inverse dynamics controller, we apply path integral stochastic optimal control over the new control space. Simulations illustrate the theoretical results. Future developments and extensions are discussed.

am

link (url) [BibTex]

2009

am Theodorou, E. A., Buchli, J., Schaal, S. Path integral-based stochastic optimal control for rigid body dynamics In Adaptive Dynamic Programming and Reinforcement Learning, 2009. ADPRL ’09. IEEE Symposium on, pages: 219-225, 2009, clmc (inproceedings)

link (url) [BibTex]

no image
Learning locomotion over rough terrain using terrain templates

Kalakrishnan, M., Buchli, J., Pastor, P., Schaal, S.

In Intelligent Robots and Systems, 2009. IROS 2009. IEEE/RSJ International Conference on, pages: 167-172, 2009, clmc (inproceedings)

Abstract
We address the problem of foothold selection in robotic legged locomotion over very rough terrain. The difficulty of the problem we address here is comparable to that of human rock-climbing, where foot/hand-hold selection is one of the most critical aspects. Previous work in this domain typically involves defining a reward function over footholds as a weighted linear combination of terrain features. However, a significant amount of effort needs to be spent in designing these features in order to model more complex decision functions, and hand-tuning their weights is not a trivial task. We propose the use of terrain templates, which are discretized height maps of the terrain under a foothold on different length scales, as an alternative to manually designed features. We describe an algorithm that can simultaneously learn a small set of templates and a foothold ranking function using these templates, from expert-demonstrated footholds. Using the LittleDog quadruped robot, we experimentally show that the use of terrain templates can produce complex ranking functions with higher performance than standard terrain features, and improved generalization to unseen terrain.

am

link (url) Project Page [BibTex]

am Kalakrishnan, M., Buchli, J., Pastor, P., Schaal, S. Learning locomotion over rough terrain using terrain templates In Intelligent Robots and Systems, 2009. IROS 2009. IEEE/RSJ International Conference on, pages: 167-172, 2009, clmc (inproceedings)

link (url) Project Page [BibTex]

no image
Modelling the interplay of central pattern generation and sensory feedback in the neuromuscular control of running

Daley, M., Righetti, L., Ijspeert, A.

In Comparative Biochemistry and Physiology - Part A: Molecular & Integrative Physiology. Annual Main Meeting for the Society for Experimental Biology, 153, Glasgow, Scotland, 2009 (inproceedings)

mg

link (url) DOI [BibTex]

mg Daley, M., Righetti, L., Ijspeert, A. Modelling the interplay of central pattern generation and sensory feedback in the neuromuscular control of running In Comparative Biochemistry and Physiology - Part A: Molecular & Integrative Physiology. Annual Main Meeting for the Society for Experimental Biology, 153, Glasgow, Scotland, 2009 (inproceedings)

link (url) DOI [BibTex]

no image
Compact models of motor primitive variations for predictible reaching and obstacle avoidance

Stulp, F., Oztop, E., Pastor, P., Beetz, M., Schaal, S.

In IEEE-RAS International Conference on Humanoid Robots (Humanoids 2009), Paris, Dec.7-10, 2009, clmc (inproceedings)

Abstract
over and over again. This regularity allows humans and robots to reuse existing solutions for known recurring tasks. We expect that reusing a set of standard solutions to solve similar tasks will facilitate the design and on-line adaptation of the control systems of robots operating in human environments. In this paper, we derive a set of standard solutions for reaching behavior from human motion data. We also derive stereotypical reaching trajectories for variations of the task, in which obstacles are present. These stereotypical trajectories are then compactly represented with Dynamic Movement Primitives. On the humanoid robot Sarcos CB, this approach leads to reproducible, predictable, and human-like reaching motions.

am

link (url) [BibTex]

am Stulp, F., Oztop, E., Pastor, P., Beetz, M., Schaal, S. Compact models of motor primitive variations for predictible reaching and obstacle avoidance In IEEE-RAS International Conference on Humanoid Robots (Humanoids 2009), Paris, Dec.7-10, 2009, clmc (inproceedings)

link (url) [BibTex]

no image
Human optimization strategies under reward feedback

Hoffmann, H., Theodorou, E., Schaal, S.

In Abstracts of Neural Control of Movement Conference (NCM 2009), Waikoloa, Hawaii, 2009, 2009, clmc (inproceedings)

Abstract
Many hypothesis on human movement generation have been cast into an optimization framework, implying that movements are adapted to optimize a single quantity, like, e.g., jerk, end-point variance, or control cost. However, we still do not understand how humans actually learn when given only a cost or reward feedback at the end of a movement. Such a reinforcement learning setting has been extensively explored theoretically in engineering and computer science, but in human movement control, hardly any experiment studied movement learning under reward feedback. We present experiments probing which computational strategies humans use to optimize a movement under a continuous reward function. We present two experimental paradigms. The first paradigm mimics a ball-hitting task. Subjects (n=12) sat in front of a computer screen and moved a stylus on a tablet towards an unknown target. This target was located on a line that the subjects had to cross. During the movement, visual feedback was suppressed. After the movement, a reward was displayed graphically as a colored bar. As reward, we used a Gaussian function of the distance between the target location and the point of line crossing. We chose such a function since in sensorimotor tasks, the cost or loss function that humans seem to represent is close to an inverted Gaussian function (Koerding and Wolpert 2004). The second paradigm mimics pocket billiards. On the same experimental setup as above, the computer screen displayed a pocket (two bars), a white disk, and a green disk. The goal was to hit with the white disk the green disk (as in a billiard collision), such that the green disk moved into the pocket. Subjects (n=8) manipulated with the stylus the white disk to effectively choose start point and movement direction. Reward feedback was implicitly given as hitting or missing the pocket with the green disk. In both paradigms, subjects increased the average reward over trials. The surprising result was that in these experiments, humans seem to prefer a strategy that uses a reward-weighted average over previous movements instead of gradient ascent. The literature on reinforcement learning is dominated by gradient-ascent methods. However, our computer simulations and theoretical analysis revealed that reward-weighted averaging is the more robust choice given the amount of movement variance observed in humans. Apparently, humans choose an optimization strategy that is suitable for their own movement variance.

am

[BibTex]

am Hoffmann, H., Theodorou, E., Schaal, S. Human optimization strategies under reward feedback In Abstracts of Neural Control of Movement Conference (NCM 2009), Waikoloa, Hawaii, 2009, 2009, clmc (inproceedings)

[BibTex]

no image
Learning and generalization of motor skills by learning from demonstration

Pastor, P., Hoffmann, H., Asfour, T., Schaal, S.

In International Conference on Robotics and Automation (ICRA2009), Kobe, Japan, May 12-19, 2009, 2009, clmc (inproceedings)

Abstract
We provide a general approach for learning robotic motor skills from human demonstration. To represent an observed movement, a non-linear differential equation is learned such that it reproduces this movement. Based on this representation, we build a library of movements by labeling each recorded movement according to task and context (e.g., grasping, placing, and releasing). Our differential equation is formulated such that generalization can be achieved simply by adapting a start and a goal parameter in the equation to the desired position values of a movement. For object manipulation, we present how our framework extends to the control of gripper orientation and finger position. The feasibility of our approach is demonstrated in simulation as well as on a real robot. The robot learned a pick-and-place operation and a water-serving task and could generalize these tasks to novel situations.

am

link (url) [BibTex]

am Pastor, P., Hoffmann, H., Asfour, T., Schaal, S. Learning and generalization of motor skills by learning from demonstration In International Conference on Robotics and Automation (ICRA2009), Kobe, Japan, May 12-19, 2009, 2009, clmc (inproceedings)

link (url) [BibTex]

no image
Compliant quadruped locomotion over rough terrain

Buchli, J., Kalakrishnan, M., Mistry, M., Pastor, P., Schaal, S.

In Intelligent Robots and Systems, 2009. IROS 2009. IEEE/RSJ International Conference on, pages: 814-820, 2009, clmc (inproceedings)

Abstract
Many critical elements for statically stable walking for legged robots have been known for a long time, including stability criteria based on support polygons, good foothold selection, recovery strategies to name a few. All these criteria have to be accounted for in the planning as well as the control phase. Most legged robots usually employ high gain position control, which means that it is crucially important that the planned reference trajectories are a good match for the actual terrain, and that tracking is accurate. Such an approach leads to conservative controllers, i.e. relatively low speed, ground speed matching, etc. Not surprisingly such controllers are not very robust - they are not suited for the real world use outside of the laboratory where the knowledge of the world is limited and error prone. Thus, to achieve robust robotic locomotion in the archetypical domain of legged systems, namely complex rough terrain, where the size of the obstacles are in the order of leg length, additional elements are required. A possible solution to improve the robustness of legged locomotion is to maximize the compliance of the controller. While compliance is trivially achieved by reduced feedback gains, for terrain requiring precise foot placement (e.g. climbing rocks, walking over pegs or cracks) compliance cannot be introduced at the cost of inferior tracking. Thus, model-based control and - in contrast to passive dynamic walkers - active balance control is required. To achieve these objectives, in this paper we add two crucial elements to legged locomotion, i.e., floating-base inverse dynamics control and predictive force control, and we show that these elements increase robustness in face of unknown and unanticipated perturbations (e.g. obstacles). Furthermore, we introduce a novel line-based COG trajectory planner, which yields a simpler algorithm than traditional polygon based methods and creates the appropriate input to our control system.We show results from bot- h simulation and real world of a robotic dog walking over non-perceived obstacles and rocky terrain. The results prove the effectivity of the inverse dynamics/force controller. The presented results show that we have all elements needed for robust all-terrain locomotion, which should also generalize to other legged systems, e.g., humanoid robots.

am

link (url) [BibTex]

am Buchli, J., Kalakrishnan, M., Mistry, M., Pastor, P., Schaal, S. Compliant quadruped locomotion over rough terrain In Intelligent Robots and Systems, 2009. IROS 2009. IEEE/RSJ International Conference on, pages: 814-820, 2009, clmc (inproceedings)

link (url) [BibTex]

no image
Inertial parameter estimation of floating-base humanoid systems using partial force sensing

Mistry, M., Schaal, S., Yamane, K.

In IEEE-RAS International Conference on Humanoid Robots (Humanoids 2009), Paris, Dec.7-10, 2009, clmc (inproceedings)

Abstract
Recently, several controllers have been proposed for humanoid robots which rely on full-body dynamic models. The estimation of inertial parameters from data is a critical component for obtaining accurate models for control. However, floating base systems, such as humanoid robots, incur added challenges to this task (e.g. contact forces must be measured, contact states can change, etc.) In this work, we outline a theoretical framework for whole body inertial parameter estimation, including the unactuated floating base. Using a least squares minimization approach, conducted within the nullspace of unmeasured degrees of freedom, we are able to use a partial force sensor set for full-body estimation, e.g. using only joint torque sensors, allowing for estimation when contact force measurement is unavailable or unreliable (e.g. due to slipping, rolling contacts, etc.). We also propose how to determine the theoretical minimum force sensor set for full body estimation, and discuss the practical limitations of doing so.

am

link (url) [BibTex]

am Mistry, M., Schaal, S., Yamane, K. Inertial parameter estimation of floating-base humanoid systems using partial force sensing In IEEE-RAS International Conference on Humanoid Robots (Humanoids 2009), Paris, Dec.7-10, 2009, clmc (inproceedings)

link (url) [BibTex]

2006

no image
Learning operational space control

Peters, J., Schaal, S.

In Robotics: Science and Systems II (RSS 2006), pages: 255-262, (Editors: Gaurav S. Sukhatme and Stefan Schaal and Wolfram Burgard and Dieter Fox), Cambridge, MA: MIT Press, RSS , 2006, clmc (inproceedings)

Abstract
While operational space control is of essential importance for robotics and well-understood from an analytical point of view, it can be prohibitively hard to achieve accurate control in face of modeling errors, which are inevitable in complex robots, e.g., humanoid robots. In such cases, learning control methods can offer an interesting alternative to analytical control algorithms. However, the resulting learning problem is ill-defined as it requires to learn an inverse mapping of a usually redundant system, which is well known to suffer from the property of non-covexity of the solution space, i.e., the learning system could generate motor commands that try to steer the robot into physically impossible configurations. A first important insight for this paper is that, nevertheless, a physically correct solution to the inverse problem does exits when learning of the inverse map is performed in a suitable piecewise linear way. The second crucial component for our work is based on a recent insight that many operational space controllers can be understood in terms of a constraint optimal control problem. The cost function associated with this optimal control problem allows us to formulate a learning algorithm that automatically synthesizes a globally consistent desired resolution of redundancy while learning the operational space controller. From the view of machine learning, the learning problem corresponds to a reinforcement learning problem that maximizes an immediate reward and that employs an expectation-maximization policy search algorithm. Evaluations on a three degrees of freedom robot arm illustrate the feasability of our suggested approach.

am ei

link (url) [BibTex]

2006

am ei Peters, J., Schaal, S. Learning operational space control In Robotics: Science and Systems II (RSS 2006), pages: 255-262, (Editors: Gaurav S. Sukhatme and Stefan Schaal and Wolfram Burgard and Dieter Fox), Cambridge, MA: MIT Press, RSS , 2006, clmc (inproceedings)

link (url) [BibTex]

no image
Reinforcement Learning for Parameterized Motor Primitives

Peters, J., Schaal, S.

In Proceedings of the 2006 International Joint Conference on Neural Networks, pages: 73-80, IJCNN, 2006, clmc (inproceedings)

Abstract
One of the major challenges in both action generation for robotics and in the understanding of human motor control is to learn the "building blocks of movement generation", called motor primitives. Motor primitives, as used in this paper, are parameterized control policies such as splines or nonlinear differential equations with desired attractor properties. While a lot of progress has been made in teaching parameterized motor primitives using supervised or imitation learning, the self-improvement by interaction of the system with the environment remains a challenging problem. In this paper, we evaluate different reinforcement learning approaches for improving the performance of parameterized motor primitives. For pursuing this goal, we highlight the difficulties with current reinforcement learning methods, and outline both established and novel algorithms for the gradient-based improvement of parameterized policies. We compare these algorithms in the context of motor primitive learning, and show that our most modern algorithm, the Episodic Natural Actor-Critic outperforms previous algorithms by at least an order of magnitude. We demonstrate the efficiency of this reinforcement learning method in the application of learning to hit a baseball with an anthropomorphic robot arm.

am ei

link (url) DOI [BibTex]

am ei Peters, J., Schaal, S. Reinforcement Learning for Parameterized Motor Primitives In Proceedings of the 2006 International Joint Conference on Neural Networks, pages: 73-80, IJCNN, 2006, clmc (inproceedings)

link (url) DOI [BibTex]

no image
Ab-initio calculations: I. Basic principles of the density functional electron theory and combination with phenomenological theories

Fähnle, M.

In Structural defects in ordered alloys and intermetallics. Characterization and modelling, pages: IX-1-IX-10, COST and CNRS, Bonascre [Ariege, France], 2006 (inproceedings)

mms

[BibTex]

mms Fähnle, M. Ab-initio calculations: I. Basic principles of the density functional electron theory and combination with phenomenological theories In Structural defects in ordered alloys and intermetallics. Characterization and modelling, pages: IX-1-IX-10, COST and CNRS, Bonascre [Ariege, France], 2006 (inproceedings)

[BibTex]

no image
Hard magnetic FePt thin films and nanostructures in L1(0) phases

Goll, D., Breitling, A., Goo, N. H., Sigle, W., Hirscher, M., Schütz, G.

In 13, pages: 97-101, Beijing, PR China, 2006 (inproceedings)

mms

[BibTex]

mms Goll, D., Breitling, A., Goo, N. H., Sigle, W., Hirscher, M., Schütz, G. Hard magnetic FePt thin films and nanostructures in L1(0) phases In 13, pages: 97-101, Beijing, PR China, 2006 (inproceedings)

[BibTex]

no image
Ab-initio calculations: II. Application to atomic defects, phase diagrams, dislocations

Fähnle, M.

In Structural defects in ordered alloys and intermetallics. Characterization and modelling, pages: XIV-1-XIV-11, COST and CNRS, Bonascre [Ariege, France], 2006 (inproceedings)

mms

[BibTex]

mms Fähnle, M. Ab-initio calculations: II. Application to atomic defects, phase diagrams, dislocations In Structural defects in ordered alloys and intermetallics. Characterization and modelling, pages: XIV-1-XIV-11, COST and CNRS, Bonascre [Ariege, France], 2006 (inproceedings)

[BibTex]

no image
Movement generation using dynamical systems : a humanoid robot performing a drumming task

Degallier, S., Santos, C. P., Righetti, L., Ijspeert, A.

In 2006 6th IEEE-RAS International Conference on Humanoid Robots, pages: 512-517, IEEE, Genova, Italy, 2006 (inproceedings)

Abstract
The online generation of trajectories in humanoid robots remains a difficult problem. In this contribution, we present a system that allows the superposition, and the switch between, discrete and rhythmic movements. Our approach uses nonlinear dynamical systems for generating trajectories online and in real time. Our goal is to make use of attractor properties of dynamical systems in order to provide robustness against small perturbations and to enable online modulation of the trajectories. The system is demonstrated on a humanoid robot performing a drumming task.

mg

link (url) DOI [BibTex]

mg Degallier, S., Santos, C. P., Righetti, L., Ijspeert, A. Movement generation using dynamical systems : a humanoid robot performing a drumming task In 2006 6th IEEE-RAS International Conference on Humanoid Robots, pages: 512-517, IEEE, Genova, Italy, 2006 (inproceedings)

link (url) DOI [BibTex]

no image
Residual stress analysis in reed pipe brass tongues of historic organs

Manescu, A., Giuliani, A., Fiori, F., Baretzky, B.

In Residual Stresses VII. 7th Europen Conference on Residual Stresses (ECRS7), pages: 969-974, Trans Tech, Berlin [Germany], 2006 (inproceedings)

mms

[BibTex]

mms Manescu, A., Giuliani, A., Fiori, F., Baretzky, B. Residual stress analysis in reed pipe brass tongues of historic organs In Residual Stresses VII. 7th Europen Conference on Residual Stresses (ECRS7), pages: 969-974, Trans Tech, Berlin [Germany], 2006 (inproceedings)

[BibTex]

no image
Design methodologies for central pattern generators: an application to crawling humanoids

Righetti, L., Ijspeert, A.

In Proceedings of Robotics: Science and Systems, Philadelphia, USA, August 2006 (inproceedings)

mg

link (url) DOI [BibTex]

mg Righetti, L., Ijspeert, A. Design methodologies for central pattern generators: an application to crawling humanoids In Proceedings of Robotics: Science and Systems, Philadelphia, USA, August 2006 (inproceedings)

link (url) DOI [BibTex]

no image
Programmable central pattern generators: an application to biped locomotion control

Righetti, L., Ijspeert, A.

In Proceedings of the IEEE International Conference on Robotics and Automation, 2006. ICRA 2006., pages: 1585-1590, IEEE, 2006 (inproceedings)

mg

[BibTex]

mg Righetti, L., Ijspeert, A. Programmable central pattern generators: an application to biped locomotion control In Proceedings of the IEEE International Conference on Robotics and Automation, 2006. ICRA 2006., pages: 1585-1590, IEEE, 2006 (inproceedings)

[BibTex]

no image
High-pressure influence on the kinetics of grain boundary segregation in the Cu-Bi system

Chang, L.-S., Straumal, B., Rabkin, E., Lojkowski, W., Gust, W.

In 258-260, pages: 390-396, Aveiro (Portugal), 2006 (inproceedings)

mms

[BibTex]

mms Chang, L.-S., Straumal, B., Rabkin, E., Lojkowski, W., Gust, W. High-pressure influence on the kinetics of grain boundary segregation in the Cu-Bi system In 258-260, pages: 390-396, Aveiro (Portugal), 2006 (inproceedings)

[BibTex]