Robotics: Science and Systems XVII

RMP2: A Structured Composable Policy Class for Robot Learning

Anqi Li*, Ching-An Cheng*, Muhammad Asif Rana, Man Xie, Karl Van Wyk, Nathan Ratliff, Byron Boots
* These authors contributed equally


We consider the problem of learning motion policies for acceleration-based robotics systems with a structured policy class. We leverage a multi-task control framework called RMPflow which has been successfully applied in many robotics problems. Using RMPflow as a structured policy class in learning has several benefits; such as sufficient expressiveness; the flexibility to inject different levels of prior knowledge as well as the ability to transfer policies between robots. However; implementing a system for end-to-end learning of RMPflow policies faces several computational challenges. In this work; we re-examine the RMPflow algorithm and propose a more practical alternative; called RMP2; that uses modern automatic differentiation tools (such as TensorFlow and PyTorch) to compute RMPflow policies. Our new design retains the strengths of RMPflow while bringing in advantages from automatic differentiation; including 1) simple programming interfaces to designing complex transformations; 2) support of general directed acyclic graph (DAG) transformation structures; 3) end-to-end differentiability for policy learning; 4) improved computational efficiency. Because of these features; RMP2 can be treated as a structured policy class for efficient robot learning that is suitable for encoding domain knowledge. Our experiments show that using the structured policy class given by RMP2 can improve policy performance and safety in reinforcement learning tasks for goal reaching in cluttered space. The video for our experimental results can be found at and the code is available at



    AUTHOR    = {Anqi Li AND Ching-An Cheng AND Muhammad Asif Rana AND Man Xie AND Karl {Van Wyk} AND Nathan Ratliff AND Byron Boots}, 
    TITLE     = {{RMP2: A Structured Composable Policy Class for Robot Learning}}, 
    BOOKTITLE = {Proceedings of Robotics: Science and Systems}, 
    YEAR      = {2021}, 
    ADDRESS   = {Virtual}, 
    MONTH     = {July}, 
    DOI       = {10.15607/RSS.2021.XVII.092}