Robotics: Science and Systems XIX

RT-1: Robotics Transformer for Real-World Control at Scale

Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alexander Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Kuang-Huei Lee, Sergey Levine, Yao Lu, Utsav Malla, Deeksha Manjunath, Igor Mordatch, Ofir Nachum, Carolina Parada, Jodilyn Peralta, Emily Perez, Karl Pertsch, Jornell Quiambao, Kanishka Rao, Michael S Ryoo, Grecia Salazar, Pannag R Sanketi, Kevin Sayed, Jaspiar Singh, Sumedh Sontakke, Austin Stone, Clayton Tan, Huong Tran, Vincent Vanhoucke, Steve Vega, Quan H Vuong, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, Brianna Zitkovich


By transferring knowledge from large, diverse, task-agnostic datasets, modern machine learning models can solve specific downstream tasks either zero-shot or with small task-specific datasets to a high level of performance. While this capability has been demonstrated in other fields such as computer vision, natural language processing or speech recognition, it remains to be shown in robotics, where the generalization capabilities of the models are particularly critical due to the difficulty of collecting real-world robotic data. We argue that one of the keys to the success of such general robotic models lies with open-ended task-agnostic training, combined with high-capacity architectures that can absorb all of the diverse, robotic data. In this paper, we present a model class, dubbed Robotics Transformer, that exhibits promising scalable model properties. We verify our conclusions in a study of different model classes and their ability to generalize as a function of the data size, model size, and data diversity based on a large-scale data collection on real robots performing real-world tasks.



    AUTHOR    = {Anthony Brohan AND Noah Brown AND Justice Carbajal AND Yevgen Chebotar AND Joseph Dabis AND Chelsea Finn AND Keerthana Gopalakrishnan AND Karol Hausman AND Alexander Herzog AND Jasmine Hsu AND Julian Ibarz AND Brian Ichter AND Alex Irpan AND Tomas Jackson AND Sally Jesmonth AND Nikhil Joshi AND Ryan Julian AND Dmitry Kalashnikov AND Yuheng Kuang AND Isabel Leal AND Kuang-Huei Lee AND Sergey Levine AND Yao Lu AND Utsav Malla AND Deeksha Manjunath AND Igor Mordatch AND Ofir Nachum AND Carolina Parada AND Jodilyn  Peralta AND Emily Perez AND Karl Pertsch AND Jornell  Quiambao AND Kanishka Rao AND Michael S Ryoo AND Grecia  Salazar AND Pannag R Sanketi AND Kevin  Sayed AND Jaspiar  Singh AND Sumedh  Sontakke AND Austin  Stone AND Clayton  Tan AND Huong  Tran AND Vincent Vanhoucke AND Steve  Vega AND Quan H Vuong AND Fei Xia AND Ted Xiao AND Peng Xu AND Sichun Xu AND Tianhe Yu AND Brianna  Zitkovich}, 
    TITLE     = {{RT-1: Robotics Transformer for Real-World Control at Scale}}, 
    BOOKTITLE = {Proceedings of Robotics: Science and Systems}, 
    YEAR      = {2023}, 
    ADDRESS   = {Daegu, Republic of Korea}, 
    MONTH     = {July}, 
    DOI       = {10.15607/RSS.2023.XIX.025}