Robotics: Science and Systems XIX
CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memory
Nur Muhammad (Mahi)Shafiullah, Chris Paxton, Lerrel Pinto, Soumith Chintala, Arthur SzlamAbstract:
We propose CLIP-Fields, an implicit scene model that can be used for a variety of tasks, such as segmentation, instance identification, semantic search over space, and view localization. CLIP-Fields learns a mapping from spatial locations to semantic embedding vectors. Importantly, we show that this mapping can be trained with supervision coming only from web-image and web-text trained models such as CLIP, Detic, and Sentence-BERT; and thus uses no direct human supervision. When compared to baselines like Mask-RCNN, our method outperforms on few-shot instance identification or semantic segmentation on the HM3D dataset with only a fraction of the examples. Finally, we show that using CLIP-Fields as a scene memory, robots can perform semantic navigation in real-world environments. Our code and demonstration videos are available here: https://clip-fields.github.io
Bibtex:
@INPROCEEDINGS{Shafiullah-RSS-23, AUTHOR = {Nur Muhammad (Mahi)Shafiullah AND Chris Paxton AND Lerrel Pinto AND Soumith Chintala AND Arthur Szlam}, TITLE = {{CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memory}}, BOOKTITLE = {Proceedings of Robotics: Science and Systems}, YEAR = {2023}, ADDRESS = {Daegu, Republic of Korea}, MONTH = {July}, DOI = {10.15607/RSS.2023.XIX.074} }