Robotics: Science and Systems XX

Demonstrating Event-Triggered Investigation and Sample Collection for Human Scientists using Field Robots and Large Foundation Models

Tirthankar Bandyopadhyay, Fletcher Talbot, Callum Bennie, Hashini Senaratne, Xun Li, Brendan Tidd, Mingze Xi, Jan Stiefel, Volkan Dedeoglu, Rod Taylor, Tea Molnar, Ziwei Wang, Josh Pinskier, Feng Xu, Lois Liow, Ben Burgess-Limerick, Jesse Haviland, Pavan Sikka, Simon Murrell, Jane Hodgkinson, Jiajun Liu, Fred Pauling, Stanislav Funiak

Abstract:

In this paper, we introduce a pioneering end-to-end system demonstrated on a team of robots and sensors, designed to augment scientific exploration and discovery for human scientists in remote or inaccessible environments. We demonstrate and analyse our system's capability in a mock-up test-bed scenario. In this futuristic hypothetical scenario human scientists located in a controlled lunar habitat, are assisted by a team of robots in investigating an unknown seismic phenomena like moon-quakes or meteor impact detected by a sensor network deployed on the lunar surface. They do so by autonomously collecting data, providing contextual semantic information and collecting scientific sample for future analysis upon the direction of humans. This work is among the earliest to present a feasible way to integration large foundational models (LFMs) into field robotic deployment, enabling easy semantic and contextual understanding of the objects in the environment and natural language-based interactions with the robot for the scientist. In addition we bring together state-of-the-art techniques in mapping, object detection, navigation, mobile manipulation, soft grippers, event detection and present details of the integration, insights and lessons learnt from the deployment.

Download:

Bibtex:

  
@INPROCEEDINGS{Bandyopadhyay-RSS-24, 
    AUTHOR    = {Tirthankar Bandyopadhyay AND Fletcher Talbot AND Callum Bennie AND Hashini Senaratne AND Xun Li AND Brendan Tidd AND Mingze Xi AND Jan Stiefel AND Volkan Dedeoglu AND Rod Taylor AND Tea Molnar AND Ziwei Wang AND Josh Pinskier AND Feng Xu AND Lois Liow AND Ben Burgess-Limerick AND Jesse Haviland AND Pavan Sikka AND Simon Murrell AND Jane Hodgkinson AND Jiajun Liu AND Fred Pauling AND Stanislav Funiak}, 
    TITLE     = {{Demonstrating Event-Triggered Investigation and Sample Collection for Human Scientists using Field Robots and Large Foundation Models}}, 
    BOOKTITLE = {Proceedings of Robotics: Science and Systems}, 
    YEAR      = {2024}, 
    ADDRESS   = {Delft, Netherlands}, 
    MONTH     = {July}, 
    DOI       = {10.15607/RSS.2024.XX.017} 
}