A team of roboticists from New York University, in collaboration with an AI expert from Meta, has unveiled a groundbreaking robot capable of identifying specific objects in an unfamiliar space and relocating them to designated positions. Published on the arXiv preprint server, the research details the robot’s programming and its successful performance across various real-world settings.
The researchers highlighted the significant advancements in visual language models (VLMs) and robot skills, emphasizing their proficiency in recognizing objects based on language prompts and executing tasks such as grasping, carrying, and placing items without damage. However, the integration of VLMs with skilled robots has been a relatively unexplored area.
In this study, the team sought to bridge this gap by utilizing a robot provided by Hello Robot, equipped with wheels, a pole, and retractable arms featuring claspers for hands. Dubbed OK-Robot, the researchers paired it with a pre-trained VLM and conducted tests in 10 volunteer homes. Using 3D videos captured with an iPhone, the robot gained a comprehensive understanding of each home’s layout. The team then tasked it with simple instructions, such as moving an item from one location to another.
Out of 170 tasks assigned, the robot successfully completed 58%, with the potential to improve the success rate to 82% by decluttering the workspace. Notably, the system employs a zero-shot algorithm, indicating that the robot was not specifically trained for the environment in which it operated. The researchers assert that the achieved success rate underscores the viability of VLM-based robot systems.
While acknowledging the possibility of enhancing the success rate through further refinement and a more sophisticated robot, the research team envisions their work as a pioneering step toward the development of advanced VLM-based robots. This innovative technology holds promise for applications in diverse environments, showcasing the potential for future breakthroughs in robotic capabilities.
