Beyond Sight: Fine-tuning Generalist Robot Policies with Heterogeneous Sensors via Language Grounding

Interacting with the world is a multisensory experience. For effective, general-purpose interaction, all available modalities—sight, touch, and hearing—must be leveraged to fill gaps in perception. For example, when vision is occluded while reaching into a bag, a robot should rely on its tactile and auditory senses. However, current generalist robot policies are typically trained with large datasets using predominantly visual and proprioceptive data. This limits their ability to utilize information from other sensory sources.

In a new study published on arXiv, researchers introduce FuSe, a novel approach for fine-tuning visuo-motor generalist robot policies. FuSe leverages natural language as a common ground across different modalities to integrate heterogeneous sensor data. This allows robots to interact even with sensors for which extensive datasets are not available.

FuSe combines a multimodal contrastive loss with a sensor-grounded language generation loss. The multimodal contrastive loss ensures that different modalities are mapped consistently with their corresponding semantic meanings in natural language. The language generation loss allows the robot to generate high-level semantic descriptions from the sensory observations. This enables a deeper understanding of the environment and the objects within it.

In the context of robot manipulation, FuSe enables the execution of complex tasks requiring joint reasoning over various modalities like vision, touch, and hearing. The researchers demonstrated FuSe's capabilities in a series of zero-shot scenarios, including multimodal prompting, compositional cross-modal prompting, and describing the objects the robot interacts with. For example, the robot could successfully execute instructions like "Pick up the red object that feels soft and makes a loud sound."

A notable aspect of FuSe is its applicability to various generalist robot strategies. The researchers showed that FuSe is compatible with both diffusion-based policies and large vision-language-action (VLA) models. This suggests that the approach could be a promising tool for developing robust and adaptable robot systems.

For the experiments, the researchers collected a dataset of 27,000 robot trajectories containing visual, tactile, acoustic, proprioceptive, and language information from three different real-world manipulation tasks. This dataset is one of the first of its kind to also include robot action data, which is crucial for performing physically grounded multimodal tasks.

The results of the experiments demonstrate that FuSe can boost success rates on robot manipulation tasks by over 20% compared to existing baselines. This highlights FuSe's potential to significantly improve the capabilities of robots in real-world scenarios.

The development of FuSe is a significant step towards a new generation of robots capable of integrating information from a variety of sensors, enabling them to tackle more complex tasks in challenging environments. Combining multimodal sensor data with natural language as a common denominator opens up new possibilities for human-robot interaction and paves the way for a future where robots are seamlessly integrated into our world.