TLDR: This research presents a real-time obstacle avoidance system for mobile robots using three end-to-end Convolutional Neural Networks (CNNs) that fuse color and depth images from an Intel RealSense D415 camera. The NetConEmb model achieved the best offline performance and demonstrated 100% success in real-time navigation in both known and unknown environments. The lighter NetEmb model offered comparable offline results with 25% fewer parameters. An ablation study highlighted that while color data performed well offline, depth data was more robust for real-time navigation, underscoring the value of sensor fusion for practical robotic systems.
Mobile robots are becoming increasingly common in various applications, from industrial automation to personal assistance. A fundamental challenge for these robots is navigating complex and unpredictable environments while effectively avoiding obstacles. This research delves into an advanced approach to tackle this challenge: real-time obstacle avoidance using Convolutional Neural Networks (CNNs) combined with sensor fusion.
Traditionally, robot navigation systems, often termed “classical” approaches, involve multiple stages. They typically use sensors like cameras or LiDAR to build a map of the environment, which is then used by global and local planners to generate steering commands. While effective, this multi-stage process can be complex and require significant engineering effort.
In contrast, “end-to-end” navigation offers a more streamlined solution. This method directly translates raw sensory data into steering commands, bypassing the need for explicit mapping or localization. This study adopts a hybrid end-to-end approach, fusing both color and depth information from an Intel RealSense D415 RGB-D camera. This sensor integration is crucial for providing a comprehensive understanding of the robot’s surroundings, combining visual features (color) with spatial information (depth).
The researchers investigated three custom CNN architectures: NetConEmb, NetEmb, and NetGated. These networks were trained to learn from human demonstrations, predicting low-level steering commands (angular velocity) based on synchronized color and depth images. A key focus was on developing a lightweight yet effective solution. The NetEmb architecture, for instance, was designed to reduce the number of trainable parameters by approximately 25% compared to NetConEmb, leading to faster convergence while aiming for comparable performance.
Offline evaluations of these models on a test dataset revealed promising results. NetConEmb demonstrated the best overall performance, particularly achieving a notably low Median Absolute Error (MedAE) of 0.58×10-3 rad/s, indicating highly consistent predictions close to the ground truth. NetEmb, despite its reduced complexity, produced comparable results with an RMSE of 21.68×10-3 rad/s, very close to NetConEmb’s 21.42×10-3 rad/s.
The true test of these systems lies in their real-time performance on a physical robot. The models were deployed on a differential-drive mobile robot and tested in both known (corridor) and unknown (laboratory lane) environments. NetConEmb proved exceptionally robust, achieving a 100% success rate in obstacle avoidance in both settings. NetEmb and NetGated, while successful in the known environment, encountered difficulties in the unfamiliar lab setting.
An important aspect of the study involved an ablation analysis and sensor failure simulation. This revealed that while color images alone yielded lower error metrics in offline tests, depth sensing proved more robust and reliable for real-time navigation, especially under varying real-world conditions like light changes. This reinforces the critical importance of sensor fusion, where combining modalities provides a more resilient system than relying on a single sensor.
Also Read:
- Zero-Shot Depth Perception: How Drones Navigate with Just a Camera and IMU
- Teaching Robots to React: The Deep Reactive Policy Approach
In conclusion, this research successfully developed and validated a real-time obstacle avoidance system for mobile robots using end-to-end CNNs with RGB-D sensor fusion. The NetConEmb model stands out for its robustness and generalization capabilities, making it a strong candidate for vision-based autonomous navigation in diverse indoor environments. The full details of this work can be found in the research paper: Real-Time Obstacle Avoidance for a Mobile Robot Using CNN-Based Sensor Fusion.
