Deep Reinforcement Learning based Obstacle Avoidance for UAVs under Measurement Uncertainty

Abstract

Deep Reinforcement Learning (DRL) has emerged as a prominent method for enhancing the training of autonomous Unmanned Aerial Vehicles (UAVs). At the heart of this advancement lies the critical and complex challenge of obstacle avoidance, which is essential for ensuring that UAVs can navigate safely and efficiently. The incorporation of DRL into UAV training protocols enhances their ability to autonomously navigate through and adapt to diverse environments. This, in turn, is pivotal for expanding the operational capabilities of UAVs, enabling them to tackle a broader array of applications across various challenging and complex scenarios. In our research, we undertake a multifaceted examination of the impact of measurement uncertainty on the performance of DRL-based waypoint navigation and obstacle avoidance for UAVs covering both static and dynamic obstacles environment scenario. The challenge increases when the obstacles changes from static to dynamic as now these obstacles can change position in time and affect the navigation. The measurement uncertainty primarily stems from sensor noise, which impacts localization accuracy and obstacle detection capabilities. We model this uncertainty as adhering to a Gaussian probability distribution, characterized by an unknown mean and variance. Our research unfolds by assessing the performance of a DRL agent, meticulously trained using the Proximal Policy Optimization (PPO) algorithm, operating within an environment characterized by continuous state and action spaces. This investigation takes place in unseen randomized environments, each exposed to varying degrees of state-space noise, effectively emulating the effects of noisy sensor readings. Our primary objective is to pinpoint the threshold at which the policy becomes susceptible to noise, ultimately paving the way for us to explore diverse filtering techniques to mitigate these detrimental effects. Our findings reveal that the DRL agent exhibits a remarkable degree of inherent robustness against specific types of noise. We leverage this inherent robustness to bolster its performance in scenarios where it would otherwise succumb to the deleterious influence of state-space noise. To empirically validate our research, we undertake extensive training and testing of the DRL agent across a spectrum of UAV navigation scenarios within the PyBullet physics simulator. In a significant stride towards practical applicability, we port the policy distilled through simulation directly onto a real UAV without any additional modifications, and subsequently, we meticulously verify its performance in a real-world operational setting. This transformative approach holds the potential to inform the selection of sensors with varying biases and variances, and intriguingly, it suggests that artifcially injecting noise into measurements can yield performance enhancements in certain scenarios. The substantiation of this proposition is achieved through rigorous testing on a real UAV, thereby solidifying the practicality and real-world utility of our approach. In summation, our research contributes to the burgeoning field of UAV autonomy by shedding light on the intricate interplay between measurement uncertainty and Deep Reinforcement Learning-based navigation and obstacle avoidance. The insights garnered from our study not only advance our understanding of UAV operation but also provide actionable guidance for sensor selection and noise injection strategies to bolster real-world performance.