Learning pixel-to-action control policies is challenging due to the high-dimensional and partially observable nature of the problem. This work combines deep learning with first-principle physics through differentiable simulation to enable autonomous navigation of multiple aerial robots through complex environments at high speed. Our approach optimizes a neural network control policy directly by backpropagating loss gradients through the robot simulation using a simple point-mass physics model and a depth rendering engine. Despite this simplicity, our method excels in challenging tasks for both multi-agent and single-agent applications with zero-shot sim-to-real transfer. In multi-agent scenarios, our system demonstrates self-organized behavior, enabling autonomous coordination without communication or centralized planning - an achievement not seen in existing traditional or learning-based methods. In real-world forest environments, it navigates at speeds up to 20 m/s. All these capabilities are deployed on a budget-friendly $21 computer, costing less than 5% of a GPU-equipped board used in existing systems.

How can we push a robot to its absolut limits in the physical world? We developed one of the world fastest autonomous drone and pushed this machine to its maximum performance in the real world, achieving a peak acceleration greater than 12g and a peak velocity of 108 km/h. The key to our success is a neural network policy trained with reinforcement learning. Our neural network policy achieved superhuman control performance within minutes of training on a standard workstation. Additinally, our study indicates that the fundamental advantage of reinforcement learning over optimal control is not that it optimizes its objective better but that it optimizes a better objective. RL can directly optimize a task-level objective and can leverage domain randomization to cope with model uncertainty, allowing the discovery of more robust control responses.

This project aims to design an optimal controller for agile drone flight. Model Predictive Control (MPC) provides near optimal performance by leveraging models and numerical optimization. However, a key challenge lies in defining an effective loss function with well-tuned hyperparameters, which is task-specific and difficult to search. To address this, we introduce a policy-search-for-model-predictive-control framework that employs policy search to automatically search high-level decision variables for MPC. Specifically, we formulate MPC as a parameterized controller, where traditionally hard-to-optimize decision variables are represented as high-level policies and learned through policy search. This approach enables MPC to adapt to changing environments.