Get access to AUTOASSESS public results
Scientific Publications
Abstract:
This study primarily aims to bridge the gap between simulated and real-life experiments of the Omnimorph platform. Through rigorous testing and analysis, it contributes to the broader goal of achieving stable flight for the Omnimorph—a milestone not yet realized. This thesis represents a significant step forward in enabling Omnimorph to operate effectively in real- world conditions for the first time.
Abstract:
This paper introduces for the first time the design, modelling, and control of a novel morphing multi-rotor Unmanned Aerial Vehicle (UAV) that we call the OmniMorph. The morphing ability allows the selection of the configuration that optimizes energy consumption while ensuring the needed maneuverability for the required task. The most energy-efficient uni-directional thrust (UDT) configuration can be used, e.g., during standard point-to-point displacements. Fully-actuated (FA) and omnidirectional (OD) configurations can be instead used for full pose tracking, such as, e.g., constant attitude horizontal motions and full rotations on the spot, and for full wrench 6D interaction control and 6D disturbance rejection. Morphing is obtained using a single servomotor, allowing possible minimization of weight, costs, and maintenance complexity. The actuation properties are studied, and an optimal controller that compromises between performance and control effort is proposed and validated in realistic simulations. Preliminary tests on the prototype are presented to assess the propellers’ mutual aerodynamic interference.
Abstract:
Quadrotors are among the most agile flying robots. Despite recent advances in learning-based control and computer vision, autonomous drones still rely on explicit state estimation. On the other hand, human pilots only rely on a first-personview video stream from the drone onboard camera to push the platform to its limits and fly robustly in unseen environments. To the best of our knowledge, we present the first visionbased quadrotor system that autonomously navigates through a sequence of gates at high speeds while directly mapping pixels to control commands. Like professional drone-racing pilots, our system does not use explicit state estimation and leverages the same control commands humans use (collective thrust and body rates). We demonstrate agile flight at speeds up to 40 km/h with accelerations up to 2 g. This is achieved by training vision-based policies with reinforcement learning (RL). The training is facilitated using an asymmetric actor-critic with access to privileged information. To overcome the computational complexity during image-based RL training, we use the inner edges of the gates as a sensor abstraction. This simple yet robust, task-relevant representation can be simulated during training without rendering images. During deployment, a Swintransformer-based gate detector is used. Our approach enables autonomous agile flight with standard, off-the-shelf hardware. Although our demonstration focuses on drone racing, we believe that our method has an impact beyond drone racing and can serve as a foundation for future research into real-world applications in structured environments.
Abstract:
In this paper, we propose a model for the thrust coefficient of propellers that can take into account cross-influence between adjacent propellers. The aerodynamic interaction between propellers in multirotor aerial vehicles reduces the thrust they can produce. The influence between propellers depends on their relative positioning and orientation, which are taken into account by the proposed model. It is validated on measurements collected by a force sensor mounted on a propeller for different configurations of the adjacent propellers in a support structure. In this work, we focus on configurations with small relative orientations. Results show that the proposed model outperforms the traditional constant model in terms of thrust prediction on the data we collected, and it performs better than other models with fewer parameters, being the only one with less than 10% maximum percentage error.
Abstract:
In this work, we provide an experimental vali-dation of the recent concepts of closed-loop state and input sensitivity in the context of robust flight control for a quadrotor (UAV) equipped with the popular PX4 1 1 https://px4.io/ controller. Our objective is to experimentally assess how the optimization of the reference trajectory w.r.t. these sensitivity metrics can improve the closed-loop system performance against model uncertainties commonly affecting the quadrotor systems. To accomplish this, we present a series of experiments designed to validate our optimization approach on two distinct trajectories, with the primary aim of assessing its precision in guiding the quadrotor through the center of a window at relatively high speeds. This approach provides some interesting insights for increasing the closed-loop robustness of the robot state and inputs against physical parametric uncertainties that may degrade the system’s performance.
Abstract:
Traditional volumetric fusion algorithms preserve the spatial structure of 3D scenes, which is beneficial for many tasks in computer vision and robotics. However, they often lack realism in terms of visualization. Emerging 3D Gaussian splatting bridges this gap, but existing Gaussian-based reconstruction methods often suffer from artifacts and inconsistencies with the underlying 3D structure, and struggle with real-time optimization, unable to provide users with immediate feedback in high quality. One of the bottlenecks arises from the massive amount of Gaussian parameters that need to be updated during optimization. Instead of using 3D Gaussian as a standalone map representation, we incorporate it into a volumetric mapping system to take advantage of geometric information and propose to use a quadtree data structure on images to drastically reduce the number of splats initialized. In this way, we simultaneously generate a compact 3D Gaussian map with fewer artifacts and a volumetric map on the fly. Our method, GSFusion, significantly enhances computational efficiency without sacrificing rendering quality, as demonstrated on both synthetic and real datasets. Code will be available at https://github.com/goldoak/GSFusion
Abstract:
This work presents techniques for scheduling the position controller gains for a class of fully-actuated morphing multi-rotor UAVs that use synchronized tilting to change their actuation capabilities. The feasible set of forces and torques that can be produced by the platform changes with the tilting angle, thus the tracking and disturbance rejection capabilities also change. To exploit the platform limits, two methods are proposed for gain scheduling using a simplified example, then one method is tested in simulation with an omnidirectional morphing multi-rotor (OmniMorph). The simulation results show that the developed techniques achieve consistent position tracking performance along the range of tilting angles when rejecting step disturbance forces of values close to the maximum force capabilities. The proposed methods offer a trade-off between simplicity and accuracy, that could be potentially applied for any multi-rotor with synchronized tilting capabilities. A video summary can be found in: https://youtu.be/kH-rrO8gWeU
Abstract:
This work answers positively the question whether non-stop flights are possible for maintaining constant the pose of cable-suspended objects. Such a counterintuitive answer paves the way for a paradigm shift where energetically efficient fixed-wing flying carriers can replace the inefficient multirotor carriers that have been used so far in precise cooperative cable-suspended aerial manipulation. First, we show that one or two flying carriers alone cannot perform non-stop flights while maintaining a constant pose of the suspended object. Instead, we prove that three flying carriers can achieve this task provided that the orientation of the load at the equilibrium is such that the components of the cable forces that balance the external force (typically gravity) do not belong to the plane of the cable anchoring points on the load. Numerical tests are presented in support of the analytical results.
(2024): Towards Instance Segmentation-Based Litter Collection with Multi-Rotor Aerial Vehicle
Abstract:
This paper presents a novel aerial robotics application of instance segmentation-based floating litter collection with a multi-rotor aerial vehicle (MRAV). In the scope of the paper, we present a review of the available datasets for litter detection and segmentation. The reviewed datasets are used to train a Mask-RCNN neural network for instance segmentation. The neural network is off-board deployed on an edge computing device and used for litter position estimation. Based on the estimated litter position, we plan a path based on a quadratic Bezier curve for the litter pickup. We compare different trajectory generation methods for the object pickup. The system is verified in a laboratory environment. Eventually, we present practical considerations and improvements necessary to enable autonomous litter collection with MRAV.
Abstract:
Quadrotor flight is an extremely challenging problem due to the limited control authority encountered at the limit of handling. Model Predictive Contouring Control (MPCC) has emerged as a promising model-based approach for time optimization problems such as drone racing. However, the standard MPCC formulation used in quadrotor racing introduces the notion of the gates directly in the cost function, creating a multi-objective optimization that continuously trades off between maximizing progress and tracking the path accurately. This paper introduces three key components that enhance the state-of-the-art MPCC approach for drone racing. First and foremost, we provide safety guarantees in the form of a track constraint and terminal set. The track constraint is designed as a spatial constraint which prevents gate collisions while allowing for time optimization only in the cost function. Second, we augment the existing first principles dynamics with a residual term that captures complex aerodynamic effects and thrust forces learned directly from real-world data. Third, we use Trust Region Bayesian Optimization (TuRBO), a state-of-the-art global Bayesian Optimization algorithm, to tune the hyperparameters of the MPCC controller given a sparse reward based on lap time minimization. The proposed approach achieves similar lap times to the best-performing RL policy and outperforms the best model-based controller while satisfying constraints. In both simulation and real world, our approach consistently prevents gate crashes with 100% success rate, while pushing the quadrotor to its physical limits reaching speeds of more than 80km/h.
Abstract:
When considering close interaction between a human & an aerial robot, one aspect that is unique to this scenario with respect to other ground-based robotic platforms is the presence of wind disturbances that act on the aerial robot. In order to realize that aerial robots can coexist & collaborate with humans, we must be able to guarantee the safety of humans even in the presence of wind disturbances. To ensure the safety of human collaborators in such scenarios, this thesis aims to establish a comprehensive safety metric that guides the robot’s motion during such interactions. The aerial-robot considered for defining a safety metric is a hexarotor with a tilted-propeller configuration (tilt-hex). In the presence of a known wind disturbance, an aerodynamic model estimates the addition wrench this disturbance provides. The deviation due to this disturbance is then used to construct a safety metric that ensures the aerial-robot’s motion is guided and it operates safely within the environment. In the presence of wind, this safety metric ensures that the aerial robot maintains a safe distance with the human collaborator. The controller that embeds this safety metric is a Non-Linear Model Predictive Controller (NMPC). The controller generates control inputs, such that the safety constraint is satisfied throughout the trajectory, thereby ensuring safe operation with the human collaborator. Simulations were performed to construct this safety metric, and the system was tested with various wind conditions. The system’s performance under these conditions was evaluated to determine if the safety metric remained satisfied throughout the whole trajectory. Finally, a brief conclusion and potential directions for future work have been detailed.
Jiaxu Xing, Angel Romero, Leonard Bauersfeld, Davide Scaramuzza: Bootstrapping Reinforcement Learning with Imitation for Vision-Based Agile Flight. CoRR abs/2403.12203 (2024)
Abstract:
Learning visuomotor policies for agile quadrotor flight presents significant difficulties, primarily from inefficient policy exploration caused by high-dimensional visual inputs and the need for precise and low-latency control. To address these challenges, we propose a novel approach that combines the performance of Reinforcement Learning (RL) and the sample efficiency of Imitation Learning (IL) in the task of vision-based autonomous drone racing. While RL provides a framework for learning high-performance controllers through trial and error, it faces challenges with sample efficiency and computational demands due to the high dimensionality of visual inputs. Conversely, IL efficiently learns from visual expert demonstrations, but it remains limited by the expert’s performance and state distribution. To overcome these limitations, our policy learning framework integrates the strengths of both approaches. Our framework contains three phases: training a teacher policy using RL with privileged state information, distilling it into a student policy via IL, and adaptive fine-tuning via RL. Testing in both simulated and real-world scenarios shows our approach can not only learn in scenarios where RL from scratch fails but also outperforms existing IL methods in both robustness and performance, successfully navigating a quadrotor through a race course using only visual information. Videos of the experiments are available at https://rpg.ifi.uzh.ch/bootstrap-rl-with-il/index.html.
Abstract:
This work explores the potential of using differentiable simulation for learning quadruped locomotion. Differentiable simulation promises fast convergence and stable training by computing low-variance first-order gradients using robot dynamics. However, its usage for legged robots is still limited to simulation. The main challenge lies in the complex optimization landscape of robotic tasks due to discontinuous dynamics. This work proposes a new differentiable simulation framework to overcome these challenges. Our approach combines a high-fidelity, non-differentiable simulator for forward dynamics with a simplified surrogate model for gradient backpropagation. This approach maintains simulation accuracy by aligning the robot states from the surrogate model with those of the precise, non-differentiable simulator. Our framework enables learning quadruped walking in simulation in minutes without parallelization. When augmented with GPU parallelization, our approach allows the quadruped robot to master diverse locomotion skills on challenging terrains in minutes. We demonstrate that differentiable simulation outperforms a reinforcement learning algorithm (PPO) by achieving significantly better sample efficiency while maintaining its effectiveness in handling large-scale environments. Our method represents one of the first successful applications of differentiable simulation to real-world quadruped locomotion, offering a compelling alternative to traditional RL methods.
Abstract:
In this letter, we detail a comprehensive framework for safe and robust planning for robots in presence of model uncertainties. Our framework is based on the recent notion of closed-loop state sensitivity, which is extended in this work to also include uncertainties in the initial state. The proposed framework, which considers the sensitivity of the nominal closed-loop system w.r.t. both model parameters and initial state mismatches, is exploited to compute tubes that accurately capture the worst-case effects of the considered uncertainties. In comparison to the current state-of-the-art for safe and robust planning, the proposed closed-loop state sensitivity framework has the important advantage of computational simplicity and minimal assumptions (and simplifications) regarding the underlying robot closed-loop dynamics. The approach is validated via both extensive simulations and real-world experiments. In the experiments we consider as case study a nonlinear trajectory optimization problem aimed at generating an intrinsically robust and safe trajectory for an aerial robot for safely performing an obstacle avoidance maneuver despite the uncertainties. Simulation and experimental results further confirm the viability and interest of the proposed approach.
Abstract:
In the last decade, usage of MRAVs (Multirotor Aerial Vehicles) across multiple domains, has expanded significantly. Technological innovations greatly enhanced their capabilities and rendered them well-suited for operating in perilous areas such as maritime environments, like ballast tanks. These confined and hazardous areas pose risks to human inspectors, making MRAVs a safer and more reliable alternative. However, there remains the need for optimized MRAV designs tailored to complex tasks, such as autonomous contact inspections in these areas. This thesis presents a generalized, systematic, design procedure aimed at optimizing MRAVs based on their tasks and through that, propose a suitable design, optimized for the task of autonomous contact-inspection on the walls of a ballast tank, under the scope of the AUTOASSESS. At the core of this procedure lies the formulation of the design problem, as a constrained multivariable nonlinear optimization problem (NLP), allowing holistic consideration of task requirements through constraints or components of a unified objective function, derived from modified cost functions from the literature. The software outputs optimized MRAV designs tailored to the tasks and with selected performance metrics, one can evaluate and compare the to make informed decisions about the optimal configuration, based on task requirements and performance.
Abstract:
Visual Odometry (VO) is crucial for autonomous robotic navigation, especially in GPS-denied environments like planetary terrains. To improve robustness, recent model-based VO systems have begun combining standard and event-based cameras. While event cameras excel in low-light and high-speed motion, standard cameras provide dense and easier-to-track features. However, the field of image- and event-based VO still predominantly relies on model-based methods and is yet to fully integrate recent image-only advancements leveraging end-to-end learning-based architectures. Seamlessly integrating the two modalities remains challenging due to their different nature, one asynchronous, the other not, limiting the potential for a more effective image- and event-based VO. We introduce RAMP-VO, the first end-to-end learned image- and event-based VO system. It leverages novel Recurrent, Asynchronous, and Massively Parallel (RAMP) encoders capable of fusing asynchronous events with image data, providing 8x faster inference and 33% more accurate predictions than existing solutions. Despite being trained only in simulation, RAMP-VO outperforms previous methods on the newly introduced Apollo and Malapert datasets, and on existing benchmarks, where it improves image- and event-based methods by 58.8% and 30.6%, paving the way for robust and asynchronous VO in space.
Abstract:
Reinforcement learning (RL) has shown great effectiveness in quadrotor control, enabling specialized policies to develop even human-champion-level performance in single-task scenarios. However, these specialized policies often struggle with novel tasks, requiring a complete retraining of the policy from scratch. To address this limitation, this paper presents a novel multi-task reinforcement learning (MTRL) framework tailored for quadrotor control, leveraging the shared physical dynamics of the platform to enhance sample efficiency and task performance. By employing a multi-critic architecture and shared task encoders, our framework facilitates knowledge transfer across tasks, enabling a single policy to execute diverse maneuvers, including high-speed stabilization, velocity tracking, and autonomous racing. Our experimental results, validated both in simulation and real-world scenarios, demonstrate that our framework outperforms baseline approaches in terms of sample efficiency and overall task performance.
Roxane Merat, Giovanni Cioffi, Leonard Bauersfeld, Davide Scaramuzza
Abstract:
Globally-consistent localization in urban environments is crucial for autonomous systems such as self-driving vehicles and drones, as well as assistive technologies for visually impaired people. Traditional Visual-Inertial Odometry (VIO) and Visual Simultaneous Localization and Mapping (VSLAM) methods, though adequate for local pose estimation, suffer from drift in the long term due to reliance on local sensor data. While GPS counteracts this drift, it is unavailable indoors and often unreliable in urban areas. An alternative is to localize the camera to an existing 3D map using visual-feature matching. This can provide centimeter-level accurate localization but is limited by the visual similarities between the current view and the map. This paper introduces a novel approach that achieves accurate and globally-consistent localization by aligning the sparse 3D point cloud generated by the VIO/VSLAM system to a digital twin using point-to-plane matching; no visual data association is needed. The proposed method provides a 6-DoF global measurement tightly integrated into the VIO/VSLAM system. Experiments run on a high-fidelity GPS simulator and real-world data collected from a drone demonstrate that our approach outperforms state-of-the-art VIO-GPS systems and offers superior robustness against viewpoint changes compared to the state-of-the-art Visual SLAM systems.
Abstract:
We propose a new multirotor aerial vehicle class of designs composed of a multi-body structure in which a main body is connected by passive joints to links equipped with propellers. We have investigated some instances of such class, some of which are shown to achieve omnidirectionality while having a minimum number of inputs equal to the main body Degrees of Freedom DoF’s, only uni-directional positive thrust propellers, and no internal forces generated at steady state. After dynamics are derived following the Euler-Lagrange approach, an I/O dynamic feedback linearization strategy is then used to show the controllability of any desired pose with stable zero dynamics. We finally verify the developed controller with closed-loop simulations.
Abstract:
The widespread adoption of quadrotors for diverse applications, from agriculture to public safety, necessitates an understanding of the aerodynamic disturbances they create. This paper introduces a computationally lightweight model for estimating the time-averaged magnitude of the induced flow below quadrotors in hover. Unlike related approaches that rely on expensive computational fluid dynamics (CFD) simulations or drone specific time-consuming empirical measurements, our method leverages classical theory from turbulent flows. By analyzing over 16 hours of flight data from drones of varying sizes within a large motion capture system, we show for the first time that the combined flow from all drone propellers is well-approximated by a turbulent jet after 2.5 drone-diameters below the vehicle. Using a novel normalization and scaling, we experimentally identify model parameters that describe a unified mean velocity field below differently sized quadrotors. The model, which requires only the drone’s mass, propeller size, and drone size for calculations, accurately describes the far-field airflow over a long-range in a very large volume which is impractical to simulate using CFD. Our model offers a practical tool for ensuring safer operations near humans, optimizing sensor placements and drone control in multi-agent scenarios. We demonstrate the latter by designing a controller that compensates for the downwash of another drone, leading to a four times lower altitude deviation when passing below.
Abstract:
In this work, we present a model-based optimal boundary control design for an aerial robotic system composed of a quadrotor carrying a flexible cable. The whole system is modeled by partial differential equations (PDEs) combined with boundary conditions described by ordinary differential equations (ODEs). The proper orthogonal decomposition (POD) method is adopted to project the original infinite-dimensional system on a finite low-dimensional space spanned by orthogonal basis functions. Based on such a reduced order model, nonlinear model predictive control (NMPC) is implemented online to realize both position and shape trajectory tracking of the flexible cable in an optimal predictive fashion. The proposed POD-based reduced modeling and optimal control paradigms are verified in simulation using an accurate high-dimensional FDM-based model and experimentally using a real quadrotor and a cable. The results show the viability of the POD-based predictive control approach (allowing closing the control loop on the full system state) and its superior performance compared to an optimally tuned PID controller (allowing closing the control loop on the quadrotor state only).
Abstract:
Autonomous exploration of unknown space is an essential component for the deployment of mobile robots in the real world. Safe navigation is crucial for all robotics applications and requires accurate and consistent maps of the robot’s surroundings. To achieve full autonomy and allow deployment in a wide variety of environments, the robot must rely on on-board state estimation which is prone to drift over time. We propose a Micro Aerial Vehicle (MAV) exploration framework based on local submaps to allow retaining global consistency by applying loop-closure corrections to the relative submap poses. To enable large-scale exploration we efficiently compute global, environment-wide frontiers from the local submap frontiers and use a sampling-based next-best-view exploration planner. Our method seamlessly supports using either a LiDAR sensor or a depth camera, making it suitable for different kinds of MAV platforms. We perform comparative evaluations in simulation against a state-of-the-art submap-based exploration framework to showcase the efficiency and reconstruction quality of our approach. Finally, we demonstrate the applicability of our method to real-world MAVs, one equipped with a LiDAR and the other with a depth camera. Video available at this https URL.
Abstract:
Enabling aerial robots to handle dynamic contacts happening at non-vanishing speeds can enlarge the range of their applications. In this work, we propose an impactaware strategy to allow aerial multirotor robots to recover from impacts. The method leverages a reactive strategy not requiring low-level changes to the motion controller commonly implemented onboard quadrotors, which might be not viable or not desirable for most users. Extensive simulation tests show that the proposed strategy considerably increases the tolerated velocity at impact in tasks in which the robot either picks an object up or collides against an object to clear its way. Preliminary experimental results using Crazyflie UAVs are also presented.
Abstract:
We propose visual-inertial simultaneous localization and mapping that tightly couples sparse reprojection errors, inertial measurement unit pre-integrals, and relative pose factors with dense volumetric occupancy mapping. Hereby depth predictions from a deep neural network are fused in a fully probabilistic manner. Specifically, our method is rigorously uncertainty-aware: first, we use depth and uncertainty predictions from a deep network not only from the robot’s stereo rig, but we further probabilistically fuse motion stereo that provides depth information across a range of baselines, therefore drastically increasing mapping accuracy. Next, predicted and fused depth uncertainty propagates not only into occupancy probabilities but also into alignment factors between generated dense submaps that enter the probabilistic nonlinear least squares estimator. This submap representation offers globally consistent geometry at scale. Our method is thoroughly evaluated in two benchmark datasets, resulting in localization and mapping accuracy that exceeds the state of the art, while simultaneously offering volumetric occupancy directly usable for downstream robotic planning and control in real-time.
Johannes, Heeg; Song, Yunlong; Scaramuzza, Davide (2024): Learning Quadrotor Control From Visual Features Using Differentiable Simulation
Abstract:
The sample inefficiency of reinforcement learning (RL) remains a significant challenge in robotics. RL requires large-scale simulation and can still cause long training times, slowing research and innovation. This issue is particularly pronounced in vision-based control tasks where reliable state estimates are not accessible. Differentiable simulation offers an alternative by enabling gradient back-propagation through the dynamics model, providing low-variance analytical policy gradients and, hence, higher sample efficiency. However, its usage for real-world robotic tasks has yet been limited. This work demonstrates the great potential of differentiable simulation for learning quadrotor control. We show that training in differentiable simulation significantly outperforms model-free RL in terms of both sample efficiency and training time, allowing a policy to learn to recover a quadrotor in seconds when providing vehicle states and in minutes when relying solely on visual features. The key to our success is two-fold. First, the use of a simple surrogate model for gradient computation greatly accelerates training without sacrificing control performance. Second, combining state representation learning with policy learning enhances convergence speed in tasks where only visual features are observable. These findings highlight the potential of differentiable simulation for real-world robotics and offer a compelling alternative to conventional RL approaches.
Abstract:
This paper proposes a method to robustify model predictive path integral (MPPI) control by directly taking into account the effects of parameter uncertainty into the controller formulation. Leveraging the recent notion of closed-loop state sensitivity, the proposed MPPI can consider the state sensitivity against parameter mismatch as a part of the system state, and consequently exploit this additional information to address the challenge of model mismatch in sampling-based model predictive control. Using an obstacle avoidance scenario, we demonstrate the use of our approach to control an aerial robot. We present an embedded implementation of our method, utilizing parallelization of computations on a GPU. Finally, we show the increased robustness of our approach over a standard MPPI controller through hardware-in-the-loop simulations and validate its embedded real-time properties.
(2025): Manipulation of Elasto-Flexible Cables with Single or Multiple UAVs
Abstract:
This work considers a large class of systems composed of multiple quadrotors manipulating deformable and extensible cables. The cable is described via a discretized representation, which decomposes it into linear springs interconnected through lumped-mass passive spherical joints. Sets of flat outputs are found for the systems. Numerical simulations support the findings by showing cable manipulation relying on flatness-based trajectories. Eventually, we present an experimental validation of the effectiveness of the proposed discretized cable model for a two-robot example. Moreover, a closed-loop controller based on the identified model and using cable-output feedback is experimentally tested.
Abstract:
To empower mobile robots with usable maps as well as highest state estimation accuracy and robustness, we present OKVIS2-X: a state-of-the-art multisensor simultaneous localization and mapping (SLAM) system building dense volumetric occupancy maps, while scalable to large environments and operating in realtime. Our unified SLAM framework seamlessly integrates different sensor modalities: visual, inertial, measured or learned depth, LiDAR, and Global Navigation Satellite System (GNSS) measurements. Unlike most state-of-the-art SLAM systems, we advocate using dense volumetric map representations when leveraging depth or range-sensing capabilities. We employ an efficient submapping strategy that allows our system to scale to large environments, showcased in sequences of up to 9 km. OKVIS2-X enhances its accuracy and robustness by tightly-coupling the estimator and submaps through map alignment factors. Our system provides globally consistent maps, directly usable for autonomous navigation. To further improve the accuracy of OKVIS2-X, we also incorporate the option of performing online calibration of camera extrinsics. Our system achieves the highest trajectory accuracy in EuRoC against state-of-the-art alternatives, outperforms all competitors in the Hilti22 VI-only benchmark, while also proving competitive in the LiDAR version, and showcases state of the art accuracy in the diverse and large-scale sequences from the VBR dataset.
(2025): Input-output extension for nonlinear systems
Abstract:
This letter proposes a method to integrate auxiliary actuators that enhance the task space capabilities of commercial underactuated systems, leaving the internal certified low level controller untouched. The additional actuators are combined with a feedback linearizing outer loop controller, enabling full pose tracking. We provide the conditions under which legacy high level commands and new actuator inputs can be cohesively coordinated to achieve decoupled control of all degrees of freedom. A comparative study with a standard quadrotor originally not designed for physical interaction demonstrates that the proposed modified platform remains stable under contact, while the baseline system diverges. Additionally, simulation results under parameter uncertainty illustrate the robustness of the approach.
Abstract:
Recent developments in 3D Gaussian Splatting have significantly enhanced novel view synthesis, yet generating high-quality renderings from extreme novel viewpoints or partially observed regions remains challenging. Meanwhile, diffusion models exhibit strong generative capabilities, but their reliance on text prompts and lack of awareness of specific scene information hinder accurate 3D reconstruction tasks. To address these limitations, we introduce GSFix3D, a novel framework that improves the visual fidelity in under-constrained regions by distilling prior knowledge from diffusion models into 3D representations, while preserving consistency with observed scene details. At its core is GSFixer, a latent diffusion model obtained via our customized fine-tuning protocol that can leverage both mesh and 3D Gaussians to adapt pretrained generative models to a variety of environments and artifact types from different reconstruction methods, enabling robust novel view repair for unseen camera poses. Moreover, we propose a random mask augmentation strategy that empowers GSFixer to plausibly inpaint missing regions. Experiments on challenging benchmarks demonstrate that our GSFix3D and GSFixer achieve state-of-the-art performance, requiring only minimal scene-specific fine-tuning on captured data. Real-world test further confirms its resilience to potential pose errors. Our code and data will be made publicly available.
Abstract:
This paper presents UniPilot, a compact hardware-software autonomy payload that can be integrated across diverse robot embodiments to enable autonomous operation in GPS-denied environments. The system integrates a multi-modal sensing suite including LiDAR, radar, vision, and inertial sensing for robust operation in conditions where uni-modal approaches may fail. UniPilot runs a complete autonomy software comprising multi-modal perception, exploration and inspection path planning, and learning-based navigation policies. The payload provides robust localization, mapping, planning, and safety and control capabilities in a single unit that can be deployed across a wide range of platforms. A large number of experiments are conducted across diverse environments and on a variety of robot platforms to validate the mapping, planning, and safe navigation capabilities enabled by the payload.
(2025): Unified Feedback Linearization for Nonlinear Systems with Dexterous and Energy-Saving Modes
Abstract:
Systems with a high number of inputs compared to the degrees of freedom (e.g. a mobile robot with Mecanum wheels) often have a minimal set of energy-efficient inputs needed to achieve a main task (e.g. position tracking) and a set of energy-intense inputs needed to achieve an additional auxiliary task (e.g. orientation tracking). This letter presents a unified control scheme, derived through feedback linearization, that can switch between two modes: an energy-saving mode, which tracks the main task using only the energy-efficient inputs while forcing the energy-intense inputs to zero, and a dexterous mode, which also uses the energy-intense inputs to track the auxiliary task as needed. The proposed control guarantees the exponential tracking of the main task and that the dynamics associated with the main task evolve independently of the a priori unknown switching signal. When the control is operating in dexterous mode, the exponential tracking of the auxiliary task is also guaranteed. Numerical simulations on an omnidirectional Mecanum wheel robot validate the effectiveness of the proposed approach and demonstrate the effect of the switching signal on the exponential tracking behavior of the main and auxiliary tasks.
(2025): Predictive Admittance Control for Aerial Physical Interaction
Abstract:
This letter introduces a novel approach for controlling aerial robots during physical interaction by integrating Admittance Control with Nonlinear Model Predictive Control (NMPC). Unlike existing methods, our technique incorporates the desired impedance dynamics directly into the NMPC prediction model, alongside the robot’s dynamics. This allows for the explicit prediction of how the robot’s impedance will respond to interaction forces within the prediction horizon. Consequently, our controller effectively tracks the desired impedance behavior during physical interaction while seamlessly transitioning to trajectory tracking in free motion, all while consistently respecting actuator constraints. The efficacy of this method is validated through real-time simulations and experiments involving physical interaction tasks with an aerial robot. Our findings demonstrate that, across most scenarios, our method significantly outperforms the state-of-the-art (which does not predict future impedance state), achieving a reduction in tracking error of up to 90%. Furthermore, the results indicate that our approach enables smoother and safer physical interaction, characterized by reduced oscillations and the absence of the unstable behavior observed with the state-of-the-art method in certain situations.
Abstract:
This letter presents a systematic framework for switching between different sets of outputs for the control of nonlinear systems via feedback linearization. We introduce the concept of a meld to formally define a valid, feedback-linearizable subset of outputs that can be selected from a larger deck of possible outputs. The main contribution is a formal proof establishing that under suitable dwell-time and compatibility conditions, it is possible to switch between different melds while guaranteeing the uniform boundedness of the system state. We further show that the error dynamics of the active outputs remain exponentially stable within each switching interval and that outputs common to consecutive melds are tracked seamlessly through transitions. The proposed theory is valid for any feedback linearizable nonlinear system, such as, e.g., robots, aerial and terrestrial vehicles, etc.. We demonstrate it on a simple numerical simulation of a robotic manipulator.
Abstract:
Computational human attention modeling in free-viewing and task-specific settings is often studied separately, with limited exploration of whether a common representation exists between them. This work investigates this question and proposes a neural network architecture that builds upon the Human Attention transformer (HAT) to test the hypothesis. Our results demonstrate that free-viewing and visual search can efficiently share a common representation, allowing a model trained in free-viewing attention to transfer its knowledge to task-driven visual search with a performance drop of only 3.86% in the predicted fixation scanpaths, measured by the semantic sequence score (SemSS) metric which reflects the similarity between predicted and human scanpaths. This transfer reduces computational costs by 92.29% in terms of GFLOPs and 31.23% in terms of trainable parameters.
Abstract:
Autonomous inspection of maritime vessels using Unmanned Aerial Vehicles can prove difficult due to low light conditions, minimum space and battery limitations. This paper proposes a strategy for optimized manhole traversal in the scenario of maritime vessel ballast tank inspection. A dense point cloud, collected during the exploration of the ballast tank, is used to localize the manhole that the UAV has to traverse through. The prediction of the manhole is performed on synthesized
360◦ panoramic views of the point cloud, originating from positions inside the mapped area. This manhole prediction enables a Model Predictive Control framework to find the optimal path through the manhole, taking into account the capabilities of the UAV and the topological constraints of the confined space. The manhole localization method is able to predict the center of the manhole with an average success rate of 89.6%, which can be raised to 94% after the ensembling of multiple predictions from different assumed camera positions. Simulated results show the robustness of the Model Predictive Control method in a wide array of different scenarios. We demonstrate the ability of the method to predict accurately the manhole and efficiently pass through it using a UAV, in a real-world experimental setup.
Abstract:
Unmanned aerial systems (UAS) offer substantial benefits in maritime vessel inspections by significantly reducing operational costs, since traditional inspection methods involve manual labor and extended downtime for vessels. To fully leverage these advantages, important barriers such as flight time must be addressed. Introducing a tether connected to a ground station enables constant power delivery, effectively overcoming this limitation; however, planning complexity is greatly increased. In our work, we developed a motion planning strategy for a system of four tethered drones connected in a string formation. This strategy allows the first drone in the chain to follow a pre-defined path in a confined space while repositioning the others to avoid collisions. To achieve this, the UAS was modeled as a serial kinematic chain, by treating the UAVs as joints and the tethers as links. MoveIt2, a popular robotic manipulator framework, is employed to provide access to sampling-based algorithms and inverse kinematic solvers, while NVIDIA’s Isaac Sim was used for visualization. A custom line-of-sight constraint was created and integrated into MoveIt2 using Bullet to enable collision checking for the tethers. Different sampling-based algorithms were compared using our heuristic under varying planning time parameters. The proposed strategy improves the performance of these algorithms across various metrics.
Abstract:
This paper presents a novel alternative approach to perception for aerial vehicles operating in confined spaces such as ballast tanks. It focuses on designing and implementing a system of small Time-of-Flight (ToF) sensors arranged in an array that could replace conventional, expensive, and bulky 2D or 3D LiDAR systems. Confined spaces often feature narrow, dark, and dusty environments. Here, conventional vision-based solutions fail for autonomous navigation and even basic flight tasks such as obstacle avoidance. Furthermore, carrying conventional LiDAR sensors significantly reduces flight time, especially for small drones(< 2.5 kg). Therefore, this paper proposes a system using VL53L8CX sensors as an optimal alternative, which significantly reduces the weight and computational power. A novel approach is developed to combine 12 of these ToF sensors across two SPI buses. This setup provides a 3600 radial field of view within a 4m radius around the aerial vehicle, which is strategically designed to accommodate these sensors. The proposed design is validated and compared against a 2D lidar system for collision prevention in a PX4 SITL simulation of a ballast tank environment. Additionally, the sparsity and accuracy of the generated point cloud are compared with those of 3D LiDAR while flying within a mock ballast tank. This design and study aim to establish a baseline for lightweight, compact, and safe navigation for small drones in confined and featureless environments.
Abstract:
This paper presents a novel semantics-aware inspection path planning paradigm called “Semantics-aware Predictive Planning” (SPP). Industrial environments that require the inspection of specific objects or structures (called “semantics”), such as ballast water tanks inside ships, often present structured and repetitive spatial arrangements of the semantics of interest. Motivated by this, we first contribute an algorithm that identifies spatially repeating patterns of semantics – exact or inexact – in a semantic scene graph representation and makes predictions about the evolution of the graph in the unseen parts of the environment using these patterns. Furthermore, two inspection path planning strategies, tailored to ballast water tank inspection, that exploit these predictions are proposed. To assess the performance of the novel predictive planning paradigm, both simulation and experimental evaluations are performed. First, we conduct a simulation study comparing the method against relevant state-of-the-art techniques and further present tests showing its ability to handle imperfect patterns. Second, we deploy our method onboard a collision-tolerant aerial robot operating inside the ballast tanks of two real ships. The results, both in simulation and field experiments, demonstrate significant improvement over the state-of-the-art in terms of inspection time while maintaining equal or better semantic surface coverage. A set of videos describing the different parts of the method and the field deployments are available at https://tinyurl.com/spp-videos. The code for this work is made available at https://github.com/ntnu-arl/predictive planning ros.
Abstract:
In this paper, we present a novel visual servoing (VS) approach based on latent Denoising Diffusion Probabilistic Models (DDPMs), that explores the application of generative models for vision-based navigation of UAVs (Uncrewed Aerial Vehicles). Opposite to classical VS methods, the proposed approach allows reaching the desired target view, even when the target is initially not visible. This is possible thanks to the learning of a latent representation that the DDPM uses for planning and a dataset of trajectories encompassing target-invisible initial views. A compact representation is learned from raw images using a Cross-Modal Variational Autoencoder. Given the current image, the DDPM generates trajectories in the latent space driving the robotic platform to the desired visual target. The approach has been validated in simulation using two generic multi-rotor UAVs (a quadrotor and a hexarotor). The results show that we can successfully reach the visual target, even if not visible in the initial view. A video summary with simulations can be found here.
Abstract:
Understanding how humans leverage prior knowledge to navigate unseen environments while making exploratory decisions is essential for developing autonomous robots with similar abilities. In this work, we propose ForesightNav, a novel exploration strategy inspired by human imagination and reasoning. Our approach equips robotic agents with the capability to predict contextual information, such as occupancy and semantic details, for unexplored regions. These predictions enable the robot to efficiently select meaningful long-term navigation goals, significantly enhancing exploration in unseen environments. We validate our imagination-based approach using the Structured3D dataset, demonstrating accurate occupancy prediction and superior performance in anticipating unseen scene geometry. Our experiments show that the imagination module improves exploration efficiency in unseen environments, achieving a 100% completion rate for PointNav and an SPL of 67% for ObjectNav on the Structured3D Validation split. These contributions demonstrate the power of imagination-driven reasoning for autonomous systems to enhance generalizable and efficient exploration.
Abstract:
Reinforcement learning (RL) has achieved outstanding success in complex robot control tasks, such as drone racing, where the RL agents have outperformed human champions in a known racing track. However, these agents fail in unseen track configurations, always requiring complete retraining when presented with new track layouts. This work aims to develop RL agents that generalize effectively to novel track configurations without retraining. The naive solution of training directly on a diverse set of track layouts can overburden the agent, resulting in suboptimal policy learning as the increased complexity of the environment impairs the agent’s ability to learn to fly. To enhance the generalizability of the RL agent, we propose an adaptive environment-shaping framework that dynamically adjusts the training environment based on the agent’s performance. We achieve this by leveraging a secondary RL policy to design environments that strike a balance between being challenging and achievable, allowing the agent to adapt and improve progressively. Using our adaptive environment shaping, one single racing policy efficiently learns to race in diverse challenging tracks. Experimental results validated in both simulation and the real world show that our method enables drones to successfully fly complex and unseen race tracks, outperforming existing environment-shaping techniques.
Abstract:
Learning control policies in simulation enables rapid, safe, and cost-effective development of advanced robotic capabilities. However, transferring these policies to the real world remains difficult due to the sim-to-real gap, where unmodeled dynamics and environmental disturbances can degrade policy performance. Existing approaches, such as domain randomization and Real2Sim2Real pipelines, can improve policy robustness, but either struggle under out-of-distribution conditions or require costly offline retraining. In this work, we approach these problems from a different perspective. Instead of relying on diverse training conditions before deployment, we focus on rapidly adapting the learned policy in the real world in an online fashion. To achieve this, we propose a novel online adaptive learning framework that unifies residual dynamics learning with real-time policy adaptation inside a differentiable simulation. Starting from a simple dynamics model, our framework refines the model continuously with real-world data to capture unmodeled effects and disturbances such as payload changes and wind. The refined dynamics model is embedded in a differentiable simulation framework, enabling gradient backpropagation through the dynamics and thus rapid, sample-efficient policy updates beyond the reach of classical RL methods like PPO. All components of our system are designed for rapid adaptation, enabling the policy to adjust to unseen disturbances within 5 seconds of training. We validate the approach on agile quadrotor control under various disturbances in both simulation and the real world. Our framework reduces hovering error by up to 81% compared to L1-MPC and 55% compared to DATT, while also demonstrating robustness in vision-based control without explicit state estimation.
Abstract:
Autonomous quadrotor flight in confined spaces such as pipes and tunnels presents significant challenges due to unsteady, self-induced aerodynamic disturbances. Very recent advances have enabled flight in such conditions, but they either rely on constant motion through the pipe to mitigate airflow recirculation effects or suffer from limited stability during hovering. In this work, we present the first closed-loop control system for quadrotors for hovering in narrow pipes that leverages real-time flow field measurements. We develop a low-latency, event-based smoke velocimetry method that estimates local airflow at high temporal resolution. This flow information is used by a disturbance estimator based on a recurrent convolutional neural network, which infers force and torque disturbances in real time. The estimated disturbances are integrated into a learning-based controller trained via reinforcement learning. The flow-feedback control proves particularly effective during lateral translation maneuvers in the pipe cross-section. There, the real-time disturbance information enables the controller to effectively counteract transient aerodynamic effects, thereby preventing collisions with the pipe wall. To the best of our knowledge, this work represents the first demonstration of an aerial robot with closed-loop control informed by real-time flow field measurements. This opens new directions for research on flight in aerodynamically complex environments. In addition, our work also sheds light on the characteristic flow structures that emerge during flight in narrow, circular pipes, providing new insights at the intersection of robotics and fluid dynamics.
Abstract:
A key open challenge in agile quadrotor flight is how to combine the flexibility and task-level generality of model-free reinforcement learning (RL) with the structure and online replanning capabilities of model predictive control (MPC), aiming to leverage their complementary strengths in dynamic and uncertain environments. This paper provides an answer by introducing a new framework called Actor-Critic Model Predictive Control. The key idea is to embed a differentiable MPC within an actor-critic RL framework. This integration allows for short-term predictive optimization of control actions through MPC, while leveraging RL for end-to-end learning and exploration over longer horizons. Through various ablation studies, conducted in the context of agile quadrotor racing, we expose the benefits of the proposed approach: it achieves better out-of-distribution behaviour, better robustness to changes in the quadrotor’s dynamics and improved sample efficiency. Additionally, we conduct an empirical analysis using a quadrotor platform that reveals a relationship between the critic’s learned value function and the cost function of the differentiable MPC, providing a deeper understanding of the interplay between the critic’s value and the MPC cost functions. Finally, we validate our method in a drone racing task on different tracks, in both simulation and the real world. Our method achieves the same superhuman performance as state-of-the-art model-free RL, showcasing speeds of up to 21 m/s. We show that the proposed architecture can achieve real-time control performance, learn complex behaviors via trial and error, and retain the predictive properties of the MPC to better handle out-of-distribution behavior.
Abstract:
Visual-inertial odometry (VIO) is widely used for state estimation in autonomous micro aerial vehicles using onboard sensors. Current methods improve VIO by incorporating a model of the translational vehicle dynamics, yet their performance degrades when faced with low-accuracy vehicle models or continuous external disturbances, like wind. Additionally, incorporating rotational dynamics in these models is computationally intractable when they are deployed in online applications, e.g., in a closed-loop control system. We present HDVIO2.0, which models full 6-DoF, translational and rotational, vehicle dynamics and tightly incorporates them into a VIO with minimal impact on the runtime. HDVIO2.0 builds upon the previous work, HDVIO, and addresses these challenges through a hybrid dynamics model combining a point-mass vehicle model with a learning-based component, with access to control commands and IMU history, to capture complex aerodynamic effects. The key idea behind modeling the rotational dynamics is to represent them with continuous-time functions. HDVIO2.0 leverages the divergence between the actual motion and the predicted motion from the hybrid dynamics model to estimate external forces as well as the robot state. Our system surpasses the performance of state-of-the-art methods in experiments using public and new drone dynamics datasets, as well as real-world flights in winds up to 25 km/h. Unlike existing approaches, we also show that accurate vehicle dynamics predictions are achievable without precise knowledge of the full vehicle state.
Public Deliverables
This is the first version of the Data Management Plan (DMP) for the AUTOASSESS project. The DMP is a live document that is updated throughout the life of the project.
This deliverable outlines the communication protocols for data exchange between the Unmanned Autonomous Systems (UAS) and offline map server employed in our activities.
The deliverable also identifies potential risks associated with data security, corruption, and transfer delays, along with proposed mitigation strategies.
The map server is a core element in autonomous vessel inspection with the main objectives as 1) Visualization of a 3D map and inspection results to human supervisors and 2) Supporting robot inspection missions by providing highly precise maps and navigation goals. Towards this goal, the map server maintains all the relevant sensor measurements and refines an offline map using available sensor observations from one or more UAS. This deliverable describes the overall data flow, data interfaces between the map server and UAS or UI-DSS, and a data structure for storage
This document defines AUTOASSESS’ communication and dissemination initial plans, outlining the target groups, as well as the channels, tools and activities to reach them.
Open Source Code
OKVIS2-X is a multi-sensor SLAM system based on a factor graph, and is a non-trivial extension of the sparse, landmark-based OKVIS2. OKVIS2-X supports fusing multiple cameras and an IMU, with optional integration of a GNSS receiver and LiDAR or depth images (from a depth network or depth sensor). For map representation, OKVIS2-X uses submap-based volumetric occupancy to seamlessly support robot exploration and navigation.
Recent developments in 3D Gaussian Splatting have significantly enhanced novel view synthesis, yet generating high-quality renderings from extreme novel viewpoints or partially observed regions remains challenging. Meanwhile, diffusion models exhibit strong generative capabilities, but their reliance on text prompts and lack of awareness of specific scene information hinder accurate 3D reconstruction tasks. To address these limitations, we introduce GSFix3D, a novel framework that improves the visual fidelity in under-constrained regions by distilling prior knowledge from diffusion models into 3D representations, while preserving consistency with observed scene details. At its core is GSFixer, a latent diffusion model obtained via our customized fine-tuning protocol that can leverage both mesh and 3D Gaussians to adapt pretrained generative models to a variety of environments and artifact types from different reconstruction methods, enabling robust novel view repair for unseen camera poses. Moreover, we propose a random mask augmentation strategy that empowers GSFixer to plausibly inpaint missing regions. Experiments on challenging benchmarks demonstrate that our GSFix3D and GSFixer achieve state-of-the-art performance, requiring only minimal scene-specific fine-tuning on captured data. Real-world test further confirms its resilience to potential pose errors.
Traditional volumetric fusion algorithms preserve the spatial structure of 3D scenes, which is beneficial for many tasks in computer vision and robotics. However, they often lack realism in terms of visualization. Emerging 3D Gaussian splatting bridges this gap, but existing Gaussian-based reconstruction methods often suffer from artifacts and inconsistencies with the underlying 3D structure, and struggle with real-time optimization, unable to provide users with immediate feedback in high quality. One of the bottlenecks arises from the massive amount of Gaussian parameters that need to be updated during optimization. Instead of using 3D Gaussian as a standalone map representation, we incorporate it into a volumetric mapping system to take advantage of geometric information and propose to use a quadtree data structure on images to drastically reduce the number of splats initialized. In this way, we simultaneously generate a compact 3D Gaussian map with fewer artifacts and a volumetric map on the fly. Our method, GSFusion, significantly enhances computational efficiency without sacrificing rendering quality, as demonstrated on both synthetic and real datasets.
A ROS 2 package for semantic detection inside Ballast Water Tanks (BWTs). It detects structural elements — walls, manholes, longitudinal stiffeners, and compartments — and fuses them into a Semantic Scene Graph (SSG) that associates each instance with a class ID, instance ID, and 3D center in world coordinates.