2. Deep Learning Approaches for Automated Calibration of Multi-Sensor Systems

Deep Learning Approaches for Automated Calibration of Multi-Sensor Systems

Robotics
ACQUAL
Staff Involved
Ville Lehtola
Additional Remarks

Students should possess strong programming skills in Python and C++, and familiarity with Linux and Docker is beneficial, though not required. The multi-sensor backpack system is equipped with 2 x VLP16 lidars, Ladybug5+ HD camera, BD990 Trimble GNSS, and XSens MTI 630R IMU.

Topic description

Calibration of multi-sensor systems is a pivotal step in ensuring the accuracy and reliability of data derived from these sensors. Traditional calibration methods often involve manual interventions, specialized equipment, and predefined calibration patterns. These methods can be labor-intensive and may not adapt well to the dynamism of real-world conditions or variations across different sensor setups. A preferred outcome would be a deep learning model capable of autonomously calibrating multi-sensor systems without the need for manual interventions. The outcomes of this research could be applied in various fields where multi-sensor systems are employed, such as robotics, autonomous vehicles, remote sensing, and industrial automation.

Image from: https://doi.org/10.48550/arXiv.2308.14414

Topic objectives and methodology

The main objective of this research is to design and evaluate deep learning-based techniques to automate the calibration process for multi-sensor systems, potentially eliminating the need for specialized calibration equipment or predefined patterns.

  1. Dataset Collection: Gather data from LIDAR+camera+IMU multi-sensor system in indoor environments. The data is used to train the deep learning model.
  2. Model Development: Investigate various deep learning architectures and choose one suitable for the calibration task. Sensors to be calibrated are Lidars and cameras.
    1. Neural Fields: https://doi.org/10.48550/arXiv.2308.14414
    2. RCNNs:          https://github.com/zjut-jianhuazhang/CalibR
    3. Transformers: https://github.com/epiception/CalibNet
    4. Review:           https://doi.org/10.1007/s10462-022-10317-y
  3. Model Training and Validation: Train the chosen model(s) on the collected dataset, employing techniques like cross-validation to assess the model's performance.

Evaluation: Test the deep learning model on new, unseen multi-sensor system configurations to evaluate its generalization capability. This will also involve comparing its performance with traditional calibration methods in terms of accuracy, speed, and reliability.