Real-time 3D hierarchical semantic mapping in outdoor scenarios
Students should have strong programming skills in Python and C++. Knowledge about Linux and ROS is useful but not necessary. Experience with deep learning frameworks is highly preferred (PyTorch or Tensorflow).
This project aims to explore using monocular cameras in the real-time construction of 3D scene graphs in outdoor applications. 3D scene graphs are high-level representations of environments crucial for the next generation of robots and autonomous systems to derive intelligent decisions with scene understanding. Our research group designed an algorithm to build real-time 3d hierarchical mapping from single camera input called Mono-Hydra only for indoor use cases. This thesis proposes that students extend this algorithm to work with outdoor scenarios where students have to develop/fine-tune a set of machine learning algorithms to work with outdoor environments.
Machine learning techniques are used to derive depth, and semantics, where odometry is derived using a novel approach called R-VIO2. This information is interfaced with the Hydra framework to generate real-time hierarchical mapping in indoor scenarios. Further improvements need to be performed on the Hydra network to configure it for outdoor use cases where the labels for the outdoor 3d scene graph need to be updated to suit outdoor scenarios.
- Hughes, N., Chang, Y. and Carlone, L. (2022) Hydra: A real-time spatial perception system for 3D scene graph construction and optimization, arXiv.org. Available at: https://doi.org/10.48550/arXiv.2201.13360
- Rosinol, A. et al. (2020) Kimera: An open-source library for real-time metric-semantic localization and mapping, arXiv.org. Available at: https://doi.org/10.48550/arXiv.1910.02490
- Rosinol, A., Gupta, A., et al. (2020) 3D dynamic scene graphs: Actionable spatial perception with places, objects, and humans, arXiv.org. Available at: https://doi.org/10.48550/arXiv.2002.06289
- Z. Huai and G. Huang, "Square-Root Robocentric Visual-Inertial Odometry With Online Spatiotemporal Calibration," in IEEE Robotics and Automation Letters, vol. 7, no. 4, pp. 9961-9968, Oct. 2022, doi: 10.1109/LRA.2022.3191209.