“Aerial path planning for online real-time exploration and offline high-quality reconstruction of large-scale urban scenes” by Liu, Cui, Xie, Gong and Huang
Conference:
Type(s):
Title:
- Aerial path planning for online real-time exploration and offline high-quality reconstruction of large-scale urban scenes
Session/Category Title: Scene Synthesis and Navigation
Presenter(s)/Author(s):
Abstract:
Existing approaches have shown that, through carefully planning flight trajectories, images captured by Unmanned Aerial Vehicles (UAVs) can be used to reconstruct high-quality 3D models for real environments. These approaches greatly simplify and cut the cost of large-scale urban scene reconstruction. However, to properly capture height discontinuities in urban scenes, all state-of-the-art methods require prior knowledge on scene geometry and hence, additional prepossessing steps are needed before performing the actual image acquisition flights. To address this limitation and to make urban modeling techniques even more accessible, we present a real-time explore-and-reconstruct planning algorithm that does not require any prior knowledge for the scenes. Using only captured 2D images, we estimate 3D bounding boxes for buildings on-the-fly and use them to guide online path planning for both scene exploration and building observation. Experimental results demonstrate that the aerial paths planned by our algorithm in realtime for unknown environments support reconstructing 3D models with comparable qualities and lead to shorter flight air time.
References:
1. Andreas Bircher, Mina Kamel, Kostas Alexis, Helen Oleynikova, and Roland Siegwart. 2016. Receding horizon next-best-view planner for 3D exploration. In Proc. IEEE Int. Conf. on Robotics & Automation. 1462–1468.
2. Xiaozhi Chen, Kaustav Kundu, Ziyu Zhang, Huimin Ma, Sanja Fidler, and Raquel Urtasun. 2016. Monocular 3D object detection for autonomous driving. In Proc. IEEE Conf. on Computer Vision & Pattern Recognition. 2147–2156.
3. Titus Cieslewski, Elia Kaufmann, and Davide Scaramuzza. 2017. Rapid exploration with multi-rotors: a frontier selection method for high speed flight. In Proc. IEEE Int. Conf. on Intelligent Robots & Systems. 2135–2142.
4. Mingyu Ding, Yuqi Huo, Hongwei Yi, Zhe Wang, Jianping Shi, Zhiwu Lu, and Ping Luo. 2020. Learning depth-guided convolutions for monocular 3D object detection. In Proc. IEEE Conf. on Computer Vision & Pattern Recognition. 11672–11681.
5. Xinyi Fan, Linguang Zhang, Benedict Brown, and Szymon Rusinkiewicz. 2016. Automated View and Path Planning for Scalable Multi-Object 3D Scanning. ACM Trans. on Graphics (Proc. SIGGRAPH Asia) 35, 6 (2016), 239:1–239:13 pages.
6. Carmelo Di Franco and Giorgio C. Buttazzo. 2015. Energy-Aware Coverage Path Planning of UAVs. In IEEE International Conference on Autonomous Robot Systems and Competitions. 111–117.
7. Simon Fuhrmann, Fabian Langguth, Nils Moehrle, Michael Waechter, and Michael Goesele. 2015. MVE – An Image-based Reconstruction Environment. Computers & Graphics 53 (2015), 44–53.
8. Yasutaka Furukawa, Brian Curless, Steven Seitz, and Richard Szeliski. 2010. Towards Internet-scale Multi-view Stereo. In Proc. IEEE Conf. on Computer Vision & Pattern Recognition, Vol. 2010. 1434–1441.
9. Tong He and Stefano Soatto. 2019. Mono3D++: Monocular 3D vehicle detection with two-scale 3D hypotheses and task priors. In Proc. AAAI Conf. on Artificial Intelligence, Vol. 33. 8409–8416.
10. Lionel Heng, Alkis Gotovos, Andreas Krause, and Marc Pollefeys. 2015. Efficient visual exploration and coverage with a micro aerial vehicle in unknown environments. In Proc. IEEE Int. Conf. on Robotics & Automation. 1071–1078.
11. Benjamin Hepp, Debadeepta Dey, Sudipta N Sinha, Ashish Kapoor, Neel Joshi, and Otmar Hilliges. 2018a. Learn-to-score: Efficient 3D scene exploration by predicting view utility. In Proc. Euro. Conf. on Computer Vision. 437–452.
12. B. Hepp, M. Nießner, and O. Hilliges. 2018b. Plan3D: Viewpoint and Trajectory Optimization for Aerial Multi-View Stereo Reconstruction. ACM Trans. on Graphics 38, 1 (2018), 4:1–4:17.
13. A. Hornung, B. Zeng, and L. Kobbelt. 2008. Image Selection for Improved Multi-View Stereo. In Proc. IEEE Conf. on Computer Vision & Pattern Recognition. 1–8.
14. Hou-Ning Hu, Qi-Zhi Cai, Dequan Wang, Ji Lin, Min Sun, Philipp Krahenbuhl, Trevor Darrell, and Fisher Yu. 2019. Joint monocular 3D vehicle detection and tracking. In Proc. Int. Conf. on Computer Vision. 5390–5399.
15. Rui Huang, Danping Zou, Richard Vaughan, and Ping Tan. 2018. Active Image-based Modeling with a Toy Drone. In Proc. IEEE Int. Conf. on Robotics & Automation. 6124–6133.
16. Sertac Karaman and Emilio Frazzoli. 2011. Sampling-based algorithms for optimal motion planning. Int. J. Robotics Research 30, 7 (2011), 846–894.
17. Tobias Koch, Marco Körner, and Friedrich Fraundorfer. 2019. Automatic and semantically-aware 3D UAV flight planning for image-based 3D reconstruction. Remote Sensing 11, 13 (2019), 1550.
18. Buyu Li, Wanli Ouyang, Lu Sheng, Xingyu Zeng, and Xiaogang Wang. 2019. Gs3D: An efficient 3D object detection framework for autonomous driving. In Proc. IEEE Conf. on Computer Vision & Pattern Recognition. 1019–1028.
19. Yilin Liu, Ke Xie, and Hui Huang. 2021a. VGF-Net: Visual-Geometric Fusion Learning for Simultaneous Drone Navigation and Height Mapping. Graphical Models 116 (2021), 101108.
20. Yilin Liu, Fuyou Xue, and Hui Huang. 2021b. UrbanScene3D: A Large Scale Urban Scene Dataset and Simulator. (2021). arXiv:2107.04286
21. Yuxuan Liu, Yuan Yixuan, and Ming Liu. 2021c. Ground-Aware Monocular 3D Object Detection for Autonomous Driving. IEEE Robotics and Automation Letters 6, 2 (2021), 919–926.
22. Xinzhu Ma, Zhihui Wang, Haojie Li, Pengbo Zhang, Wanli Ouyang, and Xin Fan. 2019. Accurate monocular 3D object detection via color-embedded 3D reconstruction for autonomous driving. In Proc. Int. Conf. on Computer Vision. 6851–6860.
23. Massimo Mauro, Hayko Riemenschneider, Alberto Signoroni, Riccardo Leonardi, and Luc Van Gool. 2014. A Unified Framework for Content-aware View Selection and Planning Through View Importance. In Proc. of British Machine Vision Conference. 1–11.
24. Zehui Meng, Hailong Qin, Ziyue Chen, Xudong Chen, Hao Sun, Feng Lin, and Marcelo H Ang Jr. 2017. A two-stage optimized next-view planning framework for 3D unknown environment exploration, and structural reconstruction. IEEE Robotics and Automation Letters 2, 3 (2017), 1680–1687.
25. Arsalan Mousavian, Dragomir Anguelov, John Flynn, and Jana Kosecka. 2017. 3D bounding box estimation using deep learning and geometry. In Proc. IEEE Conf. on Computer Vision & Pattern Recognition. 7074–7082.
26. Marc Pollefeys, David Nistér, J-M Frahm, Amir Akbarzadeh, Philippos Mordohai, Brian Clipp, Chris Engels, David Gallup, S-J Kim, Paul Merrell, et al. 2008. Detailed real-time urban 3d reconstruction from video. Int. J. Computer Vision 78, 2 (2008), 143–167.
27. Zengyi Qin, Jinglu Wang, and Yan Lu. 2019. Monogrnet: A geometric reasoning network for monocular 3D object localization. In Proc. AAAI Conf. on Artificial Intelligence, Vol. 33. 8851–8858.
28. Joseph Redmon and Ali Farhadi. 2018. YOLOv3: an incremental improvement. (2018). arXiv:1804.02767
29. Mike Roberts, Debadeepta Dey, Anh Truong, Sudipta Sinha, Shital Shah, Ashish Kapoor, Pat Hanrahan, and Neel Joshi. 2017. Submodular Trajectory Optimization for Aerial 3D Scanning. In Proc. Int. Conf. on Computer Vision. 5324–5333.
30. Antoni Rosinol, Marcus Abate, Yun Chang, and Luca Carlone. 2020. Kimera: an Open-Source Library for Real-Time Metric-Semantic Localization and Mapping. In Proc. IEEE Int. Conf. on Robotics & Automation. 1689–1696.
31. Lukas Schmid, Michael Pantic, Raghav Khanna, Lionel Ott, Roland Siegwart, and Juan Nieto. 2020. An efficient sampling-based method for online informative path planning in unknown environments. IEEE Robotics and Automation Letters 5, 2 (2020), 1500–1507.
32. Johannes Lutz Schönberger and Jan-Michael Frahm. 2016. Structure-from-Motion Revisited. In Proc. IEEE Conf. on Computer Vision & Pattern Recognition. 4104–4113.
33. Magnus Selin, Mattias Tiger, Daniel Duberg, Fredrik Heintz, and Patric Jensfelt. 2019. Efficient autonomous exploration planning of large-scale 3D environments. IEEE Robotics and Automation Letters 4, 2 (2019), 1699–1706.
34. Neil Smith, Nils Moehrle, Michael Goesele, and Wolfgang Heidrich. 2018. Aerial Path Planning for Urban Scene Reconstruction: A Continuous Optimization Method and Benchmark. ACM Trans. on Graphics (Proc. SIGGRAPH Asia) 37, 6 (2018), 183:1–183:15.
35. Yan Wang, Wei-Lun Chao, Divyansh Garg, Bharath Hariharan, Mark Campbell, and Kilian Q Weinberger. 2019. Pseudo-lidar from visual depth estimation: Bridging the gap in 3D object detection for autonomous driving. In Proc. IEEE Conf. on Computer Vision & Pattern Recognition. 8445–8453.
36. Xinshuo Weng and Kris Kitani. 2019. Monocular 3D object detection with pseudo-LIDAR point cloud. In Proc. Int. Conf. on Computer Vision Workshop. 0–0.
37. Shihao Wu, Wei Sun, Pinxin Long, Hui Huang, Daniel Cohen-Or, Minglun Gong, Oliver Deussen, and Baoquan Chen. 2014. Quality-driven Poisson-guided autoscanning. ACM Trans. on Graphics (Proc. SIGGRAPH Asia) 33 (2014), 203:1–203:12. Issue 6.
38. Alexander Zelinsky, Ray A Jarvis, JC Byrne, Shinichi Yuta, et al. 1993. Planning paths of complete coverage of an unstructured environment by a mobile robot. In Proc. Int. Conf. on Advanced Robotics, Vol. 13. 533–538.
39. Xujie Zhang, Pengcheng Zhao, Qingwu Hu, Mingyao Ai, Datian Hu, and Jiayuan Li. 2020. A UAV-based panoramic oblique photogrammetry (POP) approach using spherical projection. ISPRS J. Photogrammetry and Remote Sensing 159 (2020), 198–219.
40. Xiaohui Zhou, Ke Xie, Kai Huang, Yilin Liu, Yang Zhou, Minglun Gong, and Hui Huang. 2020. Offsite Aerial Path Planning for Efficient Urban Scene Reconstruction. ACM Trans. on Graphics (Proc. SIGGRAPH Asia) 39, 6 (2020), 192:1–192:16.
