“Continuous aerial path planning for 3D urban scene reconstruction” by Zhang, Yao, Xie, Fu, Zhang, et al. …
Conference:
Type(s):
Title:
- Continuous aerial path planning for 3D urban scene reconstruction
Session/Category Title: Scene Synthesis and Navigation
Presenter(s)/Author(s):
Abstract:
We introduce the first path-oriented drone trajectory planning algorithm, which performs continuous (i.e., dense) image acquisition along an aerial path and explicitly factors path quality into an optimization along with scene reconstruction quality. Specifically, our method takes as input a rough 3D scene proxy and produces a drone trajectory and image capturing setup, which efficiently yields a high-quality reconstruction of the 3D scene based on three optimization objectives: one to maximize the amount of 3D scene information that can be acquired along the entirety of the trajectory, another to optimize the scene capturing efficiency by maximizing the scene information that can be acquired per unit length along the aerial path, and the last one to minimize the total turning angles along the aerial path, so as to reduce the number of sharp turns. Our search scheme is based on the rapidly-exploring random tree framework, resulting in a final trajectory as a single path through the search tree. Unlike state-of-the-art works, our joint optimization for view selection and path planning is performed in a single step. We comprehensively evaluate our method not only on benchmark virtual datasets as in existing works but also on several large-scale real urban scenes. We demonstrate that the continuous paths optimized by our method can effectively reduce onsite acquisition cost using drones, while achieving high-fidelity 3D reconstruction, compared to existing planning methods and oblique photography, a mature and popular industry solution.
References:
1. Fotios Balampanis, Ivan Maza, and Anibal Ollero. 2017. Spiral-like coverage path planning for multiple heterogeneous UAS operating in coastal regions. In Proc. of Int. Conf. on Unmanned Aircraft Systems. 617–624.
2. Christian Beder and Richard Steffen. 2006. Determining an Initial Image Pair for Fixing the Scale of a 3D Reconstruction from an Image Sequence. In Pattern Recognition. 657–666.
3. Andreas Bircher, Mina Kamel, Kostas Alexis, Helen Oleynikova, and Roland Siegwart. 2016. Receding horizon “next-best-view” planner for 3D exploration. In Proc. of IEEE Int. Conf. on Robotics & Automation. 1462–1468.
4. Yasutaka Furukawa, Brian Curless, Steven Seitz, and Richard Szeliski. 2010. Towards Internet-scale Multi-view Stereo. In Proc. of IEEE Conf. on Computer Vision & Pattern Recognition. 1434–1441.
5. Yasutaka Furukawa and Carlos Hernández. 2015. Multi-View Stereo: A Tutorial. Foundations and Trends in Computer Graphics and Vision 9, 1–2 (2015), 1–148.
6. J. D. Gammell, S. S. Srinivasa, and T. D. Barfoot. 2014. Informed RRT*: Optimal sampling-based path planning focused via direct sampling of an admissible ellipsoidal heuristic. In Proc. of IEEE Int. Conf. on Intelligent Robots & Systems. 2997–3004.
7. Lionel Heng, Alkis Gotovos, Andreas Krause, and Marc Pollefeys. 2015. Efficient visual exploration and coverage with a micro aerial vehicle in unknown environments. In 2015 IEEE International Conference on Robotics and Automation (ICRA). 1071–1078.
8. Benjamin Hepp, Debadeepta Dey, Sudipta N. Sinha, Ashish Kapoor, Neel Joshi, and Otmar Hilliges. 2018a. Learn-to-score: Efficient 3D scene exploration by predicting view utility. In Proceedings of the European conference on computer vision (ECCV). 437–452.
9. Benjamin Hepp, Matthias Nießner, and Otmar Hilliges. 2018b. Plan3D: Viewpoint and Trajectory Optimization for Aerial Multi-View Stereo Reconstruction. ACM Trans. on Graphics 38, 1 (2018), 4:1–4:17.
10. A. Hornung, B. Zeng, and L. Kobbelt. 2008. Image Selection for Improved Multi-View Stereo. In Proc. of IEEE Conf. on Computer Vision & Pattern Recognition. 1–8.
11. Chong Huang, Fei Gao, Jie Pan, Zhenyu Yang, Weihao Qiu, Peng Chen, Xin Yang, Shaojie Shen, and Kwang-Ting Tim Cheng. 2018a. ACT: An Autonomous Drone Cinematography System for Action Scenes. In Proc. of IEEE Int. Conf. on Robotics & Automation. 7039–7046.
12. Rui Huang, Danping Zou, Richard Vaughan, and Ping Tan. 2018b. Active Image-based Modeling with a Toy Drone. In Proc. of IEEE Int. Conf. on Robotics & Automation. 1–8.
13. Mustafa Devrim Kaba, Mustafa Gokhan Uzunbas, and Ser Nam Lim. 2017. A reinforcement learning approach to the view planning problem. In Proc. of IEEE Conf. on Computer Vision & Pattern Recognition. 5094–5102.
14. Sertac Karaman and Emilio Frazzoli. 2011. Sampling-based algorithms for optimal motion planning. The international journal of robotics research 30, 7 (2011), 846–894.
15. Arno Knapitsch, Jaesik Park, Qian-Yi Zhou, and Vladlen Koltun. 2017. Tanks and Temples: Benchmarking Large-scale Scene Reconstruction. ACM Trans. on Graphics (Proc. of SIGGRAPH) 36, 4 (2017), 78:1–78:13.
16. Tobias Koch, Marco Körner, and Friedrich Fraundorfer. 2019. Automatic and semantically-aware 3D UAV flight planning for image-based 3D reconstruction. Remote Sensing 11, 13 (2019), 1550.
17. Andreas Krause and Daniel Golovin. 2014. Submodular Function Maximization. In Tractability: Practical Approaches to Hard Problems. Cambridge University Press, 71–104.
18. Q. Kuang, J. Wu, J. Pan, and B. Zhou. 2020. Real-Time UAV Path Planning for Autonomous Urban Scene Reconstruction. In Proc. of IEEE Int. Conf. on Robotics & Automation. 1156–1162.
19. Steven M. LaValle. 1998. Rapidly-exploring random trees: A new tool for path planning. The Annual Research Report (1998), 1–4.
20. Yilin Liu, Ke Xie, and Hui Huang. 2021. VGF-Net: Visual-Geometric Fusion Learning for Simultaneous Drone Navigation and Height Mapping. Graphical Models 116 (2021), 101108.
21. Massimo Mauro, Hayko Riemenschneider, Alberto Signoroni, Riccardo Leonardi, and Luc Van Gool. 2014. A Unified Framework for Content-aware View Selection and Planning Through View Importance. In Proc. of British Machine Vision Conference. 1–11.
22. J. Maver and R. Bajcsy. 1993. Occlusions as a guide for planning the next view. IEEE Transactions on Pattern Analysis and Machine Intelligence 15, 5 (1993), 417–433.
23. Oscar Mendez Maldonado, Simon Hadfield, Nicolas Pugeault, and Richard Bowden. 2016. Next-best Stereo: Extending Next Best View Optimization for Collaborative Sensors. In Proc. of British Machine Vision Conference. 1–12.
24. Pierre Moreels and Pietro Perona. 2007. Evaluation of features detectors and descriptors based on 3D objects. Int. J. of Computer Vision. 73, 3 (2007), 263–284.
25. Christian Mostegel, Markus Rumpler, Friedrich Fraundorfer, and Horst Bischof. 2016. Uav-based autonomous image acquisition with multi-view stereo quality assurance by confidence prediction. In Proc. of IEEE Conf. on Computer Vision & Pattern Recognition. 1–10.
26. Kourosh Naderi, Joose Rajamäki, and Perttu Hämäläinen. 2015. RT-RRT* a real-time path planning algorithm based on RRT. In Proceedings of the 8th ACM SIGGRAPH Conference on Motion in Games. Association for Computing Machinery, New York, NY, USA, 113–118.
27. Lasse Damtoft Nielsen, Inkyung Sung, and Peter Nielsen. 2019. Convex decomposition for a coverage path planning for autonomous vehicles: Interior extension of edges. Sensors 19, 19 (2019), 4165.
28. Daryl Peralta, Joel Casimiro, Aldrin Michael Nilles, Justine Aletta Aguilar, Rowel Atienza, and Rhandley Cajote. 2020. Next-best view policy for 3D reconstruction. In Proc. of Euro. Conf. on Computer Vision. 558–573.
29. Mike Roberts, Debadeepta Dey, Anh Truong, Sudipta Sinha, Shital Shah, Ashish Kapoor, Pat Hanrahan, and Neel Joshi. 2017. Submodular Trajectory Optimization for Aerial 3D Scanning. In Proc. of Int. Conf. on Computer Vision. 5324–5333.
30. Mike Roberts and Pat Hanrahan. 2016. Generating Dynamically Feasible Trajectories for Quadrotor Cameras. ACM Trans. on Graphics (Proc. of SIGGRAPH) 35, 4 (2016), 61:1–61:11.
31. Markus Rumpler, Arnold Irschara, and Horst Bischof. 2011. Multi-view stereo: Redundancy benefits for 3D reconstruction. In Proc. of Workshop of the Austrian Association for Pattern Recognition. 1–8.
32. Korbinian Schmid, Heiko Hirschmüller, Andreas Dömel, Iris Grixa, Michael Suppa, and Gerd Hirzinger. 2012. View planning for multi-view stereo 3D reconstruction using an autonomous multicopter. J. of Intelligent & Robotic Systems 65, 1-4 (2012), 309–323.
33. Lukas Schmid, Michael Pantic, Raghav Khanna, Lionel Ott, Roland Siegwart, and Juan Nieto. 2020. An Efficient Sampling-based Method for Online Informative Path Planning in Unknown Environments. IEEE Robotics and Automation Letters 5, 2 (2020), 1500–1507.
34. Steven M. Seitz, Brian Curless, James Diebel, Daniel Scharstein, and Richard Szeliski. 2006. A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms. In Proc. of IEEE Conf. on Computer Vision & Pattern Recognition. 519–528.
35. Magnus Selin, Mattias Tiger, Daniel Duberg, Fredrik Heintz, and Patric Jensfelt. 2019. Efficient autonomous exploration planning of large-scale 3-D environments. IEEE Robotics and Automation Letters 4, 2 (2019), 1699–1706.
36. Ruben M. Smelik, Tim Tutenel, Rafael Bidarra, and Bedrich Benes. 2014. A Survey on Procedural Modelling for Virtual Worlds. Computer Graphics Forum 33, 6 (2014), 31–55.
37. Neil Smith, Nils Moehrle, Michael Goesele, and Wolfgang Heidrich. 2018. Aerial Path Planning for Urban Scene Reconstruction: A Continuous Optimization Method and Benchmark. ACM Trans. on Graphics (Proc. of SIGGRAPH Asia) 37, 6 (2018), 183:1–183:15.
38. Noah Snavely, Steven M. Seitz, and Richard Szeliski. 2006. Photo Tourism: Exploring Photo Collections in 3D. ACM Trans. on Graphics (Proc. of SIGGRAPH) 25, 3 (2006), 835–846.
39. Noah Snavely, Steven M. Seitz, and Richard Szeliski. 2008. Modeling the World from Internet Photo Collections. Int. J. of Computer Vision. 80 (2008), 189–210.
40. P. Whaite and F. P. Ferrie. 1997. Autonomous exploration: driven by uncertainty. IEEE Transactions on Pattern Analysis and Machine Intelligence 19, 3 (1997), 193–205.
41. Christian Witting, Marius Fehr, Rik Bähnemann, Helen Oleynikova, and Roland Siegwart. 2018. History-aware autonomous exploration in confined environments using MAVs. In Proc. of IEEE Int. Conf. on Intelligent Robots & Systems. 1–9.
42. Ke Xie, Hao Yang, Shengqiu Huang, Dani Lischinski, Marc Christie, Kai Xu, Minglun Gong, Daniel Cohen-Or, and Hui Huang. 2018. Creating and Chaining Camera Moves for Quadrotor Videography. ACM Trans. on Graphics (Proc. of SIGGRAPH) 37, 4 (2018), 88:1–88:13.
43. Hao Yang, Ke Xie, Shengqiu Huang, and Hui Huang. 2018. Uncut Aerial Video via a Single Sketch. Computer Graphics Forum (Proc. of Pacific Conf. on Computer Graphics & Applications) 37, 7 (2018), 191–199.
44. Xiaohui Zhou, Ke Xie, Kai Huang, Yilin Liu, Yang Zhou, Minglun Gong, and Hui Huang. 2020. Offsite Aerial Path Planning for Efficient Urban Scene Reconstruction. ACM Trans. on Graphics (Proc. of SIGGRAPH Asia) 39, 6 (2020), 192:1–192:16.
