“Aerial path planning for urban scene reconstruction: a continuous optimization method and benchmark”
Conference:
Type(s):
Title:
- Aerial path planning for urban scene reconstruction: a continuous optimization method and benchmark
Session/Category Title: Aerial propagation
Presenter(s)/Author(s):
Moderator(s):
Abstract:
Small unmanned aerial vehicles (UAVs) are ideal capturing devices for high-resolution urban 3D reconstructions using multi-view stereo. Nevertheless, practical considerations such as safety usually mean that access to the scan target is often only available for a short amount of time, especially in urban environments. It therefore becomes crucial to perform both view and path planning to minimize flight time while ensuring complete and accurate reconstructions.In this work, we address the challenge of automatic view and path planning for UAV-based aerial imaging with the goal of urban reconstruction from multi-view stereo. To this end, we develop a novel continuous optimization approach using heuristics for multi-view stereo reconstruction quality and apply it to the problem of path planning. Even for large scan areas, our method generates paths in only a few minutes, and is therefore ideally suited for deployment in the field.To evaluate our method, we introduce and describe a detailed benchmark dataset for UAV path planning in urban environments which can also be used to evaluate future research efforts on this topic. Using this dataset and both synthetic and real data, we demonstrate survey-grade urban reconstructions with ground resolutions of 1 cm or better on large areas (30 000 m2).
References:
1. Henrik Aanæs, Rasmus Ramsbøl Jensen, George Vogiatzis, Engin Tola, and Anders Bjorholm Dahl. 2016. Large-Scale Data for Multiple-View Stereopsis. IJCV 120, 2 (01 Nov. 2016), 153–168. Google ScholarDigital Library
2. Bashar Alsadik, Markus Gerke, and George Vosselman. 2013. Automated Camera Network Design for 3D Modeling of Cultural Heritage Objects. J. Cultural Heritage 14, 6 (2013).Google ScholarCross Ref
3. Peter W. Battaglia, Jessica B. Hamrick, and Joshua B. Tenenbaum. 2013. Simulation as an engine of physical scene understanding. Proc. National Academy of Sciences 110, 45 (2013), 18327–18332.Google ScholarCross Ref
4. Christian Beder and Richard Steffen. 2006. Determining an Initial Image Pair for Fixing the Scale of a 3D Reconstruction from an Image Sequence. In Proc. Joint Pattern Recognition Symposium. Google ScholarDigital Library
5. P. J. Besl and N. D. McKay. 1992. A Method for Registration of 3-D Shapes. IEEE Trans. PAMI 14, 2 (1992), 239–256. Google ScholarDigital Library
6. Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. 2017. ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes. In Proc. CVPR.Google ScholarCross Ref
7. Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. 2017. CARLA: An Open Urban Driving Simulator. In Proc. Ann. Conf. Robot Learning. 1–16.Google Scholar
8. Enrique Dunn and Jan-Michael Frahm. 2009. Next Best View Planning for Active Model Improvement. In Proc. BMVC.Google ScholarCross Ref
9. Xinyi Fan, Linguang Zhang, Benedict Brown, and Szymon Rusinkiewicz. 2016. Automated View and Path Planning for Scalable Multi-Object 3D Scanning. ACM Trans. Graph. 35, 6 (Nov. 2016). Google ScholarDigital Library
10. Christian Forster, Matia Pizzoli, and Davide Scaramuzza. 2014. Appearance-Based Active, Monocular, Dense Reconstruction for Micro Aerial Vehicles. In Robotics: Science and Systems.Google Scholar
11. Simon Fuhrmann and Michael Goesele. 2014. Floating Scale Surface Reconstruction. ACM Trans. Graph. 33, 4 (2014). Google ScholarDigital Library
12. Simon Fuhrmann, Fabian Langguth, Nils Moehrle, Michael Waechter, and Michael Goesele. 2015. MVE – An Image-Based Reconstruction Environment. Computers & Graphics 53 (2015). Google ScholarDigital Library
13. Yasutaka Furukawa, Brian Curless, Steven M. Seitz, and Richard Szeliski. 2010. Towards Internet-Scale Multi-View Stereo. In Proc. CVPR.Google ScholarCross Ref
14. Adrien Gaidon, Qiao Wang, Yohann Cabon, and Eleonora Vig. 2016. Virtual worlds as proxy for multi-object tracking analysis. In Proc. CVPR. 4340–4349.Google Scholar
15. David Gallup, Jan-Michael Frahm, Philippos Mordohai, and Marc Pollefeys. 2008. Variable Baseline/Resolution Stereo. In Proc. CVPR.Google ScholarCross Ref
16. Quentin Galvane, Christophe Lino, Marc Christie, Julien Fleureau, Fabien Servant, François-Louis Tariolle, and Philippe Guillotel. 2017. Directing Cinematographic Drones. CoRR abs/1712.04216 (2017). arXiv:1712.04216 http://arxiv.org/abs/1712.04216Google Scholar
17. Christoph Gebhardt, Stefan Stevsic, and Otmar Hilliges. 2018. Optimizing for Aesthetically Pleasing Quadrotor Camera Motion. ACM Trans. Graph. 37, 4, Article 90 (Aug. 2018), 11 pages. Google ScholarDigital Library
18. Michael Goesele, Noah Snavely, Brian Curless, Hugues Hoppe, and Steven M. Seitz. 2007. Multi-View Stereo for Community Photo Collections. In Proc. ICCV.Google Scholar
19. A. Handa, T. Whelan, J. McDonald, and A. J. Davison. 2014. A benchmark for RGB-D visual odometry, 3D reconstruction and SLAM. In 2014 IEEE International Conference on Robotics and Automation (ICRA). 1524–1531.Google Scholar
20. Benjamin Hepp, Matthias Niessner, and Otmar Hilliges. 2017. Plan3D: Viewpoint and Trajectory Optimization for Aerial Multi-View Stereo Reconstruction. CoRR abs/1705.09314 (2017).Google Scholar
21. Christof Hoppe, Andreas Wendel, Stefanie Zollmann, Katrin Pirker, Arnold Irschara, Horst Bischof, and Stefan Kluckner. 2012. Photogrammetric Camera Network Design for Micro Aerial Vehicles. In Proc. Computer Vision Winter Workshop, Vol. 8.Google Scholar
22. Rui Huang, Danping Zou, Richard Vaughan, and Ping Tan. 2017. Active Image-based Modeling. CoRR abs/1705.01010 (2017). arXiv:1705.01010 http://arxiv.org/abs/1705.01010Google Scholar
23. Brian Karis and Epic Games. 2013. Real Shading in Unreal Engine 4. (2013).Google Scholar
24. Arno Knapitsch, Jaesik Park, Qian-Yi Zhou, and Vladlen Koltun. 2017. Tanks and Temples: Benchmarking Large-Scale Scene Reconstruction. ACM Trans. Graph. 36, 4 (2017). Google ScholarDigital Library
25. Fabian Langguth, Kalyan Sunkavalli, Sunil Hadap, and Michael Goesele. 2016. Shading-Aware Multi-View Stereo. In Proc. ECCV.Google ScholarCross Ref
26. John J Leonard and Hugh F Durrant-Whyte. 1991. Simultaneous Map Building and Localization for an Autonomous Mobile Robot. In Proc. IROS.Google ScholarCross Ref
27. Ligang Liu, Xi Xia, Han Sun, Qi Shen, Junzhan Xu, Bin Chen, Hui Huang, and Kai Xu. 2018. Object-Aware Guidance for Autonomous Scene Reconstruction. ACM Trans. Graph. 37, 4 (2018), 104:1–104:12. Google ScholarDigital Library
28. O. Mendez, Simon Hadfield, Nicolas Pugeault, and Richard Bowden. 2016. Next-Best Stereo: Extending Next-Best View Optimisation for Collaborative Sensors. In Proc. BMVC.Google Scholar
29. P. Merrell, P. Mordohai, J. M. Frahm, and M. Pollefeys. 2007. Evaluation of Large Scale Scene Reconstruction. In Proc. ICCV. 1–8.Google Scholar
30. Pierre Moreels and Pietro Perona. 2007. Evaluation of Features Detectors and Descriptors Based on 3D Objects. IJCV 73, 3 (2007). Google ScholarDigital Library
31. Christian Mostegel, Markus Rumpler, Friedrich Fraundorfer, and Horst Bischof. 2016. UAV-Based Autonomous Image Acquisition with Multi-View Stereo Quality Assurance by Confidence Prediction. In Proc. CVPR Workshops.Google ScholarCross Ref
32. Matthias Mueller, Neil Smith, and Bernard Ghanem. 2016. A Benchmark and Simulator for UAV Tracking. In Proc. ECCV. 445–461.Google ScholarCross Ref
33. Gustavo Olague and Roger Mohr. 2002. Optimal Camera Placement for Accurate Reconstruction. Pattern Recognition 35, 4 (2002).Google Scholar
34. Molly A O’Neil, Dan Tamir, and Martin Burtscher. 2011. A Parallel GPU Version of the Traveling Salesman Problem. In Proc. PDPTA.Google Scholar
35. Jeremie Papon and Markus Schoeler. 2015. Semantic Pose using Deep Networks Trained on Synthetic RGB-D. CoRR abs/1508.00835 (2015). http://arxiv.org/abs/1508.00835 Google ScholarDigital Library
36. Ken Perlin. 2002. Improving Noise. ACM Trans. Graph. 21, 3 (2002). Google ScholarDigital Library
37. Norbert Pfeifer, Philipp Glira, and Christian Briese. 2012. Direct Georeferencing with on Board Navigation Components of Light Weight UAV Platforms. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences (2012).Google ScholarCross Ref
38. Weichao Qiu, Fangwei Zhong, Yi Zhang, Siyuan Qiao, Zihao Xiao, Tae Soo Kim, Yizhou Wang, and Alan Yuille. 2017. UnrealCV: Virtual Worlds for Computer Vision. ACM Multimedia Open Source Software Competition (2017). Google ScholarDigital Library
39. Stephan R. Richter, Zeeshan Hayder, and Vladlen Koltun. 2017. Playing for Benchmarks. In Proc. ICCV.Google ScholarCross Ref
40. Mike Roberts, Debadeepta Dey, Anh Truong, Sudipta Sinha, Shital Shah, Ashish Kapoor, Pat Hanrahan, and Neel Joshi. 2017. Submodular Trajectory Optimization for Aerial 3D Scanning. In Proc. ICCV.Google ScholarCross Ref
41. Mike Roberts and Pat Hanrahan. 2016. Generating Dynamically Feasible Trajectories for Quadrotor Cameras. ACM Trans. Graph. 35, 4 (2016). Google ScholarDigital Library
42. German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio Lopez. 2016. The SYNTHIA Dataset: A Large Collection of Synthetic Images for Semantic Segmentation of Urban Scenes. In Proc. CVPR.Google ScholarCross Ref
43. Markus Rumpler, Arnold Irschara, and Horst Bischof. 2011. Multi-View Stereo: Redundancy Benefits for 3D Reconstruction. In Proc. Workshop of Austrian Assoc. for Patt. Recog., Vol. 4.Google Scholar
44. Markus Rumpler, Alexander Tscharf, Christian Mostegel, Shreyansh Daftry, Christof Hoppe, Rudolf Prettenthaler, Friedrich Fraundorfer, Gerhard Mayer, and Horst Bischof. 2016. Evaluations on Multi-Scale Camera Networks for Precise and Geo-Accurate Reconstructions from Aerial and Terrestrial Images with User Guidance. Computer Vision & Image Understanding (2016). Google ScholarDigital Library
45. Korbinian Schmid, Heiko Hirschmüller, Andreas Dömel, Iris Grixa, Michael Suppa, and Gerd Hirzinger. 2012. View Planning for Multi-View Stereo 3D Reconstruction using an Autonomous Multicopter. J. Inteligent & Robotic Sys. 65, 1–4 (2012). Google ScholarDigital Library
46. Johannes Lutz Schönberger and Jan-Michael Frahm. 2016. Structure-from-Motion Revisited. In Conference on Computer Vision and Pattern Recognition (CVPR).Google Scholar
47. Thomas Schöps, Johannes L. Schönberger, Silvano Galliani, Torsten Sattler, Konrad Schindler, Marc Pollefeys, and Andreas Geiger. 2017. A Multi-View Stereo Benchmark with High-Resolution Images and Multi-Camera Videos. In Proc. CVPR.Google ScholarCross Ref
48. Steven M. Seitz, Brian Curless, James Diebel, Daniel Scharstein, and Richard Szeliski. 2006. A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms. In Proc. CVPR, Vol. 1. Google ScholarDigital Library
49. Ben Semerjian. 2014. A New Variational Framework for Multiview Surface Reconstruction. In Proc. ECCV.Google ScholarCross Ref
50. Shital Shah, Debadeepta Dey, Chris Lovett, and Ashish Kapoor. 2017. AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles. In Proc. Field and Service Robotics.Google Scholar
51. Ching-Kuang Shene. 1998. Curve Global Interpolation. (1998). https://www.cs.mtu.edu/~shene/COURSES/cs3621/NOTES/INT-APP/CURVE-INT-global.htmlGoogle Scholar
52. Amarjeet Singh, Andreas Krause, and William J. Kaiser. 2009. Nonmyopic Adaptive Informative Path Planning for Multiple Robots. In IJCAI. Google ScholarDigital Library
53. Noah Snavely, Steven M Seitz, and Richard Szeliski. 2006. Photo Tourism: Exploring Photo Collections in 3D. ACM Trans. Graph. 25, 3 (2006). Google ScholarDigital Library
54. Noah Snavely, Steven M. Seitz, and Richard Szeliski. 2008. Skeletal Graphs for Efficient Structure from Motion. In Proc. CVPR, Vol. 1.Google ScholarCross Ref
55. Christoph Strecha, Wolfgang von Hansen, Luc van Gool, Pascal Fua, and Ulrich Thoennessen. 2008. On Benchmarking Camera Calibration and Multi-View Stereo for High Resolution Imagery. In Proc. CVPR.Google ScholarCross Ref
56. W. Treible, P. Saponaro, S. Sorensen, A. Kolagunda, M. O’Neal, B. Phelan, K. Sherbondy, and C. Kambhamettu. 2017. CATS: A Color and Thermal Stereo Benchmark. In Proc. CVPR. 134–142.Google Scholar
57. J. Irving Vasquez-Gomez, L. Enrique Sucar, Rafael Murrieta-Cid, and Efrain Lopez-Damian. 2014. Volumetric Next-Best-View Planning for 3D Object Reconstruction with Positioning Error. Int. J. Adv. Robotic Sys. 11 (2014).Google ScholarCross Ref
58. Michael Waechter, Mate Beljan, Simon Fuhrmann, Nils Moehrle, Johannes Kopf, and Michael Goesele. 2017. Virtual Rephotography: Novel View Prediction Error for 3D Reconstruction. ACM Trans. Graph. 36, 1, Article 8 (Jan. 2017), 11 pages. Google ScholarDigital Library
59. Ingo Wald. 2007. On Fast Construction of SAH-Based Bounding Volume Hierarchies. In Proc. Symp. Interactive Ray Tracing. Google ScholarDigital Library
60. Andreas Wendel, Michael Maurer, Gottfried Graber, Thomas Pock, and Horst Bischof. 2012. Dense Reconstruction on-the-fly. In Proc. CVPR. Google ScholarDigital Library
61. Stefan Wenhardt, Benjamin Deutsch, Elli Angelopoulou, and Heinrich Niemann. 2007. Active Visual Object Reconstruction using D-, E-, and T-Optimal Next Best Views. In Proc. CVPR.Google ScholarCross Ref
62. Shihao Wu, Wei Sun, Pinxin Long, Hui Huang, Daniel Cohen-Or, Minglun Gong, Oliver Deussen, and Baoquan Chen. 2014. Quality-driven Poisson-guided Autoscanning. ACM Trans. Graph. 33, 6, Article 203 (Nov. 2014), 12 pages. Google ScholarDigital Library
63. Ke Xie, Hao Yang, Shengqiu Huang, Dani Lischinski, Marc Christie, Kai Xu, Minglun Gong, Daniel Cohen-Or, and Hui Huang. 2018. Creating and Chaining Camera Moves for Quadrotor Videography. ACM Trans. Graph. 37, 4 (2018), 88:1–88:13. Google ScholarDigital Library
