“Real-time planning for automated multi-view drone cinematography” by Naegeli, Meier, Domahidi, Mora and Hilliges
Conference:
Type:
Title:
- Real-time planning for automated multi-view drone cinematography
Session/Category Title: Video
Presenter(s)/Author(s):
Moderator(s):
Abstract:
We propose a method for automated aerial videography in dynamic and cluttered environments. An online receding horizon optimization formulation facilitates the planning process for novices and experts alike. The algorithm takes high-level plans as input, which we dub virtual rails, alongside interactively defined aesthetic framing objectives and jointly solves for 3D quadcopter motion plans and associated velocities. The method generates control inputs subject to constraints of a non-linear quadrotor model and dynamic constraints imposed by actors moving in an a priori unknown way. The output plans are physically feasible, for the horizon length, and we apply the resulting control inputs directly at each time-step, without requiring a separate trajectory tracking algorithm. The online nature of the method enables incorporation of feedback into the planning and control loop, makes the algorithm robust to disturbances. Furthermore, we extend the method to include coordination between multiple drones to enable dynamic multi-view shots, typical for action sequences and live TV coverage. The algorithm runs in real-time on standard hardware and computes motion plans for several drones in the order of milliseconds. Finally, we evaluate the approach qualitatively with a number of challenging shots, involving multiple drones and actors and qualitatively characterize the computational performance experimentally.
References:
1. 2015. 3DR Solo. (2015). http://3drobotics.com/solo.Google Scholar
2. 2015. APM Autopilot Suite. (2015). http://ardupilot.com.Google Scholar
3. 2015. DJI Ground Station. (2015). http://www.dji.com/product/pc-ground-station.Google Scholar
4. 2015. Parrot SDK. (2015). http://developer.parrot.com/.Google Scholar
5. 2015. VC Technology Litchi Tool. (2015). https://flylitchi.com/.Google Scholar
6. A Pedro Aguiar, João P Hespanha, and Petar V Kokotović. 2008. Performance limitations in reference tracking and path following for nonlinear systems. Automatica 44, 3 (2008), 598–610.Google ScholarDigital Library
7. Javier Alonso-Mora, Tobias Naegeli, Roland Siegwart, and Paul Beardsley. 2015. Collision Avoidance for Aerial Vehicles in Multi-agent Scenarios. Auton. Robot. (Jan. 2015).Google Scholar
8. John T Betts. 2010. Practical methods for optimal control and estimation using nonlinear programming. Vol. 19. Siam.Google Scholar
9. Adam Bry, Charles Richter, Abraham Bachrach, and Nicholas Roy. 2015. Aggressive flight of fixed-wing and quadrotor aircraft in dense indoor environments. The International Journal of Robotics Research 34, 7 (2015), 969–1002. Google ScholarDigital Library
10. Yaobin Chen, Stanley Y-P Chien, and ALAN A DESROCHERS. 1992. General structure of time-optimal control of robotic manipulators moving along prescribed paths. Internat. J. Control 56, 4 (1992), 767–782. Google ScholarCross Ref
11. Marc Christie, Éric Languénou, and Laurent Granvilliers. 2002. Modeling Camera Control with Constrained Hypertubes. Springer Berlin Heidelberg, Berlin, Heidelberg, 618–632. Google ScholarCross Ref
12. Marc Christie, Patrick Olivier, and Jean-Marie Normand. 2008. Camera Control in Computer Graphics. Computer Graphics Forum 27, 8 (Dec. 2008), 2197–2218. Google ScholarCross Ref
13. Daniel Arijon. 1976. Grammar of the film language. https://scholar.google.ch/citations?viewGoogle Scholar
14. Alexander Domahidi and Juan Jerez. 2016. FORCES Pro: code generation for embedded optimization. (September 2016). https://www.embotech.com/FORCES-Pro.Google Scholar
15. Alexander Domahidi, Aldo U Zgraggen, Melanie N Zeilinger, Manfred Morari, and Colin N Jones. 2012. Efficient interior point methods for multistage problems arising in receding horizon control. In Decision and Control, 2008. CDC 2008. 47th IEEE Conference on. IEEE, 668–674.Google ScholarCross Ref
16. Steven M. Drucker and David Zeltzer. 1994. Intelligent Camera Control in a Virtual Environment. In In Proceedings of Graphics Interface ’94. 190–199.Google Scholar
17. Timm Faulwasser, Benjamin Kern, and Rolf Findeisen. 2009. Model predictive path-following for constrained nonlinear systems. In Decision and Control, 2009 held jointly with the 2009 28th Chinese Control Conference. CDC/CCC 2009. Proceedings of the 48th IEEE Conference on. IEEE, 8642–8647. Google ScholarCross Ref
18. Quentin Galvane, Julien Fleureau, Francois-Louis Tariolle, and Philippe Guillotel. 2016. Automated Cinematography with Unmanned Aerial Vehicles. In Eurographics Workshop on Intelligent Cinematography and Editing. The Eurographics Association.Google Scholar
19. Christoph Gebhardt, Benjamin Hepp, Tobias Naegeli, Stefan Stevsic, and Otmar Hilliges. 2016. Airways: Optimization-based Planning of Quadrotor Trajectories according to High-Level User Goals. In SIGCHI Conference on Human Factors in Computing Systems (CHI ’16). ACM, New York, NY, USA. Google ScholarDigital Library
20. T. Geijtenbeek and N. Pronost. 2012. Interactive Character Animation Using Simulated Physics: A State-of-the-Art Review. Comput. Graph. Forum 31, 8 (Dec. 2012), 2492–2515. Google ScholarDigital Library
21. Michael Gleicher and Andrew Witkin. 1992. Through-the-lens camera control. In ACM SIGGRAPH Computer Graphics, Vol. 26. ACM, 331–340. Google ScholarDigital Library
22. Niels Joubert, L. E. Jane, Dan B. Goldman, Floraine Berthouzoz, Mike Roberts, James A. Landay, and Pat Hanrahan. 2016. Towards a Drone Cinematographer: Guiding Quadrotor Cameras using Visual Composition Principles. CoRR abs/1610.01691 (2016). http://arxiv.org/abs/1610.01691Google Scholar
23. Niels Joubert, Mike Roberts, Anh Truong, Floraine Berthouzoz, and Pat Hanrahan. 2015. An Interactive Tool for Designing Quadrotor Camera Shots. ACM Transactions on Graphics (SIGGRAPH Asia 2015) (2015).Google Scholar
24. Sertac Karaman and Emilio Frazzoli. 2011. Sampling-based algorithms for optimal motion planning. International Journal of Robotics Research 30, 7 (June 2011), 846–894. Google ScholarDigital Library
25. Eric C Kerrigan and Jan M Maciejowski. 2000. Soft constraints and exact penalty functions in model predictive control. In Control 2000 Conference, Cambridge.Google Scholar
26. Denise Lam, Chris Manzie, and Malcolm Good. 2010. Model predictive contouring control. In 49th IEEE Conference on Decision and Control (CDC). IEEE, 6137–6142. Google ScholarCross Ref
27. Alexander Liniger, Alexander Domahidi, and Manfred Morari. 2015. Optimization-based autonomous racing of 1:43 scale RC cars. Optimal Control Applications and Methods 36, 5 (2015), 628–647. Google ScholarCross Ref
28. C Lino and M Christie. 2015. Intuitive and efficient camera control with the toric space. ACM Transactions on Graphics (TOG) (2015).Google Scholar
29. Christophe Lino, Marc Christie, Roberto Ranon, and William Bares. 2011. The Director’s Lens: An Intelligent Assistant for Virtual Cinematography. In Proceedings of the 19th ACM International Conference on Multimedia (MM ’11). ACM, New York, NY, USA, 323–332. Google ScholarDigital Library
30. Brian MacAllister, Jonathan Butzke, Alex Kushleyev, Harsh Pandey, and Maxim Likhachev. 2013. Path planning for non-circular micro aerial vehicles in constrained environments. In IEEE International Conference on Robotics and Automation (ICRA). IEEE, 3933–3940. Google ScholarCross Ref
31. Daniel Mellinger and Vijay Kumar. 2011. Minimum snap trajectory generation and control for quadrotors. In IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2520–2525. Google ScholarCross Ref
32. Mark W Mueller and Raffaello D’Andrea. 2013. A model predictive controller for quadrocopter state interception. In Control Conference (ECC), 2013 European. IEEE, 1383–1389.Google ScholarCross Ref
33. Tobias Nägeli, Javier Alonso-Mora, Alexander Domahidi, Daniela Rus, and Otmar Hilliges. 2017. Real-time Motion Planning for Aerial Videography with Dynamic Obstacle Avoidance and Viewpoint Optimization. IEEE Robotics and Automation Letters (2017).Google Scholar
34. Morgan Quigley, Ken Conley, Brian P. Gerkey, Josh Faust, Tully Foote, Jeremy Leibs, Rob Wheeler, and Andrew Y. Ng. 2009. ROS: an open-source Robot Operating System. In IEEE ICRA Workshop on Open Source Software.Google Scholar
35. Mike Roberts and Pat Hanrahan. 2016. Generating Dynamically Feasible Trajectories for Quadrotor Cameras. ACM Transactions on Graphics (Proc. SIGGRAPH 2016) 35, 4 (2016).Google Scholar
36. Charles Rose, Brian Guenter, Bobby Bodenheimer, and Michael F Cohen. 1996. Efficient generation of motion transitions using spacetime constraints. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques. ACM, 147–154. Google ScholarDigital Library
37. Jeffery B Saunders, Brandon Call, Andrew Curtis, Randal W Beard, and Timothy W McLain. 2005. Static and dynamic obstacle avoidance in miniature air vehicles. AIAA [email protected] Aerospace 96 (2005).Google Scholar
38. Andrew Witkin and Michael Kass. 1988. Spacetime constraints. ACM Siggraph Computer Graphics 22, 4 (1988), 159–168. Google ScholarDigital Library
