“Virtual Rephotography: Novel View Prediction Error for 3D Reconstruction” by Waechter, Beljian, Fuhrmann, Moehrle, Kopf, et al. …
Conference:
Type(s):
Title:
- Virtual Rephotography: Novel View Prediction Error for 3D Reconstruction
Session/Category Title: Get More Out of Your Photo
Presenter(s)/Author(s):
Moderator(s):
Abstract:
The ultimate goal of many image-based modeling systems is to render photo-realistic novel views of a scene without visible artifacts. Existing evaluation metrics and benchmarks focus mainly on the geometric accuracy of the reconstructed model, which is, however, a poor predictor of visual accuracy. Furthermore, using only geometric accuracy by itself does not allow evaluating systems that either lack a geometric scene representation or utilize coarse proxy geometry. Examples include a light field and most image-based rendering systems. We propose a unified evaluation approach based on novel view prediction error that is able to analyze the visual quality of any method that can render novel views from input images. One key advantage of this approach is that it does not require ground truth geometry. This dramatically simplifies the creation of test datasets and benchmarks. It also allows us to evaluate the quality of an unknown scene during the acquisition and reconstruction process, which is useful for acquisition planning. We evaluate our approach on a range of methods, including standard geometry-plus-texture pipelines as well as image-based rendering techniques, compare it to existing geometry-based benchmarks, demonstrate its utility for a range of use cases, and present a new virtual rephotography-based benchmark for image-based modeling and rendering systems.
References:
1. Henrik Aanæs, Rasmus Ramsbøl Jensen, George Vogiatzis, Engin Tola, and Anders Bjorholm Dahl. 2016. Large-scale data for multiple-view stereopsis. IJCV 120, 2, 153–168. Google ScholarDigital Library
2. Tunç Ozan Aydın, Rafał Mantiuk, Karol Myszkowski, and Hans-Peter Seidel. 2008. Dynamic range independent image quality assessment. In SIGGRAPH. Google ScholarDigital Library
3. Soonmin Bae, Aseem Agarwala, and Frédo Durand. 2010. Computational rephotography. ACM Transactions on Graphics 29, 3, 24:1–24:15.Google ScholarDigital Library
4. Simon Baker, Daniel Scharstein, J. P. Lewis, Stefan Roth, Michael J. Black, and Richard Szeliski. 2011. A database and evaluation methodology for optical flow. IJCV 92, 1, 1–31.Google ScholarDigital Library
5. Kai Berger, Christian Lipski, Christian Linz, Anita Sellent, and Marcus Magnor. 2010. A ghosting artifact detector for interpolated image quality assessment. In International Symposium on Consumer Electronics. Google ScholarCross Ref
6. Chris Buehler, Michael Bosse, Leonard McMillan, Steven Gortler, and Michael F. Cohen. 2001. Unstructured Lumigraph rendering. In SIGGRAPH. Google ScholarDigital Library
7. Fatih Calakli, Ali O. Ulusoy, Maria I. Restrepo, Gabriel Taubin, and Joseph L. Mundy. 2012. High resolution surface reconstruction from multi-view aerial imagery. In 3DIMPVT.Google Scholar
8. Neill D. Campbell, George Vogiatzis, Carlos Hernández, and Roberto Cipolla. 2008. Using multiple hypotheses to improve depth-maps for multi-view stereo. In ECCV. Google ScholarDigital Library
9. Scott Daly. 1993. The visible differences predictor: An algorithm for the assessment of image fidelity. In Digital Images and Human Vision.Google Scholar
10. Andrew Fitzgibbon, Yonatan Wexler, and Andrew Zisserman. 2005. Image-based rendering using image-based priors. IJCV 63, 2, 141–151. Google ScholarDigital Library
11. Wolfgang Förstner. 1996. 10 pros and cons against performance characterization of vision algorithms. In Workshop on Performance Characteristics of Vision Algorithms.Google Scholar
12. Simon Fuhrmann and Michael Goesele. 2011. Fusion of depth maps with multiple scales. In SIGGRAPH Asia. Google ScholarDigital Library
13. Simon Fuhrmann, Fabian Langguth, Nils Moehrle, Michael Waechter, and Michael Goesele. 2015. MVE — An image-based reconstruction environment. Computers 8 Graphics 53, Part A (2015).Google Scholar
14. Yasutaka Furukawa, Brian Curless, Steven M. Seitz, and Richard Szeliski. 2010. Towards internet-scale multi-view stereo. In CVPR. Google ScholarCross Ref
15. Yasutaka Furukawa and Jean Ponce. 2010. Accurate, dense, and robust multi-view stereopsis. PAMI 32, 8, 1362–1376. Google ScholarDigital Library
16. Michael Goesele, Noah Snavely, Brian Curless, Hugues Hoppe, and Steven M. Seitz. 2007. Multi-view stereo for community photo collections. In ICCV. Google ScholarCross Ref
17. Steven J. Gortler, Radek Grzeszczuk, Richard Szeliski, and Michael F. Cohen. 1996. The Lumigraph. In SIGGRAPH. Google ScholarDigital Library
18. Stefan Guthe, Douglas Cunningham, Pascal Schardt, and Michael Goesele. 2016. Ghosting and popping detection for image-based rendering. In 3DTV Conference. Google ScholarCross Ref
19. Tom Haber, Christian Fuchs, Philippe Bekaert, Hans-Peter Seidel, Michael Goesele, and Hendrik P. A. Lensch. 2009. Relighting objects from image collections. In CVPR. Google ScholarCross Ref
20. Christian Hofsetz, Kim Ng, George Chen, Peter McGuinness, Nelson Max, and Yang Liu. 2004. Image-based rendering of range data with estimated depth uncertainty. Computer Graphics and Applications 24, 4, 34–42. Google ScholarDigital Library
21. Yuan Hongxing, Guo Li, Yu Li, and Cheng Long. 2010. Multi-view reconstruction using band graph-cuts. Journal of Computer-Aided Design 8 Computer Graphics 4 (2010).Google Scholar
22. Christof Hoppe, Manfred Klopschitz, Markus Rumpler, Andreas Wendel, Stefan Kluckner, Horst Bischof, and Gerhard Reitmayr. 2012a. Online feedback for structure-from-motion image acquisition. In BMVC. Google ScholarCross Ref
23. Christof Hoppe, Andreas Wendel, Stefanie Zollmann, Katrin Pirker, Arnold Irschara, Horst Bischof, and Stefan Kluckner. 2012b. Photogrammetric camera network design for micro aerial vehicles. In Computer Vision Winter Workshop.Google Scholar
24. Michael Kazhdan, Matthew Bolitho, and Hugues Hoppe. 2006. Poisson surface reconstruction. In SGP.Google Scholar
25. Johannes Kopf, Michael F. Cohen, and Richard Szeliski. 2014. First-person hyper-lapse videos. In SIGGRAPH. Google ScholarDigital Library
26. Yvan G. Leclerc, Quang-Tuan Luong, and Pascal Fua. 2000. Measuring the self-consistency of stereo algorithms. In ECCV. Google ScholarCross Ref
27. Marc Levoy and Pat Hanrahan. 1996. Light field rendering. In SIGGRAPH. Google ScholarDigital Library
28. Rafał Mantiuk. 2013. Quantifying image quality in graphics: Perspective on subjective and objective metrics and their performance. In SPIE, Vol. 8651.Google ScholarCross Ref
29. Rafał Mantiuk, Kil Joong Kim, Allan G. Rempel, and Wolfgang Heidrich. 2011. HDR-VDP-2: A calibrated visual metric for visibility and quality predictions in all luminance conditions. In SIGGRAPH.Google Scholar
30. Paul Merrell, Amir Akbarzadeh, Liang Wang, Philippos Mordohai, Jan-Michael Frahm, Ruigang Yang, David Nistér, and Marc Pollefeys. 2007. Real-time visibility-based fusion of depth maps. In ICCV. Google ScholarCross Ref
31. Ken Perlin. 2002. Improving noise. In SIGGRAPH. Google ScholarDigital Library
32. Jens Preiss, Felipe Fernandes, and Philipp Urban. 2014. Color-image quality assessment: From prediction to optimization. IEEE Transactions on Image Processing 23, 3, 1366–1378. Google ScholarDigital Library
33. Ganesh Ramanarayanan, James Ferwerda, Bruce Walter, and Kavita Bala. 2007. Visual equivalence: Towards a new standard for image fidelity. In SIGGRAPH.Google Scholar
34. Michael Schwarz and Marc Stamminger. 2009. On predicting visual popping in dynamic scenes. In Applied Perception in Graphics and Visualization. Google ScholarDigital Library
35. Steven M. Seitz, Brian Curless, James Diebel, Daniel Scharstein, and Richard Szeliski. 2006. A comparison and evaluation of multi-view stereo reconstruction algorithms. In CVPR. Google ScholarDigital Library
36. Qi Shan, Riley Adams, Brian Curless, Yasutaka Furukawa, and Steven M. Seitz. 2013. The visual Turing test for scene reconstruction. In 3DV.Google Scholar
37. Noah Snavely, Steven M. Seitz, and Richard Szeliski. 2006. Photo tourism: Exploring photo collections in 3D. In SIGGRAPH.Google Scholar
38. Christoph Strecha, Wolfgang von Hansen, Luc Van Gool, Pascal Fua, and Ulrich Thoennessen. 2008. On benchmarking camera calibration and multi-view stereo for high resolution imagery. In CVPR. Google ScholarCross Ref
39. Richard Szeliski. 1999. Prediction error as a quality metric for motion and stereo. In ICCV. Google ScholarCross Ref
40. James Tompkin, Min H. Kim, Kwang In Kim, Jan Kautz, and Christian Theobalt. 2013. Preference and artifact analysis for video transitions of places. ACM Transactions on Applied Perception 10, 3, 13:1–13:19.Google ScholarCross Ref
41. Kathleen Tuite, Noah Snavely, Dun-yu Hsiao, Nadine Tabing, and Zoran Popović. 2011. PhotoCity: Training experts at large-scale image acquisition through a competitive game. In SIGCHI.Google Scholar
42. Peter Vangorp, Gaurav Chaurasia, Pierre-Yves Laffont, Roland Fleming, and George Drettakis. 2011. Perception of visual artifacts in image-based rendering of façades. In Eurographics Symposium on Rendering. Google ScholarDigital Library
43. Peter Vangorp, Christian Richardt, Emily A. Cooper, Gaurav Chaurasia, Martin S. Banks, and George Drettakis. 2013. Perception of perspective distortions in image-based rendering. In SIGGRAPH. Google ScholarDigital Library
44. Kenneth Vanhoey, Basile Sauvage, Pierre Kraemer, Frédéric Larue, and Jean-Michel Dischler. 2015. Simplification of meshes with digitized radiance. The Visual Computer 31, 6, 1011–1021. Google ScholarDigital Library
45. Michael Waechter, Nils Moehrle, and Michael Goesele. 2014. Let there be color! Large-scale texturing of 3D reconstructions. In ECCV.Google Scholar
46. Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. 2004. Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing 13, 4, 600–612. Google ScholarDigital Library
47. Robert H. Webb, Diane E. Boyer, and Raymond M. Turner. 2010. Repeat Photography. Island Press.Google Scholar
48. Yizhou Yu, Paul Debevec, Jitendra Malik, and Tim Hawkins. 1999. Inverse global illumination: Recovering reflectance models of real scenes from photographs. In SIGGRAPH.Google Scholar
49. Ramin Zabih and John Woodfill. 1994. Non-parametric local transforms for computing visual correspondence. In ECCV. Google ScholarCross Ref
50. Matthias Zwicker, Hanspeter Pfister, Jeroen van Baar, and Markus Gross. 2001. Surface splatting. In SIGGRAPH. Google ScholarDigital Library
