“Appearance Modeling via Proxy-to-Image Alignment” by Huang, Xie, Ma, Lischinski, Gong, et al. …
Conference:
Type(s):
Title:
- Appearance Modeling via Proxy-to-Image Alignment
Session/Category Title: Layers, Glints and Surface Microstructure
Presenter(s)/Author(s):
Abstract:
Endowing 3D objects with realistic surface appearance is a challenging and time-demanding task, as real-world surfaces typically exhibit a plethora of spatially variant geometric and photometric detail. Not surprisingly, computer artists commonly use images of real-world objects as an inspiration and a reference for their digital creations. However, despite two decades of research on image-based modeling, there are still no tools available for automatically extracting the detailed appearance (microgeometry and texture) of a 3D surface from a single image. In this article, we present a novel user-assisted approach for quickly and easily extracting a nonparametric appearance model from a single photograph of a reference object.
The extraction process requires a user-provided proxy, whose geometry roughly approximates that of the object in the image. Since the proxy is just a rough approximation, it is necessary to align and deform it so as to match the reference object. The main contribution of this work is a novel technique to perform such an alignment, which enables accurate joint recovery of geometric detail and reflectance. The correlations between the recovered geometry at various scales and the spatially varying reflectance constitute a nonparametric appearance model. Once extracted, the appearance model may then be applied to various 3D shapes, whose large-scale geometry may differ considerably from that of the original reference object. Thus, our approach makes it possible to construct an appearance library, allowing users to easily enrich detail-less 3D shapes with realistic geometric detail and surface texture.
References:
- Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B. Goldman. 2009. PatchMatch: A randomized correspondence algorithm for structural image editing. ACM Transactions on Graphics 28, 3, 24:1–24:11.
- Jonathan T. Barron and Jitendra Malik. 2015. Shape, illumination, and reflectance from shading. IEEE Transactions on Pattern Analysis and Machine Intelligence 37, 8, 1670–1687.
- Nicolas Bonneel, Michiel Van de Panne, Sylvain Lefebvre, and George Drettakis. 2010. Proxy-guided texture synthesis for rendering natural scenes. In Proceedings of the Conference on Vision, Modeling, and Visualization.
- Adrien Bousseau, Sylvain Paris, and Frédo Durand. 2009. User-assisted intrinsic images. ACM Transactions on Graphics 28, 5, 130:1–130:10.
- Tao Chen, Zhe Zhu, Ariel Shamir, Shi-Min Hu, and Daniel Cohen-Or. 2013. 3-sweep: Extracting editable objects from a single photo. ACM Transactions on Graphics 32, 6, 195:1–195:10.
- Soheil Darabi, Eli Shechtman, Connelly Barnes, Dan B. Goldman, and Pradeep Sen. 2012. Image melding: Combining inconsistent images using patch-based synthesis. ACM Transactions on Graphics 31, 4, 82–91.
- Paul E. Debevec, Camillo J. Taylor, and Jitendra Malik. 1996. Modeling and rendering architecture from photographs: A hybrid geometry-and image-based approach. In Proceedings of the 1996 SIGGRAPH Conference. 11–20.
- Olga Diamanti, Connelly Barnes, Sylvain Paris, Eli Shechtman, and Olga Sorkine-Hornung. 2015. Synthesis of complex image appearance from limited exemplars. ACM Transactions on Graphics 34, 2, 22:1–22:14.
- J.-M. Dischler, K. Maritaud, and D. Ghazanfarpour. 2002. Coherent bump map recovery from a single texture image. In Proceedings of the Canadian Conference on Graphics Interface. 201–208.
- Piotr Dollár and C. Lawrence Zitnick. 2013. Structured forests for fast edge detection. In Proceedings of the International Conference on Computer Vision. 1841–1848.
- Yue Dong, Xin Tong, Fabio Pellacini, and Baining Guo. 2011. AppGen: Interactive material modeling from a single image. ACM Transactions on Graphics 30, 6, 146:1–146:10.
- Hui Fang and John C. Hart. 2004. Textureshop: Texture synthesis as a photograph editing tool. ACM Transactions on Graphics 23, 3, 354–359.
- Berthold Horn. 1986. Robot Vision. MIT Press, Cambridge, NY.
- Qixing Huang, Hai Wang, and Vladlen Koltun. 2015. Single-view reconstruction via joint analysis of image and shape collections. ACM Transactions on Graphics 34, 4, 87:1–87:10.
- Thouis R. Jones, Frédo Durand, and Mathieu Desbrun. 2003. Non-iterative, feature-preserving mesh smoothing. ACM Transactions on Graphics 22, 3, 943–949.
- Alexandre Kaspar, Boris Neubert, Dani Lischinski, Mark Pauly, and Johannes Kopf. 2015. Self tuning texture optimization. Computer Graphics Forum 34, 2, 349–359.
- Erum Arif Khan, Erik Reinhard, Roland W. Fleming, and Heinrich H. Bülthoff. 2006. Image-based material editing. ACM Transactions on Graphics 25, 3, 654–663.
- Natasha Kholgade, Tomas Simon, Alexei Efros, and Yaser Sheikh. 2014. 3D object manipulation in a single photograph using stock 3D models. ACM Transactions on Graphics 33, 4, 127:1–127:12.
- Vladislav Kraevoy, Alla Sheffer, and Michiel van de Panne. 2009. Modeling from contour drawings. In Proceedings of the Eurographics Symposium on Sketch-Based Interfaces and Modeling. 37–44.
- Vivek Kwatra, Irfan Essa, Aaron Bobick, and Nipun Kwatra. 2005. Texture optimization for example-based synthesis. ACM Transactions on Graphics 24, 3, 795–802.
- Tom Mertens, Jan Kautz, Jiawen Chen, Philippe Bekaert, and Frédo Durand. 2006. Texture transfer using geometry correlation. In Proceedings of the Eurographics Symposium on Rendering, Vol. 273. 273–284.
- Mark Meyer, Mathieu Desbrun, Peter Schröder, and Alan H. Barr. 2003. Discrete differential-geometry operators for triangulated 2-manifolds. In Visualization and Mathematics III. Springer, 35–57.
- Jorge Nocedal and Stephen J. Wright. 2006. Numerical Optimization (2nd ed.). Springer-Verlag, New York, NY.
- Byong Mok Oh, Max Chen, Julie Dorsey, and Frédo Durand. 2001. Image-based modeling and photo editing. In Proceedings of the 2001 SIGGRAPH Conference. 433–442.
- Manuel M. Oliveira. 2002. Image-based modeling and rendering techniques: A survey. RITA—Revista de Informática Teórica e Aplicada 9, 2, 37–66.
- Lawrence R. Rabiner. 1989. A tutorial on hidden Markov models and selected applications in speech recognition. Proceedings of the IEEE 77, 257–286.
- Konstantinos Rematas, Chuong Nguyen, Mario Fritz, and Tinne Tuytelaars. 2017. Novel views of objects from a single image. IEEE Transactions on Pattern Analysis and Machine Intelligence 39, 8, 1576–1590.
- Olga Sorkine and Marc Alexa. 2007. As-rigid-as-possible surface modeling. Computer Graphics Forum 4, 109–116.
- Hao Su, Qixing Huang, Niloy J. Mitra, Yangyan Li, and Leonidas Guibas. 2014. Estimating image depth using shape collections. ACM Transactions on Graphics 33, 4, 37:1–37:11.
- Tuanfeng Y. Wang, Hao Su, Qixing Huang, Jingwei Huang, Leonidas Guibas, and Niloy J. Mitra. 2016. Unsupervised texture transfer from images to model collections. ACM Trans.actions on Graphics 35, 6, 177:1–177:13.
- Xi Wang, Lifeng Wang, Ligang Liu, Shimin Hu, and Baining Guo. 2003. Interactive modeling of tree bark. In Proceedings of the Pacific Conference on Computer Graphics and Applications.83–90.
- Li-Yi Wei, Sylvain Lefebvre, Vivek Kwatra, and Greg Turk. 2009. State of the art in example-based texture synthesis. In Eurographics State of the Art (EG-STAR) Reports. 93–117.
- Chenglei Wu, Michael Zollhöfer, Matthias Nießner, Marc Stamminger, Shahram Izadi, and Christian Theobalt. 2014. Real-time shading-based refinement for consumer depth cameras. ACM Transactions on Graphics 33, 6, 200:1–200:10.
- Hongzhi Wu, Zhaotian Wang, and Kun Zhou. 2016. Simultaneous localization and appearance estimation with a consumer RGB-D camera. IEEE Transactions on Visualization and Computer Graphics 22, 8, 2012–2023.
- Kai Xu, Hao Zhang, Andrea Tagliasacchi, Ligang Liu, Guo Li, Min Meng, and Yueshan Xiong. 2009. Partial intrinsic reflectional symmetry of 3D shapes. ACM Transactions on Graphics 28, 5, 138:1–138:10.
- Kai Xu, Hanlin Zheng, Hao Zhang, Daniel Cohen-Or, Ligang Liu, and Yueshan Xiong. 2011. Photo-inspired model-driven 3D object modeling. ACM Transactions on Graphics 30, 4, 80:1–80:10.
- Su Xue, Jiaping Wang, Xin Tong, Qionghai Dai, and Baining Guo. 2008. Image-based material weathering. Computer Graphics Forum 27, 2, 617–626.
- Lap-Fai Yu, Sai-Kit Yeung, Yu-Wing Tai, and Stephen Lin. 2013. Shading-based shape refinement of RGB-D images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1415–1422.
- Steve Zelinka, Hui Fang, Michael Garland, and John C. Hart. 2005. Interactive material replacement in photographs. In Proceedings of the Canadian Conference on Graphics Interface. 227–232.
- Youyi Zheng, Xiang Chen, Ming-Ming Cheng, Kun Zhou, Shi-Min Hu, and Niloy J. Mitra. 2012. Interactive images: Cuboid proxies for smart image manipulation. ACM Transactions on Graphics 31, 4, 99:1–99:11.
- Qian-Yi Zhou and Vladlen Koltun. 2014. Color map optimization for 3D reconstruction with consumer depth cameras. ACM Transactions on Graphics 33, 4, 155:1–155:10.