“Free-form scanning of non-planar appearance with neural trace photography” by Ma, Kang, Zhu, Wu and Zhou

  • ©Xiaohe Ma, Kaizhang Kang, Ruisheng Zhu, Hongzhi Wu, and Kun Zhou




    Free-form scanning of non-planar appearance with neural trace photography



    We propose neural trace photography, a novel framework to automatically learn high-quality scanning of non-planar, complex anisotropic appearance. Our key insight is that free-form appearance scanning can be cast as a geometry learning problem on unstructured point clouds, each of which represents an image measurement and the corresponding acquisition condition. Based on this connection, we carefully design a neural network, to jointly optimize the lighting conditions to be used in acquisition, as well as the spatially independent reconstruction of reflectance from corresponding measurements. Our framework is not tied to a specific setup, and can adapt to various factors in a data-driven manner. We demonstrate the effectiveness of our framework on a number of physical objects with a wide variation in appearance. The objects are captured with a light-weight mobile device, consisting of a single camera and an RGB LED array. We also generalize the framework to other common types of light sources, including a point, a linear and an area light.


    1. Miika Aittala, Timo Aila, and Jaakko Lehtinen. 2016. Reflectance Modeling by Neural Texture Synthesis. ACM Trans. Graph. 35, 4, Article 65 (July 2016), 13 pages.Google ScholarDigital Library
    2. Miika Aittala, Tim Weyrich, and Jaakko Lehtinen. 2013. Practical SVBRDF Capture in the Frequency Domain. ACM Trans. Graph. 32, 4, Article 110 (July 2013), 12 pages.Google ScholarDigital Library
    3. Miika Aittala, Tim Weyrich, and Jaakko Lehtinen. 2015. Two-shot SVBRDF Capture for Stationary Materials. ACM Trans. Graph. 34, 4, Article 110 (July 2015), 13 pages.Google ScholarDigital Library
    4. Artec. 2021. Space Spider Portable 3D Scanner. Retrieved January, 2021 from https://www.artec3d.com/portable-3d-scanners/artec-spiderGoogle Scholar
    5. Sai Bi, Zexiang Xu, Kalyan Sunkavalli, David Kriegman, and Ravi Ramamoorthi. 2020. Deep 3D Capture: Geometry and Reflectance from Sparse Multi-View Images. In CVPR. 5960–5969.Google Scholar
    6. Guojun Chen, Yue Dong, Pieter Peers, Jiawan Zhang, and Xin Tong. 2014. Reflectance Scanning: Estimating Shading Frame and BRDF with Generalized Linear Light Sources. ACM Trans. Graph. 33, 4, Article 117 (July 2014), 11 pages.Google ScholarDigital Library
    7. Frederique Crete, Thierry Dolmiere, Patricia Ladret, and Marina Nicolas. 2007. The blur effect: perception and estimation with a new no-reference perceptual blur metric. In Human Vision and Electronic Imaging XII, Vol. 6492. 196 — 206.Google Scholar
    8. Kristin J. Dana, Bram van Ginneken, Shree K. Nayar, and Jan J. Koenderink. 1999. Reflectance and Texture of Real-world Surfaces. ACM Trans. Graph. 18, 1 (Jan. 1999), 1–34.Google ScholarDigital Library
    9. Valentin Deschaintre, Miika Aittala, Fredo Durand, George Drettakis, and Adrien Bousseau. 2018. Single-image SVBRDF Capture with a Rendering-aware Deep Network. ACM Trans. Graph. 37, 4, Article 128 (July 2018), 15 pages.Google ScholarDigital Library
    10. Valentin Deschaintre, Miika Aittala, Frédo Durand, George Drettakis, and Adrien Bousseau. 2019. Flexible SVBRDF Capture with a Multi-Image Deep Network. In CGF, Vol. 38. 1–13.Google ScholarCross Ref
    11. Yue Dong. 2019. Deep appearance modeling: A survey. Visual Informatics 3, 2 (2019), 59–68.Google ScholarCross Ref
    12. Yue Dong, Guojun Chen, Pieter Peers, Jiawan Zhang, and Xin Tong. 2014. Appearance-from-motion: Recovering Spatially Varying Surface Reflectance Under Unknown Lighting. ACM Trans. Graph. 33, 6, Article 193 (Nov. 2014), 12 pages.Google ScholarDigital Library
    13. Yue Dong, Jiaping Wang, Xin Tong, John Snyder, Yanxiang Lan, Moshe Ben-Ezra, and Baining Guo. 2010. Manifold Bootstrapping for SVBRDF Capture. ACM Trans. Graph. 29, 4, Article 98 (July 2010), 10 pages.Google ScholarDigital Library
    14. Mark Fiala. 2005. ARTag, a fiducial marker system using digital techniques. In CVPR.Google Scholar
    15. Duan Gao, Guojun Chen, Yue Dong, Pieter Peers, Kun Xu, and Xin Tong. 2020. Deferred Neural Lighting: Free-Viewpoint Relighting from Unstructured Photographs. ACM Trans. Graph. 39, 6, Article 258 (Nov. 2020), 15 pages.Google ScholarDigital Library
    16. Duan Gao, Xiao Li, Yue Dong, Pieter Peers, Kun Xu, and Xin Tong. 2019. Deep Inverse Rendering for High-resolution SVBRDF Estimation from an Arbitrary Number of Images. ACM Trans. Graph. 38, 4, Article 134 (July 2019), 15 pages.Google ScholarDigital Library
    17. Andrew Gardner, Chris Tchou, Tim Hawkins, and Paul Debevec. 2003. Linear light source reflectometry. ACM Trans. Graph. 22, 3 (2003), 749–758.Google ScholarDigital Library
    18. Abhijeet Ghosh, Tongbo Chen, Pieter Peers, Cyrus A. Wilson, and Paul Debevec. 2009. Estimating Specular Roughness and Anisotropy from Second Order Spherical Gradient Illumination. CGF 28, 4 (2009), 1161–1170.Google ScholarDigital Library
    19. Darya Guarnera, Giuseppe C. Guarnera, Abhijeet Ghosh, Cornelia Denk, and Mashhuda Glencross. 2016. BRDF Representation and Acquisition. Computer Graphics Forum 35, 2 (2016), 625–650.Google ScholarCross Ref
    20. Yu Guo, Cameron Smith, Miloš Hašan, Kalyan Sunkavalli, and Shuang Zhao. 2020. MaterialGAN: reflectance capture using a generative SVBRDF model. ACM Trans. Graph. 39, 6 (2020), 1–13.Google ScholarDigital Library
    21. Michael Holroyd, Jason Lawrence, and Todd Zickler. 2010. A Coaxial Optical Scanner for Synchronous Acquisition of 3D Geometry and Surface Reflectance. ACM Trans. Graph. 29, 4, Article 99 (July 2010), 12 pages.Google ScholarDigital Library
    22. Zhuo Hui, Kalyan Sunkavalli, Joon-Young Lee, Sunil Hadap, Jian Wang, and Aswin C. Sankaranarayanan. 2017. Reflectance Capture Using Univariate Sampling of BRDFs. In ICCV.Google Scholar
    23. Kaizhang Kang, Zimin Chen, Jiaping Wang, Kun Zhou, and Hongzhi Wu. 2018. Efficient Reflectance Capture Using an Autoencoder. ACM Trans. Graph. 37, 4, Article 127 (July 2018), 10 pages.Google ScholarDigital Library
    24. Kaizhang Kang, Cihui Xie, Chengan He, Mingqi Yi, Minyi Gu, Zimin Chen, Kun Zhou, and Hongzhi Wu. 2019. Learning Efficient Illumination Multiplexing for Joint Capture of Reflectance and Shape. ACM Trans. Graph. 38, 6, Article 165 (Nov. 2019), 12 pages.Google ScholarDigital Library
    25. Jason Lawrence, Aner Ben-Artzi, Christopher DeCoro, Wojciech Matusik, Hanspeter Pfister, Ravi Ramamoorthi, and Szymon Rusinkiewicz. 2006. Inverse Shade Trees for Non-parametric Material Representation and Editing. ACM Trans. Graph. 25, 3 (July 2006), 735–745.Google ScholarDigital Library
    26. Hendrik P. A. Lensch, Jan Kautz, Michael Goesele, Wolfgang Heidrich, and Hans-Peter Seidel. 2003. Image-based Reconstruction of Spatial Appearance and Geometric Detail. ACM Trans. Graph. 22, 2 (April 2003), 234–257.Google ScholarDigital Library
    27. Xiao Li, Yue Dong, Pieter Peers, and Xin Tong. 2017. Modeling Surface Appearance from a Single Photograph Using Self-augmented Convolutional Neural Networks. ACM Trans. Graph. 36, 4, Article 45 (July 2017), 11 pages.Google ScholarDigital Library
    28. N. J. W. Morris and K. N. Kutulakos. 2007. Reconstructing the Surface of Inhomogeneous Transparent Scenes by Scatter-Trace Photography. In ICCV.Google Scholar
    29. A. Myronenko and X. Song. 2010. Point Set Registration: Coherent Point Drift. IEEE PAMI 32, 12 (2010), 2262–2275.Google ScholarDigital Library
    30. Giljoo Nam, Joo Ho Lee, Diego Gutierrez, and Min H Kim. 2018. Practical SVBRDF acquisition of 3D objects with unstructured flash photography. In SIGGRAPH Asia Technical Papers. 267.Google Scholar
    31. Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. 2017. Pointnet: Deep learning on point sets for 3d classification and segmentation. In CVPR. 652–660.Google Scholar
    32. Peiran Ren, Jiaping Wang, John Snyder, Xin Tong, and Baining Guo. 2011. Pocket reflectometry. ACM Trans. Graph. 30, 4 (2011), 1–10.Google ScholarDigital Library
    33. Jérémy Riviere, Pieter Peers, and Abhijeet Ghosh. 2016. Mobile surface reflectometry. In CGF, Vol. 35. 191–202.Google ScholarDigital Library
    34. Johannes Lutz Schönberger, Enliang Zheng, Marc Pollefeys, and Jan-Michael Frahm. 2016. Pixelwise View Selection for Unstructured Multi-View Stereo. In ECCV.Google Scholar
    35. Shining3D. 2021. EinScan Pro 2X Plus Handheld Industrial Scanner. Retrieved January, 2021 from https://www.einscan.com/handheld-3d-scanner/2x-plus/Google Scholar
    36. Borom Tunwattanapong, Graham Fyffe, Paul Graham, Jay Busch, Xueming Yu, Abhijeet Ghosh, and Paul Debevec. 2013. Acquiring Reflectance and Shape from Continuous Spherical Harmonic Illumination. ACM Trans. Graph. 32, 4, Article 109 (July 2013), 12 pages.Google ScholarDigital Library
    37. Bruce Walter, Stephen R. Marschner, Hongsong Li, and Kenneth E. Torrance. 2007. Microfacet Models for Refraction through Rough Surfaces. In Rendering Techniques (Proc. EGWR).Google ScholarDigital Library
    38. Michael Weinmann and Reinhard Klein. 2015. Advances in Geometry and Reflectance Acquisition. In SIGGRAPH Asia Courses. Article 1, 71 pages.Google Scholar
    39. Tim Weyrich, Jason Lawrence, Hendrik P. A. Lensch, Szymon Rusinkiewicz, and Todd Zickler. 2009. Principles of Appearance Acquisition and Representation. Found. Trends. Comput. Graph. Vis. 4, 2 (2009), 75–191.Google ScholarDigital Library
    40. Hongzhi Wu, Zhaotian Wang, and Kun Zhou. 2016a. Simultaneous Localization and Appearance Estimation with a Consumer RGB-D Camera. IEEE TVCG 22, 8 (Aug 2016), 2012–2023.Google Scholar
    41. Hongzhi Wu and Kun Zhou. 2015. AppFusion: Interactive Appearance Acquisition Using a Kinect Sensor. CGF 34, 6 (2015), 289–298.Google ScholarDigital Library
    42. Zhe Wu, Sai-Kit Yeung, and Ping Tan. 2016b. Towards Building an RGBD-M Scanner. CoRR abs/1603.03875 (2016).Google Scholar
    43. Jianzhao Zhang, Guojun Chen, Yue Dong, Jian Shi, Bob Zhang, and Enhua Wu. 2020. Deep Inverse Rendering for Practical Object Appearance Scan with Uncalibrated Illumination. In Advances in Computer Graphics. 71–82.Google Scholar
    44. Zhiming Zhou, Guojun Chen, Yue Dong, David Wipf, Yong Yu, John Snyder, and Xin Tong. 2016. Sparse-as-Possible SVBRDF Acquisition. ACM Trans. Graph. 35, 6, Article Article 189 (Nov. 2016), 12 pages.Google ScholarDigital Library
    45. Todd Zickler, Sebastian Enrique, Ravi Ramamoorthi, and Peter Belhumeur. 2005. Reflectance Sharing: Image-based Rendering from a Sparse Set of Images. In Proc. EGSR. 253–264.Google ScholarDigital Library

ACM Digital Library Publication:

Overview Page: