“Towards Material Digitization with a Dual-scale Optical System” by Garces, Arellano, Rodriguez-Pardo, Suja and Lopez-Moreno
Conference:
Type:
Title:
- Towards Material Digitization with a Dual-scale Optical System
Session/Category Title: Material Acquisition
Presenter(s)/Author(s):
Moderator(s):
Abstract:
Existing devices for measuring material appearance in spatially-varying samples are limited to a single scale, either micro or mesoscopic. This is a practical limitation when the material has a complex multi-scale structure. In this paper, we present a system and methods to digitize materials at two scales, designed to include high-resolution data in spatially-varying representations at larger scales. We design and build a hemispherical light dome able to digitize flat material samples up to 11x11cm. We estimate geometric properties, anisotropic reflectance and transmittance at the microscopic level using polarized directional lighting with a single orthogonal camera. Then, we propagate this structured information to the mesoscale, using a neural network trained with the data acquired by the device and image-to-image translation methods. To maximize the compatibility of our digitization, we leverage standard BSDF models commonly adopted in the industry. Through extensive experiments, we demonstrate the precision of our device and the quality of our digitization process using a set of challenging real-world material samples and validation scenes. Further, we demonstrate the optical resolution and potential of our device for acquiring more complex material representations by capturing microscopic attributes which affect the global appearance: we characterize the properties of textile materials such as the yarn twist or the shape of individual fly-out fibers. We also release the SEDDIDOME dataset of materials, including raw data captured by the machine and optimized parameteres.
References:
1. 2016. X-RITE: Tac7. https://www.xrite.com/categories/appearance/total-appearance-capture-ecosystem/tac7. Accessed: 2022-04-28.
2. Miika Aittala, Tim Weyrich, Jaakko Lehtinen, et al. 2015. Two-shot SVBRDF capture for stationary materials. ACM Transactions on Graphics (TOG) 34, 4 (2015), 110–1.
3. Raúl Alcain, Carlos Heras, Iñigo Salinas, Jorge López, and Carlos Aliaga. 2019. Microscale Optical Capture System for Digital Fabric Recreation. In Photoptics. 114–119.
4. Carlos Aliaga, Carlos Castillo, Diego Gutierrez, Miguel A Otaduy, Jorge Lopez-Moreno, and Adrian Jarabo. 2017. An appearance model for textile fibers. In Computer Graphics Forum, Vol. 36. Wiley Online Library, 35–45.
5. Michael Ashikhmin and Peter Shirley. 2000. An anisotropic phong brdf model. Journal of Graphics Tools 5, 2 (2000), 25–32.
6. Brent Burley. 2015. Extending the Disney BRDF to a BSDF with integrated subsurface scattering. Physically Based Shading in Theory and Practice SIGGRAPH Course (2015).
7. Brent Burley and Walt Disney Animation Studios. 2012. Physically-based shading at disney. In ACM SIGGRAPH, Vol. 2012. vol. 2012, 1–7.
8. Carlos Castillo, Jorge López-Moreno, and Carlos Aliaga. 2019. Recent Advances in Fabric Appearance Reproduction. Computers & Graphics 84, C (Nov 2019), 103–121.
9. Thomas Chambon, Eric Heitz, and Laurent Belcour. 2021. Passing Multi-Channel Material Textures to a 3-Channel Loss. In ACM SIGGRAPH 2021 Talks. 1–2.
10. Jasmine Collins, Shubham Goel, Kenan Deng, Achleshwar Luthra, Leon Xu, Erhan Gundogdu, Xi Zhang, Tomas F Yago Vicente, Thomas Dideriksen, Himanshu Arora, et al. 2022. Abo: Dataset and benchmarks for real-world 3d object understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 21126–21136.
11. Robert L Cook and Kenneth E Torrance. 1981. A reflectance model for computer graphics. ACM Siggraph Computer Graphics 15, 3 (1981), 307–316.
12. Kristin J Dana. 2001. BRDF/BTF measurement device. In Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, Vol. 2. IEEE, 460–466.
13. Valentin Deschaintre, Miika Aittala, Fredo Durand, George Drettakis, and Adrien Bousseau. 2018. Single-image svbrdf capture with a rendering-aware deep network. ACM Transactions on Graphics (TOG) 37, 4 (2018), 1–15.
14. Valentin Deschaintre, Miika Aittala, Frédo Durand, George Drettakis, and Adrien Bousseau. 2019. Flexible svbrdf capture with a multi-image deep network. In Computer Graphics Forum, Vol. 38. Wiley Online Library, 1–13.
15. Valentin Deschaintre, George Drettakis, and Adrien Bousseau. 2020. Guided Fine-Tuning for Large-Scale Material Transfer. In Computer Graphics Forum, Vol. 39. Wiley Online Library, 91–105.
16. Foivos I Diakogiannis, François Waldner, Peter Caccetta, and Chen Wu. 2020. ResUNet-a: A deep learning framework for semantic segmentation of remotely sensed data. ISPRS Journal of Photogrammetry and Remote Sensing 162 (2020), 94–114.
17. Kirill Dmitriev and Evgeny Makarov. 2011. Generating Displacement from Normal Map for Use in 3D Games. In ACM SIGGRAPH 2011 Talks (Vancouver, British Columbia, Canada) (SIGGRAPH ’11). Association for Computing Machinery, New York, NY, USA, Article 9, 1 pages.
18. Yue Dong. 2019. Deep appearance modeling: A survey. Visual Informatics 3, 2 (2019), 59–68.
19. Yue Dong, Jiaping Wang, Xin Tong, John Snyder, Yanxiang Lan, Moshe Ben-Ezra, and Baining Guo. 2010. Manifold bootstrapping for SVBRDF capture. ACM Transactions on Graphics (TOG) 29, 4 (2010), 1–10.
20. Zhao Dong, Bruce Walter, Steve Marschner, and Donald P Greenberg. 2015. Predicting appearance from measured microgeometry of metal surfaces. ACM Transactions on Graphics (TOG) 35, 1 (2015), 1–13.
21. Julie Dorsey, Holly Rushmeier, and François Sillion. 2010. Digital modeling of material appearance. Elsevier.
22. Jonathan Dupuy and Wenzel Jakob. 2018. An adaptive parameterization for efficient material acquisition and rendering. ACM Transactions on graphics (TOG) 37, 6 (2018), 1–14.
23. Jirí Filip and Radomír Vávra. 2014. Template-based sampling of anisotropic BRDFs. In Computer Graphics Forum, Vol. 33. Wiley Online Library, 91–99.
24. Roland W. Fleming and Heinrich H. Bülthoff. 2005. Low-Level Image Cues in the Perception of Translucent Materials. ACM Transactions on Applied Perception 2, 3 (jul 2005), 346–382.
25. Jeppe Revall Frisvad, Soeren A Jensen, Jonas Skovlund Madsen, António Correia, Li Yang, Søren Kimmer Schou Gregersen, Youri Meuret, and P-E Hansen. 2020. Survey of models for acquiring the optical properties of translucent materials. In Computer Graphics Forum, Vol. 39. Wiley Online Library, 729–755.
26. David Geisler-Moroder and Arne Dür. 2010. A new ward brdf model with bounded albedo. In Computer Graphics Forum, Vol. 29. Wiley Online Library, 1391–1398.
27. Paul Graham, Borom Tunwattanapong, Jay Busch, Xueming Yu, Andrew Jones, Paul Debevec, and Abhijeet Ghosh. 2013. Measurement-based synthesis of facial micro-geometry. In Computer Graphics Forum, Vol. 32. Wiley Online Library, 335–344.
28. Darya Guarnera, Giuseppe Claudio Guarnera, Abhijeet Ghosh, Cornelia Denk, and Mashhuda Glencross. 2016. BRDF representation and acquisition. In Computer Graphics Forum, Vol. 35. Wiley Online Library, 625–650.
29. Giuseppe Claudio Guarnera, Peter Hall, Alain Chesnais, and Mashhuda Glencross. 2017. Woven fabric model creation from a single image. ACM Transactions on Graphics (TOG) 36, 5 (2017), 1–13.
30. MG Guerra, SS Gregersen, Jeppe Revall Frisvad, Leonardo De Chiffre, F Lavecchia, and LM Galantucci. 2019. Measurement of polymers with 3D optical scanners: evaluation of the subsurface scattering effect through five miniature step gauges. Measurement Science and Technology 31, 1 (2019), 015010.
31. Jie Guo, Shuichang Lai, Chengzhi Tao, Yuelong Cai, Lei Wang, Yanwen Guo, and Ling-Qi Yan. 2021. Highlight-aware two-stream network for single-image SVBRDF acquisition. ACM Transactions on Graphics (TOG) 40, 4 (2021), 1–14.
32. Yu Guo, Cameron Smith, Miloš Hašan, Kalyan Sunkavalli, and Shuang Zhao. 2020. MaterialGAN: reflectance capture using a generative SVBRDF model. ACM Transactions on Graphics (TOG) 39, 6 (2020), 1–13.
33. Vlastimil Havran, Jan Hošek, Šárka Němcová, Jiří Čáp, and Jiří Bittner. 2017. Light-drum—Portable light stage for accurate BTF measurement on site. Sensors 17, 3 (2017), 423.
34. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 770–778.
35. Philipp Henzler, Valentin Deschaintre, Niloy J Mitra, and Tobias Ritschel. 2021. Generative Modelling of BRDF Textures from Flash Images. ACM Transactions on Graphics (TOG) 40, 6 (2021).
36. Michael Holroyd, Jason Lawrence, and Todd Zickler. 2010. A coaxial optical scanner for synchronous acquisition of 3D geometry and surface reflectance. ACM Transactions on Graphics (TOG) 29, 4 (2010), 1–12.
37. Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning. PMLR, 448–456.
38. Tomáš Iser, Tobias Rittig, Emilie Nogué, Thomas Klaus Nindel, and Alexander Wilkie. 2022. Affordable Spectral Measurements of Translucent Materials. ACM Transactions on Graphics (TOG) 41, 6, Article 199 (Nov 2022), 13 pages.
39. Wenzel Jakob, Sébastien Speierer, Nicolas Roussel, Merlin Nimier-David, Delio Vicini, Tizian Zeltner, Baptiste Nicolet, Miguel Crespo, Vincent Leroy, and Ziyi Zhang. 2022. Mitsuba 3 renderer. https://mitsuba-renderer.org.
40. Michael Janner, Jiajun Wu, Tejas D Kulkarni, Ilker Yildirim, and Josh Tenenbaum. 2017. Self-supervised intrinsic image decomposition. Advances in Neural Information Processing Systems 30 (2017).
41. Henrik Wann Jensen, Stephen R. Marschner, Marc Levoy, and Pat Hanrahan. 2001. A Practical Model for Subsurface Light Transport (SIGGRAPH ’01). Association for Computing Machinery, New York, NY, USA, 511–518.
42. Jorge Jimenez and Diego Gutierrez. 2010. GPU Pro: Advanced Rendering Techniques. AK Peters Ltd., Chapter Screen-Space Subsurface Scattering, 335–351.
43. Wenhua Jin, Beibei Wang, Miloš Hašan, Yu Guo, Steve Marschner, and Ling-Qi Yan. 2022. Woven Fabric Capture from a Single Photo. In Proceedings of SIGGRAPH Asia 2022.
44. Pramook Khungurn, Daniel Schroeder, Shuang Zhao, Kavita Bala, and Steve Marschner. 2015. Matching Real Fabrics with Micro-Appearance Models. ACM Transactions on Graphics (TOG) 35, 1 (2015), 1–1.
45. Guillaume Lavoué, Nicolas Bonneel, Jean-Philippe Farrugia, and Cyril Soler. 2021. Perceptual quality of BRDF approximations: dataset and metrics. Computer Graphics Forum 40, 2 (2021), 327–338.
46. Bo Li, Jie Feng, and Bingfeng Zhou. 2019. A SVBRDF Modeling Pipeline using Pixel Clustering. https://arxiv.org/abs/1912.00321
47. Zhengqin Li, Kalyan Sunkavalli, and Manmohan Chandraker. 2018. Materials for masses: SVBRDF acquisition with a single mobile phone image. In Proceedings of the European Conference on Computer Vision (ECCV). 72–87.
48. Wan-Chun Ma, Tim Hawkins, Pieter Peers, Charles-Felix Chabert, Malte Weiss, and Paul Debevec. 2007. Rapid acquisition of specular and diffuse normal maps from polarized spherical gradient illumination. In Proceedings of the 18th Eurographics Conference on Rendering Techniques. 183–194.
49. Xiaohe Ma, Kaizhang Kang, Ruisheng Zhu, Hongzhi Wu, and Kun Zhou. 2021. Free-form scanning of non-planar appearance with neural trace photography. ACM Transactions on Graphics (TOG) 40, 4 (2021), 1–13.
50. Stephen R Marschner, Henrik Wann Jensen, Mike Cammarano, Steve Worley, and Pat Hanrahan. 2003. Light scattering from human hair fibers. ACM Transactions on Graphics (TOG) 22, 3 (2003), 780–791.
51. Stephen R Marschner, Stephen H Westin, Adam Arbree, and Jonathan T Moon. 2005. Measuring and modeling the appearance of finished wood. In ACM SIGGRAPH 2005 Papers. 727–734.
52. Rosalie Martin, Arthur Roullier, Romain Rouffet, Adrien Kaiser, and Tamy Boubekeur. 2022. MaterIA: Single Image High-Resolution Material Capture in the Wild. In Computer Graphics Forum, Vol. 41. Wiley Online Library, 163–177.
53. Wojciech Matusik, Hanspeter Pfister, Matt Brand, and Leonard McMillan. 2003a. A data-driven reflectance model. ACM Transactions on Graphics (TOG) 22, 3 (2003), 759–769.
54. Wojciech Matusik, Hanspeter Pfister, Matthew Brand, and Leonard McMillan. 2003b. Efficient isotropic BRDF measurement. In Proceedings of the 14th Eurographics Workshop on Rendering. 241–247.
55. Sebastian Merzbach, Max Hermann, Martin Rump, and Reinhard Klein. 2019. Learned fitting of spatially varying brdfs. In Computer Graphics Forum, Vol. 38. Wiley Online Library, 193–205.
56. Zahra Montazeri, Søren B. Gammelmark, Shuang Zhao, and Henrik Wann Jensen. 2020. A Practical Ply-Based Appearance Model of Woven Fabrics. ACM Trans. Graph. 39, 6, Article 251 (nov 2020), 13 pages.
57. Giljoo Nam, Joo Ho Lee, Hongzhi Wu, Diego Gutierrez, and Min H Kim. 2016. Simultaneous acquisition of microscale reflectance and normals. ACM Transactions on Graphics (TOG) 35, 6 (2016), 185–1.
58. Addy Ngan and Frédo Durand. 2006. Statistical acquisition of texture appearance. In Proceedings of the 17th Eurographics Conference on Rendering Techniques. 31–40.
59. FE Nicodemus, JC Richmond, JJ Hsia, IW Ginsberg, and T Limperis. 1992. Geometrical considerations and nomenclature for reflectance. In Radiometry. 94–145.
60. Jannik Boll Nielsen, Henrik Wann Jensen, and Ravi Ramamoorthi. 2015. On optimal, minimal BRDF sampling for reflectance acquisition. ACM Transactions on Graphics (TOG) 34, 6 (2015), 1–11.
61. Peiran Ren, Jiaping Wang, John Snyder, Xin Tong, and Baining Guo. 2011. Pocket reflectometry. ACM Transactions on Graphics (TOG) 30, 4 (2011), 1–10.
62. Carlos Rodriguez-Pardo and Elena Garces. 2021. Neural Photometry-guided Visual Attribute Transfer. IEEE Transactions on Visualization and Computer Graphics (2021), 1–1.
63. Carlos Rodriguez-Pardo and Elena Garces. 2022. SeamlessGAN: Self-Supervised Synthesis of Tileable Texture Maps. IEEE Transactions on Visualization and Computer Graphics (2022).
64. Roland Ruiters, Christopher Schwartz, and Reinhard Klein. 2013. Example-based Interpolation and Synthesis of Bidirectional Texture Functions. In Computer Graphics Forum, Vol. 32. Wiley Online Library, 361–370.
65. Iman Sadeghi, Oleg Bisker, Joachim De Deken, and Henrik Wann Jensen. 2013. A practical microcylinder appearance model for cloth rendering. ACM Transactions on Graphics (TOG) 32, 2 (2013), 1–12.
66. Christopher Schwartz, Ralf Sarlette, Michael Weinmann, Martin Rump, and Reinhard Klein. 2014. Design and implementation of practical bidirectional texture function measurement devices focusing on the developments at the University of Bonn. Sensors 14, 5 (2014), 7753–7819.
67. Liang Shi, Beichen Li, Miloš Hašan, Kalyan Sunkavalli, Tamy Boubekeur, Radomir Mech, and Wojciech Matusik. 2020. Match: differentiable material graphs for procedural material capture. (2020).
68. Heinz Christian Steinhausen, Dennis den Brok, Matthias B Hullin, and Reinhard Klein. 2014. Acquiring bidirectional texture functions for large-scale material samples. (2014).
69. Kenneth E Torrance and Ephraim M Sparrow. 1967. Theory for off-specular reflection from roughened surfaces. Journal of the Optical Society of America 57, 9 (1967), 1105–1114.
70. Giuseppe Vecchio, Simone Palazzo, and Concetto Spampinato. 2021. SurfaceNet: Adversarial SVBRDF Estimation from a Single Image. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 12840–12848.
71. Raquel Vidaurre, Dan Casas, Elena Garces, and Jorge Lopez-Moreno. 2019. Brdf estimation of complex materials with nested learning. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, 1347–1356.
72. Chun-Po Wang, Noah Snavely, and Steve Marschner. 2011. Estimating dual-scale properties of glossy surfaces from step-edge lighting. In Proceedings of the 2011 SIGGRAPH Asia Conference. 1–12.
73. D Rod White, Peter Saunders, Stuart J Bonsey, John van de Ven, and Hamish Edgar. 1998. Reflectometer for measuring the bidirectional reflectance of rough surfaces. Applied Optics 37, 16 (1998), 3450–3454.
74. Robert J Woodham. 1980. Photometric method for determining surface orientation from multiple images. Optical Engineering 19, 1 (1980), 139–144.
75. Yuxin Wu and Kaiming He. 2018. Group normalization. In Proceedings of the European Conference on Computer Vision (ECCV). 3–19.
76. Bei Xiao, Bruce Walter, Ioannis Gkioulekas, Todd Zickler, Edward Adelson, and Kavita Bala. 2014. Looking against the light: How perception of translucency depends on lighting direction. Journal of Vision 14, 3 (03 2014), 17–17.
77. Bei Xiao, Shuang Zhao, Ioannis Gkioulekas, Wenyan Bi, and Kavita Bala. 2020. Effect of geometric sharpness on translucent material perception. Journal of Vision 20, 7 (07 2020), 10–10.
78. Jiyang Yu, Zexiang Xu, Matteo Mannino, Henrik Wann Jensen, and Ravi Ramamoorthi. 2016. Sparse sampling for image-based SVBRDF acquisition. In Proceedings of the Eurographics 2016 Workshop on Material Appearance Modeling. 19–22.
79. Shuang Zhao, Wenzel Jakob, Steve Marschner, and Kavita Bala. 2011. Building volumetric appearance models of fabric using micro CT imaging. ACM Transactions on Graphics (TOG) 30, 4 (2011), 1–10.
80. Shuang Zhao, Wenzel Jakob, Steve Marschner, and Kavita Bala. 2012. Structure-aware synthesis for predictive woven fabric appearance. ACM Transactions on Graphics (TOG) 31, 4 (2012), 1–10.
81. Shuang Zhao, Fujun Luan, and Kavita Bala. 2016. Fitting procedural yarn models for realistic cloth rendering. ACM Transactions on Graphics (TOG) 35, 4 (2016), 1–11.
82. Xilong Zhou and Nima Khademi Kalantari. 2021. Adversarial Single-Image SVBRDF Estimation with Hybrid Training. In Computer Graphics Forum, Vol. 40. Wiley Online Library, 315–325.
83. Xilong Zhou and Nima Khademi Kalantari. 2022. Look-Ahead Training with Learned Reflectance Loss for Single-Image SVBRDF Estimation. ACM Transactions on Graphics (TOG) 41, 6 (2022), 1–12.
