“Differentiable refraction-tracing for mesh reconstruction of transparent objects” by Lyu, Wu, Lischinski, Cohen-Or and Huang
Conference:
Type(s):
Title:
- Differentiable refraction-tracing for mesh reconstruction of transparent objects
Session/Category Title: Differentiable Graphics
Presenter(s)/Author(s):
Abstract:
Capturing the 3D geometry of transparent objects is a challenging task, ill-suited for general-purpose scanning and reconstruction techniques, since these cannot handle specular light transport phenomena. Existing state-of-the-art methods, designed specifically for this task, either involve a complex setup to reconstruct complete refractive ray paths, or leverage a data-driven approach based on synthetic training data. In either case, the reconstructed 3D models suffer from over-smoothing and loss of fine detail. This paper introduces a novel, high precision, 3D acquisition and reconstruction method for solid transparent objects. Using a static background with a coded pattern, we establish a mapping between the camera view rays and locations on the background. Differentiable tracing of refractive ray paths is then used to directly optimize a 3D mesh approximation of the object, while simultaneously ensuring silhouette consistency and smoothness. Extensive experiments and comparisons demonstrate the superior accuracy of our method.
References:
1. Kfir Aberman, Oren Katzir, Qiang Zhou, Zegang Luo, Andrei Sharf, Chen Greif, Baoquan Chen, and Daniel Cohen-Or. 2017. Dip Transform for 3D Shape Reconstruction. ACM Trans. on Graphics (Proc. of SIGGRAPH) 36, 4 (2017), 79:1–79:11.Google ScholarDigital Library
2. Moshe Ben-Ezra and Shree K. Nayar. 2003. What Does Motion Reveal About Transparency? Proc. Int. Conf. on Computer Vision (2003), 1025–1032.Google ScholarDigital Library
3. Max Born and Emil Wolf. 2013. Principles of optics: electromagnetic theory of propagation, interference and diffraction of light. Elsevier.Google Scholar
4. Yung-Yu Chuang, Douglas E. Zongker, Joel Hindorff, Brian Curless, David H. Salesin, and Richard Szeliski. 2000. Environment Matting Extensions: Towards Higher Accuracy and Real-time Capture. ACM Trans. on Graphics (Proc. of SIGGRAPH) (2000), 121–130.Google ScholarDigital Library
5. Matthias B. Hullin, Martin Fuchs, Ivo Ihrke, Hans-Peter Seidel, and Hendrik P. A. Lensch. 2008. Fluorescent Immersion Range Scanning. ACM Trans. on Graphics (Proc. of SIGGRAPH) 27, 3 (2008), 87:1–87:10.Google ScholarDigital Library
6. Ivo Ihrke, Kiriakos N. Kutulakos, Hendrik Lensch, Marcus Magnor, and Wolfgang Heidrich. 2010. Transparent and specular object reconstruction. Computer Graphics Forum 29, 8 (2010), 2400–2426.Google ScholarCross Ref
7. Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. 2018. Neural 3d mesh renderer. Proc. IEEE Conf. on Computer Vision & Pattern Recognition (2018), 3907–3916.Google ScholarCross Ref
8. Kiriakos N. Kutulakos and Steven M. Seitz. 2000. A Theory of Shape by Space Carving. Int. J. Computer Vision 38, 3 (2000), 199–218.Google ScholarDigital Library
9. Kiriakos N. Kutulakos and Eron Steger. 2008. A theory of refractive and specular 3D shape by light-path triangulation. Int. J. Computer Vision 76, 1 (2008), 13–29.Google ScholarDigital Library
10. Anat Levin, Dani Lischinski, and Yair Weiss. 2008. A Closed-Form Solution to Natural Image Matting. IEEE Trans. Pattern Anal. Mach. Intell. 30, 2 (2008), 228–242.Google ScholarDigital Library
11. Tzu-Mao Li, Miika Aittala, Frédo Durand, and Jaakko Lehtinen. 2018. Differentiable Monte Carlo Ray Tracing through Edge Sampling. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 37, 6 (2018), 222:1–222:11.Google Scholar
12. Zhengqin Li, Yu-Ying Yeh, and Manmohan Chandraker. 2020. Through the Looking Glass: Neural 3D Reconstruction of Transparent Shapes. Proc. IEEE Conf. on Computer Vision & Pattern Recognition (2020).Google ScholarCross Ref
13. Shichen Liu, Tianye Li, Weikai Chen, and Hao Li. 2019. Soft rasterizer: A differentiable renderer for image-based 3d reasoning. Proc. Int. Conf. on Computer Vision (2019), 7708–7717.Google ScholarCross Ref
14. Nigel JW Morris and Kiriakos N. Kutulakos. 2007. Reconstructing the surface of inhomogeneous transparent scenes by scatter-trace photography. Proc. Int. Conf. on Computer Vision (2007), 1–8.Google Scholar
15. Nigel JW Morris and Kiriakos N. Kutulakos. 2011. Dynamic refraction stereo. IEEE Trans. Pattern Analysis & Machine Intelligence 33, 8 (2011), 1518–1531.Google ScholarDigital Library
16. Merlin Nimier-David, Delio Vicini, Tizian Zeltner, and Wenzel Jakob. 2019. Mitsuba 2: a retargetable forward and inverse renderer. ACM Trans. on Graphics (Proc. of SIGGRAPH Asia) 38, 6 (2019), 203:1–203:17.Google Scholar
17. Marios Papas, Wojciech Jarosz, Wenzel Jakob, Szymon Rusinkiewicz, Wojciech Matusik, and Tim Weyrich. 2011. Goal-based caustics. Computer Graphics Forum (Proc. of Eurographics) 30, 2 (2011), 503–511.Google ScholarCross Ref
18. Steven G Parker, James Bigler, Andreas Dietrich, Heiko Friedrich, Jared Hoberock, David Luebke, David McAllister, Morgan McGuire, Keith Morley, Austin Robison, et al. 2010. OptiX: a general purpose ray tracing engine. ACM Trans. on Graphics (Proc. of SIGGRAPH) 29, 4 (2010), 1–13.Google ScholarDigital Library
19. Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in PyTorch. NIPS-W (2017).Google Scholar
20. Pieter Peers and Philip Dutré. 2003. Wavelet environment matting. Proc. Eurographics Workshop on Rendering (2003), 157–166.Google ScholarDigital Library
21. Nico Pietroni, Marco Tarini, and Paolo Cignoni. 2010. Almost isometric mesh parameterization through abstract domains. IEEE Trans. Visualization & Computer Graphics 16, 4 (2010).Google ScholarDigital Library
22. Thomas Porter and Tom Duff. 1984. Compositing Digital Images. Proc. of SIGGRAPH 18, 3 (1984), 253–259.Google ScholarDigital Library
23. Yiming Qian, Minglun Gong, and Yee-Hong Yang. 2015. Frequency-based environment matting by compressive sensing. Proc. Int. Conf. on Computer Vision (2015), 3532–3540.Google ScholarDigital Library
24. Yiming Qian, Minglun Gong, and Yee-Hong Yang. 2017. Stereo-Based 3D Reconstruction of Dynamic Fluid Surfaces by Global Optimization. Proc. IEEE Conf. on Computer Vision & Pattern Recognition (2017), 6650–6659.Google ScholarCross Ref
25. Shreeyak Sajjan, Matthew Moore, Mike Pan, Ganesh Nagaraja, Johnny Chung Lee, Andy Zeng, and Shuran Song. 2020. ClearGrasp: 3D Shape Estimation of Transparent Objects for Manipulation. Proc. IEEE Int. Conf. on Robotics & Automation (2020).Google Scholar
26. Qi Shan, Sameer Agarwal, and Brian Curless. 2012. Refractive height fields from single and multiple images. Proc. IEEE Conf. on Computer Vision & Pattern Recognition (2012), 286–293.Google ScholarDigital Library
27. Jonathan Stets, Zhengqin Li, Jeppe Revall Frisvad, and Manmohan Chandraker. 2019. Single-shot analysis of refractive shape using convolutional neural networks. Proc. IEEE Winter Conf. on Applications of Computer Vision (2019), 995–1003.Google ScholarCross Ref
28. Kenichiro Tanaka, Yasuhiro Mukaigawa, Hiroyuki Kubo, Yasuyuki Matsushita, and Yasushi Yagi. 2016. Recovering Transparent Shape from Time-of-Flight Distortion. Proc. IEEE Conf. on Computer Vision & Pattern Recognition (2016), 4387–4395.Google ScholarCross Ref
29. Borislav Trifonov, Derek Bradley, and Wolfgang Heidrich. 2006. Tomographic reconstruction of transparent objects. Proc. Eurographics Conf. on Rendering Techniques (2006), 51–60.Google ScholarDigital Library
30. Chia-Yin Tsai, Ashok Veeraraghavan, and Aswin C Sankaranarayanan. 2015. What does a single light-ray reveal about a transparent object? Proc. IEEE Int. Conf. on Image Processing (2015), 606–610.Google ScholarDigital Library
31. Gordon Wetzstein, David Roodnick, Wolfgang Heidrich, and Ramesh Raskar. 2011. Refractive shape from light field distortion. Proc. Int. Conf. on Computer Vision (2011), 1180–1186.Google ScholarDigital Library
32. Bojian Wu, Yang Zhou, Yiming Qian, Minglun Gong, and Hui Huang. 2018. Full 3D Reconstruction of Transparent Objects. ACM Trans. on Graphics (Proc. of SIGGRAPH) 37, 4 (2018), 103:1–103:11.Google ScholarDigital Library
33. Sai-Kit Yeung, Tai-Pang Wu, Chi-Keung Tang, Tony F Chan, and Stanley Osher. 2011. Adequate reconstruction of transparent objects on a shoestring budget. Proc. IEEE Conf. on Computer Vision & Pattern Recognition (2011), 2513–2520.Google ScholarDigital Library
34. Mingjie Zhang, Xing Lin, Mohit Gupta, Jinli Suo, and Qionghai Dai. 2014. Recovering Scene Geometry under Wavy Fluid via Distortion and Defocus Analysis. Proc. Euro. Conf. on Computer Vision (2014), 234–250.Google ScholarCross Ref
35. Zhengyou Zhang. 2000. A Flexible New Technique for Camera Calibration. IEEE Trans. Pattern Analysis & Machine Intelligence 22, 11 (2000), 1330–1334.Google ScholarDigital Library
36. Douglas E. Zongker, Dawn M. Werner, Brian Curless, and David H. Salesin. 1999. Environment Matting and Compositing. ACM Trans. on Graphics (Proc. of SIGGRAPH) (1999), 205–214.Google Scholar
