“Kaleidoscopic structured light” by Ahn, Gkioulekas and Sankaranarayanan
Conference:
Type(s):
Title:
- Kaleidoscopic structured light
Session/Category Title: Computational Photography
Presenter(s)/Author(s):
Abstract:
Full surround 3D imaging for shape acquisition is essential for generating digital replicas of real-world objects. Surrounding an object we seek to scan with a kaleidoscope, that is, a configuration of multiple planar mirrors, produces an image of the object that encodes information from a combinatorially large number of virtual viewpoints. This information is practically useful for the full surround 3D reconstruction of the object, but cannot be used directly, as we do not know what virtual viewpoint each image pixel corresponds—the pixel label. We introduce a structured light system that combines a projector and a camera with a kaleidoscope. We then prove that we can accurately determine the labels of projector and camera pixels, for arbitrary kaleidoscope configurations, using the projector-camera epipolar geometry. We use this result to show that our system can serve as a multi-view structured light system with hundreds of virtual projectors and cameras. This makes our system capable of scanning complex shapes precisely and with full coverage. We demonstrate the advantages of the kaleidoscopic structured light system by scanning objects that exhibit a large range of shapes and reflectances.
References:
1. Kfir Aberman, Oren Katzir, Qiang Zhou, Zegang Luo, Andrei Sharf, Chen Greif, Baoquan Chen, and Daniel Cohen-Or. 2017. Dip transform for 3D shape reconstruction. ACM Transactions on Graphics (TOG) 36, 4 (2017).
2. Byeongjoo Ahn, Ioannis Gkioulekas, and Aswin C. Sankaranarayanan. 2021. Project page: Kaleidoscopic structured light. https://imaging.cs.cmu.edu/kaleidoscopic_structured_light
3. Shaun Bangay and Judith D Radloff. 2004. Kaleidoscope configurations for reflectance measurement. In International Conference on Computer Graphics, Virtual Reality, Visualisation and Interaction in Africa.
4. Paul J Besl and Neil D McKay. 1992. A method for registration of 3-D shapes. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 14, 2 (1992).
5. David Brewster. 1858. The kaleidoscope, its history, theory and construction: with its application to theft ne and useful arts. J. Murray.
6. Yan Cui, Sebastian Schuon, Derek Chan, Sebastian Thrun, and Christian Theobalt. 2010. 3D shape scanning with a time-of-flight camera. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
7. Martin A Fischler and Robert C Bolles. 1981. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24, 6 (1981).
8. Keith Forbes, Fred Nicolls, Gerhard De Jager, and Anthon Voigt. 2006. Shape-from-silhouette with two mirrors and an uncalibrated camera. In European Conference on Computer Vision (ECCV).
9. Martin Fuchs, Markus Kächele, and Szymon Rusinkiewicz. 2013. Design and fabrication of faceted mirror arrays for light field capture. In Computer Graphics Forum, Vol. 32.
10. Gaurav Garg, Eino-Ville Talvala, Marc Levoy, and Hendrik PA Lensch. 2006. Symmetric Photography: Exploiting data-sparseness in reflectance fields. In Symposium on Rendering.
11. Abhijeet Ghosh, Graham Fyffe, Borom Tunwattanapong, Jay Busch, Xueming Yu, and Paul Debevec. 2011. Multiview face capture using polarized spherical gradient illumination. ACM Transactions on Graphics (TOG) 30, 6 (2011).
12. Joshua Gluckman and Shree K Nayar. 2001. Catadioptric stereo using planar mirrors. International Journal of Computer Vision (IJCV) 44, 1 (2001).
13. Joshua Gluckman and Shree K Nayar. 2002. Rectified catadioptric stereo sensors. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 24, 2 (2002).
14. Ardeshir Goshtasby and William A Gruver. 1993. Design of a single-lens stereo camera system. Pattern Recognition 26, 6 (1993).
15. Jefferson Y Han and Ken Perlin. 2003. Measuring bidirectional texture reflectance with a kaleidoscope. ACM Transactions on Graphics (TOG) 22, 3 (2003).
16. Richard Hartley and Andrew Zisserman. 2004. Multiple view geometry in computer vision (2nd ed.). Cambridge University Press.
17. Michael Holroyd, Jason Lawrence, and Todd Zickler. 2010. A coaxial optical scanner for synchronous acquisition of 3D geometry and surface reflectance. ACM Transactions on Graphics (TOG) 29, 4 (2010).
18. Hugues Hoppe, Tony DeRose, Tom Duchamp, John McDonald, and Werner Stuetzle. 1992. Surface Reconstruction from Unorganized Points. Computer Graphics (SIGGRAPH’92 proceedings) 26, 2 (1992).
19. Bo Hu, Christopher Brown, and Randal Nelson. 2005. Multiple-view 3-D reconstruction using a mirror. Technical Report. University of Rochester, Department of Computer Science.
20. Po-Hao Huang and Shang-Hon Lai. 2006. Contour-based structure from reflection. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
21. Ivo Ihrke, Ilya Reshetouski, Alkhazur Manakov, Art Tevs, Michael Wand, and Hans-Peter Seidel. 2012. A kaleidoscopic approach to surround geometry and reflectance acquisition. In IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).
22. Hanbyul Joo, Tomas Simon, Xulong Li, Hao Liu, Lei Tan, Lin Gui, Sean Banerjee, Timothy Godisart, Bart Nabbe, Iain Matthews, et al. 2017. Panoptic studio: A massively multiview system for social interaction capture. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 41, 1 (2017).
23. Kaizhang Kang, Cihui Xie, Chengan He, Mingqi Yi, Minyi Gu, Zimin Chen, Kun Zhou, and Hongzhi Wu. 2019. Learning efficient illumination multiplexing for joint capture of reflectance and shape. ACM Transactions on Graphics (TOG) 38, 6 (2019).
24. Michael Kazhdan and Hugues Hoppe. 2013. Screened poisson surface reconstruction. ACM Transactions on Graphics (TOG) 32, 3 (2013).
25. Kalin Kolev, Petri Tanskanen, Pablo Speciale, and Marc Pollefeys. 2014. Turning mobile phones into 3D scanners. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
26. Kiriakos N Kutulakos and Steven M Seitz. 2000. A theory of shape by space carving. International Journal of Computer Vision (IJCV) 38, 3 (2000).
27. Douglas Lanman, Daniel Crispell, and Gabriel Taubin. 2009. Surround structured lighting: 3-D scanning with orthographic illumination. Computer Vision and Image Understanding (CVIU) 113, 11 (2009).
28. Daniel Lichy, Jiaye Wu, Soumyadip Sengupta, and David W Jacobs. 2021. Shape and material capture at home. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
29. Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. 2020. Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision (ECCV).
30. Hiroshi Mitsumoto, Shinichi Tamura, Kozo Okazaki, Naoki Kajimi, and Yutaka Fukui. 1992. 3-D reconstruction using mirror images based on a plane symmetry recovering method. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 14, 09 (1992).
31. Daniel Moreno and Gabriel Taubin. 2012. Simple, accurate, and robust projector-camera calibration. In IEEE International Conference on 3D Imaging, Modeling, Processing, Visualization and Transmission (3DIMPVT).
32. David W Murray. 1995. Recovering range using virtual multicamera stereo. Computer Vision and Image Understanding (CVIU) 61, 2 (1995).
33. Giljoo Nam, Joo Ho Lee, Diego Gutierrez, and Min H Kim. 2018. Practical svbrdf acquisition of 3d objects with unstructured flash photography. ACM Transactions on Graphics (TOG) 37, 6 (2018).
34. Sameer A Nene and Shree K Nayar. 1998. Stereo with mirrors. In IEEE International Conference on Computer Vision (ICCV).
35. Richard A Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux, David Kim, Andrew J Davison, Pushmeet Kohi, Jamie Shotton, Steve Hodges, and Andrew Fitzgibbon. 2011. KinectFusion: Real-time dense surface mapping and tracking. In IEEE International Symposium on Mixed and Augmented Reality (ISMAR).
36. NextEngine. 2000. . http://www.nextengine.com [Accessed Sep. 9, 2021].
37. Shohei Nobuhara, Takashi Kashino, Takashi Matsuyama, Kouta Takeuchi, and Kensaku Fujii. 2016. A single-shot multi-path interference resolution for mirror-based full 3D shape measurement with a correlation-based ToF camera. In International Conference on 3D Vision (3DV).
38. Peter Ondrúška, Pushmeet Kohli, and Shahram Izadi. 2015. Mobilefusion: Real-time volumetric surface reconstruction and dense tracking on mobile phones. IEEE Transactions on Visualization and Computer Graphics (TVCG) 21, 11 (2015).
39. Jaesik Park, Sudipta N Sinha, Yasuyuki Matsushita, Yu-Wing Tai, and In So Kweon. 2016. Robust multiview photometric stereo using planar mesh parameterization. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 39, 8 (2016).
40. Ilya Reshetouski, Alkhazur Manakov, Hans-Peter Seidel, and Ivo Ihrke. 2011. Three-dimensional kaleidoscopic imaging. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
41. Christopher Schwartz, Ralf Sarlette, Michael Weinmann, and Reinhard Klein. 2013. DOME II: a parallelized BTF acquisition system. In Eurographics Workshop on Material Appearance Modeling: Issues and Acquisition.
42. Yuichi Taguchi, Amit Agrawal, Srikumar Ramalingam, and Ashok Veeraraghavan. 2010a. Axial light field for curved mirrors: Reffect your perspective, widen your view. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
43. Yuichi Taguchi, Amit Agrawal, Ashok Veeraraghavan, Srikumar Ramalingam, and Ramesh Raskar. 2010b. Axial-cones: Modeling spherical catadioptric cameras for wide-angle light field rendering. ACM Transactions on Graphics (TOG) 29, 6 (2010).
44. Tomu Tahara, Ryo Kawahara, Shohei Nobuhara, and Takashi Matsuyama. 2015. Interference-free epipole-centered structured light pattern for mirror-based multi-view active stereo. In International Conference on 3D Vision (3DV).
45. Kosuke Takahashi, Akihiro Miyata, Shohei Nobuhara, and Takashi Matsuyama. 2017. A linear extrinsic calibration of kaleidoscopic imaging system from single 3d point. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
46. Kosuke Takahashi and Shohei Nobuhara. 2021. Structure of multiple mirror system from kaleidoscopic projections of single 3d point. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) (2021).
47. Hongzhi Wu, Zhaotian Wang, and Kun Zhou. 2015. Simultaneous localization and appearance estimation with a consumer RGB-D camera. IEEE Transactions on Visualization and Computer Graphics (TVCG) 22, 8 (2015).
48. Hongzhi Wu and Kun Zhou. 2015. AppFusion: Interactive appearance acquisition using a kinect sensor. Computer Graphics Forum 34, 6 (2015).
49. Rui Xia, Yue Dong, Pieter Peers, and Xin Tong. 2016. Recovering shape and spatially-varying surface reflectance under unknown illumination. ACM Transactions on Graphics (TOG) 35, 6 (2016).
50. Ruilin Xu, Mohit Gupta, and Shree K Nayar. 2018. Trapping light for time of flight. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
51. Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman. 2020. Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance. In Neural Information Processing Systems (NeurIPS).
52. Xianghua Ying, Kun Peng, Ren Ren, and Hongbin Zha. 2010. Geometric properties of multiple reflections in catadioptric camera with two planar mirrors. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
53. Zhengyou Zhang. 2000. A flexible new technique for camera calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 22, 11 (2000).
54. Zhenglong Zhou, Zhe Wu, and Ping Tan. 2013. Multi-view photometric stereo with spatially varying isotropic materials. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
