“Recovering shape and spatially-varying surface reflectance under unknown illumination”
Conference:
Type(s):
Title:
- Recovering shape and spatially-varying surface reflectance under unknown illumination
Session/Category Title: Materials
Presenter(s)/Author(s):
Abstract:
We present a novel integrated approach for estimating both spatially-varying surface reflectance and detailed geometry from a video of a rotating object under unknown static illumination. Key to our method is the decoupling of the recovery of normal and surface reflectance from the estimation of surface geometry. We define an apparent normal field with corresponding reflectance for each point (including those not on the object’s surface) that best explain the observations. We observe that the object’s surface goes through points where the apparent normal field and corresponding reflectance exhibit a high degree of consistency with the observations. However, estimating the apparent normal field requires knowledge of the unknown incident lighting. We therefore formulate the recovery of shape, surface reflectance, and incident lighting, as an iterative process that alternates between estimating shape and lighting, and simultaneously recovers surface reflectance at each step. To recover the shape, we first form an initial surface that passes through locations with consistent apparent temporal traces, followed by a refinement that maximizes the consistency of the surface normals with the underlying apparent normal field. To recover the lighting, we rely on appearance-from-motion using the recovered geometry from the previous step. We demonstrate our integrated framework on a variety of synthetic and real test cases exhibiting a wide variety of materials and shape.
References:
1. Ackermann, J., Ritz, M., Stork, A., and Goesele, M. 2012. Removing the example from example-based photometric stereo. In ECCV, 197–210.
2. Adato, Y., Vasilyev, Y., Zickler, T., and Ben-Shahar, O. 2010. Shape from specular flow. IEEE PAMI 32, 11 (Nov), 2054–2070.
3. Ashikhmin, M., Premoze, S., and Shirley, P. 2000. A microfacet-based BRDF generator. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques, 65–74.
4. Barron, J. T., and Malik, J. 2015. Shape, illumination, and reflectance from shading. IEEE PAMI.
5. Basri, R., Jacobs, D. W., and Kemelmacher, I. 2007. Photometric stereo with general, unknown lighting. IJCV 72, 3, 239–257.
6. Chandraker, M. 2014. On shape and material recovery from motion. In ECCV, 202–217.
7. Cook, R. L., and Torrance, K. E. 1982. A reflectance model for computer graphics. ACM Trans. Graph. 1, 1, 7–24.
8. Dong, Y., Wang, J., Tong, X., Snyder, J., Lan, Y., BenEzra, M., and Guo, B. 2010. Manifold bootstrapping for SVBRDF capture. ACM Trans. Graph. 29, 4, 98:1–98:10.
9. Dong, Y., Chen, G., Peers, P., Zhang, J., and Tong, X. 2014. Appearance-from-motion: Recovering spatially varying surface reflectance under unknown lighting. ACM Trans. Graph. 33, 6, 193:1–193:12.
10. Fiala, M. 2005. Artag, a fiducial marker system using digital techniques. In CVPR, vol. 2, 590–596.
11. Hertzmann, A., and Seitz, S. M. 2003. Shape and materials by example: A photometric stereo approach. In CVPR, 533–540.
12. Kazhdan, M., and Hoppe, H. 2013. Screened poisson surface reconstruction. ACM Trans. Graph. 32, 3, 29:1–29:13.
13. Lombardi, S., and Nishino, K. 2016. Reflectance and illumination recovery in the wild. IEEE PAMI 38, 1, 129–141.
14. Lu, F., Matsushita, Y., Sato, I., Okabe, T., and Sato, Y. 2013. Uncalibrated photometric stereo for unknown isotropic reflectances. In CVRP, 1490–1497.
15. Nehab, D., Rusinkiewicz, S., Davis, J., and Ramamoorthi, R. 2005. Efficiently combining positions and normals for precise 3d geometry. ACM Trans. Graph. 24, 3, 536–543.
16. Nicodemus, F. E., Richmond, J. C., Hsia, J. J., Ginsberg, I. W., and Limperis, T. 1977. Geometric considerations and nomenclature for reflectance. Monograph 161,National Bureau of Standards (US).
17. Oxholm, G., and Nishino, K. 2012. Shape and reflectance from natural illumination. In ECCV, 528–541.
18. Oxholm, G., and Nishino, K. 2014. Multiview shape and reflectance from natural illumination. In CVPR, 2163–2170.
19. Palma, G., Callieri, M., Dellepiane, M., and Scopigno, R. 2012. A statistical method for svbrdf approximation from video sequences in general lighting conditions. Comput. Graph. Forum 31, 4, 1491–1500.
20. Romeiro, F., and Zickler, T. 2010. Blind reflectometry. In ECCV, 45–58.
21. Seitz, S. M., Curless, B., Diebel, J., Scharstein, D., and Szeliski, R. 2006. A comparison and evaluation of multi-view stereo reconstruction algorithms. In CVPR, 519–528.
22. Treuille, A., Hertzmann, A., and Seitz, S. M. 2004. Example-based stereo with general BRDFs. In ECCV, 457–469.
23. Triggs, B., McLauchlan, P. F., Hartley, R. I., and Fitzgibbon, A. W. 1999. Bundle adjustment – a modern synthesis. In ICCV, 298–372.
24. Valgaerts, L., Wu, C., Bruhn, A., Seidel, H.-P., and Theobalt, C. 2012. Lightweight binocular facial performance capture under uncontrolled lighting. ACM Trans. Graph. 31, 6, 187:1–187:11.
25. Wang, J., Zhao, S., Tong, X., Snyder, J., and Guo, B. 2008. Modeling anisotropic surface reflectance with example-based microfacet synthesis. ACM Trans. Graph. 27, 3, 41:1–41:9.
26. Wang, T.-C., Chandraker, M., Efros, A., and Ramamoorthi, R. 2016. Svbrdf-invariant shape and reflectance estimation from light-field cameras. In CVPR.
27. Weinmann, M., and Klein, R. 2015. Advances in geometry and reflectance acquisition. In ACM SIGGRAPH Asia, Course Notes.
28. Woodham, R. J. 1980. Photometric method for determining surface orientation from multiple images. Optical Engineering 19, 1, 3050–3068. Cross Ref
29. Wu, C., Wilburn, B., Matsushita, Y., and Theobalt, C. 2011. High-quality shape from multi-view stereo and shading under general illumination. In CVPR, 969–976.
30. Wu, H., Wang, Z., and Zhou, K. 2016. Simultaneous localization and appearance estimation with a consumer RGB-D camera. IEEE Trans. Vis. and Comp. Graph. 2, 8, 2012–2023.
31. Xu, D., Duan, Q., Zheng, J., Zhang, J., Cai, J., and Cham, T.-J. 2014. Recovering surface details under general unknown illumination using shading and coarse multi-view stereo.
32. Zhang, Z. 2000. A flexible new technique for camera calibration. In IEEE PAMI, vol. 22, 1330–1334.
