“Appearance-from-motion: recovering spatially varying surface reflectance under unknown lighting” by Dong, Chen, Peers, Zhang and Tong
Conference:
Type(s):
Title:
- Appearance-from-motion: recovering spatially varying surface reflectance under unknown lighting
Session/Category Title: Displays, Reflectance and Texture
Presenter(s)/Author(s):
Abstract:
We present “appearance-from-motion”, a novel method for recovering the spatially varying isotropic surface reflectance from a video of a rotating subject, with known geometry, under unknown natural illumination. We formulate the appearance recovery as an iterative process that alternates between estimating surface reflectance and estimating incident lighting. We characterize the surface reflectance by a data-driven microfacet model, and recover the microfacet normal distribution for each surface point separately from temporal changes in the observed radiance. To regularize the recovery of the incident lighting, we rely on the observation that natural lighting is sparse in the gradient domain. Furthermore, we exploit the sparsity of strong edges in the incident lighting to improve the robustness of the surface reflectance estimation. We demonstrate robust recovery of spatially varying isotropic reflectance from captured video as well as an internet video sequence for a wide variety of materials and natural lighting conditions.
References:
1. Aittala, M., Weyrich, T., and Lehtinen, J. 2013. Practical svbrdf capture in the frequency domain. ACM Trans. Graph. 32, 4, 110:1–110:12.
2. Ashikhmin, M., Premoze, S., and Shirley, P. 2000. A microfacet-based BRDF generator. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques, 65–74.
3. Barron, J. T., and Malik, J. 2013. Shape, illumination, and reflectance from shading. Tech. Rep. UCB/EECS-2013-117, EECS, UC Berkeley, May.
4. cho, S., and Lee, S. 2009. Fast motion deblurring. ACM Trans. Graph. 28, 5, 145:1–145:8.
5. Dong, Y., Wang, J., Tong, X., Snyder, J., Lan, Y., Benezra, M., and Guo, B. 2010. Manifold bootstrapping for SVBRDF capture. ACM Trans. Graph. 29, 4, 98:1–98:10.
6. Dorsey, J., Rushmeier, H., and Sillion, F. 2008. Digital Modeling of Material Appearance. Morgan Kaufmann Publishers Inc.
7. Gardner, A., Tchou, C., Hawkins, T., and Debevec, P. 2003. Linear light source reflectometry. ACM Trans. Graph. 22, 3, 749–758.
8. Gregson, J., Heide, F., Hullin, M. B., Rouf, M., and Heidrich, W. 2013. Stochastic Deconvolution. In CVPR, 1043–1050.
9. Haber, T., Fuchs, C., Bekaer, P., Seidel, H. P., Goesele, M., and Lensch, H. 2009. Relighting objects from image collections. In CVPR, 627–634.
10. Hertzmann, A., and Seitz, S. M. 2003. Shape and materials by example: A photometric stereo approach. In CVPR, 533–540.
11. Holroyd, M., Lawrence, J., and Zickler, T. 2010. A coaxial optical scanner for synchronous acquisition of 3D geometry and surface reflectance. ACM Trans. Graph. 29, 4, 99:1–99:12.
12. Lensch, H. P. A., Kautz, J., Goesele, M., Heidrich, W., and Seidel, H.-P. 2003. Image-based reconstruction of spatial appearance and geometric detail. ACM Trans. Graph. 22, 2, 234–257.
13. Levin, A., Fergus, R., Durand, F., and Freeman, W. T. 2007. Image and depth from a conventional camera with a coded aperture. ACM Trans. Graph. 26, 3, 70:1–70:9.
14. Li, G., Wu, C., Stoll, C., Liu, Y., Varanasi, K., Dai, Q., and Theobalt, C. 2013. Capturing relightable human performances under general uncontrolled illumination. Comput. Graph. Forum 32, 2, 275–284.Cross Ref
15. Lombardi, S., and Nishino, K. 2012. Reflectance and natural illumination from a single image. In ECCV, 582–595.
16. Lucas, B., and Kanade, T. 1981. An iterative image registration technique with an application to stereo vision. In Proc. Int. Joint Conf. on Artificial Intelligence, 674–679.
17. Marschner, S. R. 1998. Inverse Rendering for Computer Graphics. PhD thesis, Cornell University.
18. Matusik, W., Pfister, H., Brand, M., and McMillan, L. 2003. A data-driven reflectance model. ACM Trans. Graph. 22, 3, 759–769.
19. Nicodemus, F. E., Richmond, J. C., Hsia, J. J., Ginsberg, I. W., and Limperis, T. 1977. Geometric considerations and nomenclature for reflectance. Monograph 161,National Bureau of Standards (US).
20. Nishino, K., Zhang, Z., and Ikeuchi, K. 2001. Determining reflectance parameters and illumination distribution from a sparse set of images for view-dependent image synthesis. In ICCV, 599–606.
21. Osher, S., and Rudin, L. I. 1990. Feature-oriented image enhancement using shock filters. SIAM J. Numer. Anal. 27, 4, 919–940.
22. Ramamoorthi, R., and Hanrahan, P. 2001. A signal-processing framework for inverse rendering. In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH ’01, 117–128.
23. Ren, P., Wang, J., Snyder, J., Tong, X., and Guo, B. 2011. Pocket reflectometry. ACM Trans. Graph. 30, 4, 45:1–45:10.
24. Romeiro, F., and Zickler, T. 2010. Blind reflectometry. In ECCV, 45–58.
25. Romeiro, F., Vasilyev, Y., and Zickler, T. 2008. Passive reflectometry. In ECCV, 859–872.
26. Shan, Q., Adams, R., Curless, B., Furukawa, Y., and Seitz, S. M. 2013. The visual turing test for scene reconstruction. In 3DV, 25–32.
27. Shi, J., and Tomasi, C. 1994. Good features to track. In CVPR, 593–600.
28. Treuille, A., Hertzmann, A., and Seitz, S. M. 2004. Example-based stereo with general BRDFs. In ECCV, 457–469.
29. Tunwattanapong, B., Fyffe, G., Graham, P., Busch, J., Yu, X., Ghosh, A., and Debevec, P. E. 2013. Acquiring reflectance and shape from continuous spherical harmonic illumination. ACM Trans. Graph. 32, 4, 109.
30. Wang, C.-P., Snavely, N., and Marschner, S. 2011. Estimating dual-scale properties of glossy surfaces from step-edge lighting. ACM Trans. Graph. 30, 6, 172:1–172:12.
31. Xu, L., and Jia, J. 2010. Two-phase kernel estimation for robust motion deblurring. In ECCV, 157–170.
32. Yu, T., Wang, H., Ahuja, N., and Chen, W.-C. 2006. Sparse lumigraph relighting by illumination and reflectance estimation from multi-view images. In Rendering Techniques, 41–50.
33. Zhang, Z. 2000. A flexible new technique for camera calibration. In IEEE PAMI, vol. 22, 1330–1334.
