“Joint Motion and Reflectance Capture for Relightable 3D Video” by Theobalt, Ahmed, de Aguiar, Ziegler, Lensch, et al. …

3D Videos of Human Actors can be faithfully reconstructed from multiple synchronized video streams by means of a model-based analysis-by-synthesis approach [Carranza et al. 2003]. The recon- structed videos play back in real-time and the virtual viewpoint onto the scene can be arbitrarily changed. By this means, authentically animated, photo-realistically and view-dependently textured models of real people can be created that look real under fixed illumination conditions. To import real-world characters into virtual environments, however, relighting is vital and thus also surface reflectance properties must be known. To serve this purpose we have developed a video-based modeling approach that captures human motion as well as reflectance characteristics from a handful of synchronized video recordings. The presented method is able to re- cover spatially varying reflectance properties of skin and clothes by exploiting the time-varying orientation of each surface point with respect to camera and light direction. The resulting model description enables us to match animated subject appearance to different lighting conditions, as well as to interchange surface attributes among different people, e.g. for virtual dressing. Our relightable 3D videos allow populating virtual worlds with convincingly relit real-world people.

Carranza, J., Theobalt, C., Magnor, M., And Seidel, H.-P. 2003. Freeviewpoint video of human actors. In Proc. of SIGGRAPH’03, 569–577.]]
Lafortune, E., Foo, S., Torrance, K., And Greenberg, D. 1997. Non-Linear Approximation of Reflectance Functions. In SIGGRAPH, 117–126.]]
Lensch, H. P. A., Kautz, J., Goesele, M., Heidrich, W., And Seidel, H.-P. Image-based reconstruction of spatial appearance and geometric detail. ACM Transactions on Graphics.]]
Phong, B.-T. 1975. Illumnation for computer generated pictures. Communications of the ACM, 311–317.]]