“Image based relighting using neural networks” by Ren, Dong, Lin, Tong and Guo

We present a neural network regression method for relighting realworld scenes from a small number of images. The relighting in this work is formulated as the product of the scene’s light transport matrix and new lighting vectors, with the light transport matrix reconstructed from the input images. Based on the observation that there should exist non-linear local coherence in the light transport matrix, our method approximates matrix segments using neural networks that model light transport as a non-linear function of light source position and pixel coordinates. Central to this approach is a proposed neural network design which incorporates various elements that facilitate modeling of light transport from a small image set. In contrast to most image based relighting techniques, this regression-based approach allows input images to be captured under arbitrary illumination conditions, including light sources moved freely by hand. We validate our method with light transport data of real scenes containing complex lighting effects, and demonstrate that fewer input images are required in comparison to related techniques.

1. Beale, M. H., Hagan, M. T., and Demuth, H. B. 2012. Neural network toolbox users guide.Google Scholar
2. Chuang, Y.-Y., Zongker, D. E., Hindorff, J., Curless, B., Salesin, D. H., and Szeliski, R. 2000. Environment matting extensions: Towards higher accuracy and real-time capture. In Proceedings of SIGGRAPH 2000, 121–130. Google ScholarDigital Library
3. Debevec, P., Hawkins, T., Tchou, C., Duiker, H.-P., Sarokin, W., and Sagar, M. 2000. Acquiring the reflectance field of a human face. In Proceedings of SIGGRAPH 2000, 145–156. Google ScholarDigital Library
4. Fuchs, M., Blanz, V., and Seidel, H.-P. 2005. Bayesian Relighting. In Eurographics Symposium on Rendering (2005), The Eurographics Association, K. Bala and P. Dutre, Eds. Google ScholarDigital Library
5. Fuchs, M., Blanz, V., Lensch, H. P., and Seidel, H.-P. 2007. Adaptive sampling of reflectance fields. ACM Trans. Graph. 26, 2 (June). Google ScholarDigital Library
6. Garg, G., Talvala, E.-V., Levoy, M., and Lensch, H. P. 2006. Symmetric photography: Exploiting data-sparseness in reflectance fields. In Proceedings of the EGSR 2006, 251–262. Google ScholarDigital Library
7. Hagan, M., and Menhaj, M.-B. 1994. Training feedforward networks with the marquardt algorithm. Neural Networks, IEEE Transactions on 5, 6 (Nov), 989–993. Google ScholarDigital Library
8. Hansen, L. K., and Salamon, P. 1990. Neural network ensembles. IEEE Trans. Pattern Anal. Mach. Intell. 12, 10 (Oct.), 993–1001. Google ScholarDigital Library
9. Hašan, M., Pellacini, F., and Bala, K. 2007. Matrix row-column sampling for the many-light problem. ACM Trans. Graph. 26, 3 (July). Google ScholarDigital Library
10. Hawkins, T., Einarsson, P., and Debevec, P. 2005. A dual light stage. In Proceedings of EGSR 2005, 91–98. Google ScholarDigital Library
11. Hinton, G. E. 1989. Connectionist learning procedures. Artif. Intell. 40, 1–3 (Sept.), 185–234. Google ScholarDigital Library
12. Jang, H., Park, A., and Jung, K. 2008. Neural network implementation using cuda and openmp. In Digital Image Computing: Techniques and Applications (DICTA), 2008, 155–161. Google ScholarDigital Library
13. Mahajan, D., Shlizerman, I. K., Ramamoorthi, R., and Belhumeur, P. 2007. A theory of locally low dimensional light transport. ACM Trans. Graph. 26, 3 (July). Google ScholarDigital Library
14. Malzbender, T., Gelb, D., and Wolters, H. 2001. Polynomial texture maps. In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, ACM, New York, NY, USA, SIGGRAPH ’01, 519–528. Google ScholarDigital Library
15. Marwah, K., Wetzstein, G., Bando, Y., and Raskar, R. 2013. Compressive Light Field Photography using Overcomplete Dictionaries and Optimized Projections. ACM Trans. Graph. (Proc. SIGGRAPH) 32, 4, 1–11. Google ScholarDigital Library
16. Masselus, V., Peers, P., Dutré, P., and Willems, Y. D. 2003. Relighting with 4d incident light fields. ACM Trans. Graph. 22, 3 (July), 613–620. Google ScholarDigital Library
17. Masselus, V., Peers, P., Dutré, P., and Willemsy, Y. D. 2004. Smooth reconstruction and compact representation of reflectance functions for image-based relighting. In Proceedings of the Fifteenth Eurographics Conference on Rendering Techniques, Eurographics Association, Aire-la-Ville, Switzerland, Switzerland, EGSR’04, 287–298. Google ScholarDigital Library
18. Matusik, W., Loper, M., and Pfister, H. 2004. Progressively-refined reflectance functions from natural illumination. In Proceedings of EGSR 2004, 299–308. Google ScholarDigital Library
19. Nayar, S. K. 1989. Sphereo: Determining depth using two specular spheres and a single camera. International Society for Optics and Photonics, 245–254.Google Scholar
20. Ng, R., Ramamoorthi, R., and Hanrahan, P. 2003. All-frequency shadows using non-linear wavelet lighting approximation. ACM Trans. Graph. 22, 3 (July), 376–381. Google ScholarDigital Library
21. Nguyen, D., and Widrow, B. 1990. Improving the learning speed of 2-layer neural networks by choosing initial values of the adaptive weights. In Proceedings of the International Joint Conference on Neural Networks, vol. 3, 21–26.Google Scholar
22. Nowrouzezahrai, D., and Snyder, J. 2009. Fast global illumination on dynamic height fields. Comput. Graph. Forum 28, 4, 1131–1139. Google ScholarDigital Library
23. O’Toole, M., and Kutulakos, K. N. 2010. Optical computing for fast light transport analysis. ACM Trans. Graph. 29, 6 (Dec.), 164:1–164:12. Google ScholarDigital Library
24. O’Toole, M., Raskar, R., and Kutulakos, K. N. 2012. Primal-dual coding to probe light transport. ACM Trans. Graph. 31, 4 (July), 39:1–39:11. Google ScholarDigital Library
25. Ou, J., and Pellacini, F. 2011. Lightslice: Matrix slice sampling for the many-lights problem. ACM Trans. Graph. 30, 6 (Dec.), 179:1–179:8. Google ScholarDigital Library
26. Peers, P., and Dutré, P. 2003. Wavelet environment matting. In Proceedings of EGRW 03, 157–166. Google ScholarDigital Library
27. Peers, P., and Dutré, P. 2005. Inferring reflectance functions from wavelet noise. In Proceedings of the EGSR 2005, 173–182. Google ScholarDigital Library
28. Peers, P., Mahajan, D. K., Lamond, B., Ghosh, A., Matusik, W., Ramamoorthi, R., and Debevec, P. 2009. Compressive light transport sensing. ACM Trans. Graph. 28. Google ScholarDigital Library
29. Ramamoorthi, R. 2009. Precomputation-Based Rendering. NOW Publishers Inc. Google ScholarDigital Library
30. Reddy, D., Ramamoorthi, R., and Curless, B. 2012. Frequency-space decomposition and acquisition of light transport under spatially varying illumination. European Conference on Computer Vision. Google ScholarDigital Library
31. Ren, P., Wang, J., Gong, M., Lin, S., Tong, X., and Guo, B. 2013. Global illumination with radiance regression functions. ACM Trans. Graph. 32, 4 (July), 130:1–130:12. Google ScholarDigital Library
32. Sen, P., and Darabi, S. 2009. Compressive Dual Photography. Computer Graphics Forum 28, 2, 609–618.Google ScholarCross Ref
33. Sen, P., Chen, B., Garg, G., Marschner, S. R., Horowitz, M., Levoy, M., and Lensch, H. P. A. 2005. Dual photography. ACM Trans. Graph. 24, 3 (July), 745–755. Google ScholarDigital Library
34. Sloan, P.-P., Kautz, J., and Snyder, J. 2002. Precomputed radiance transfer for real-time rendering in dynamic, low-frequency lighting environments. ACM Trans. Graph. 21, 3 (July), 527–536. Google ScholarDigital Library
35. Sloan, P.-P., Hall, J., Hart, J., and Snyder, J. 2003. Clustered principal components for precomputed radiance transfer. ACM Trans. Graph. 22, 3 (July), 382–391. Google ScholarDigital Library
36. Tsai, Y.-T., and Shih, Z.-C. 2006. All-frequency precomputed radiance transfer using spherical radial basis functions and clustered tensor approximation. ACM Trans. Graph. 25, 3 (July), 967–976. Google ScholarDigital Library
37. Turmon, M. J., and Fine, T. L. 1995. Sample size requirements for feedforward neural networks. In Advances in Neural Information Processing Systems, MIT Press, NIPS 7.Google Scholar
38. Vasilescu, M. A. O., and Terzopoulos, D. 2004. Tensortextures: multilinear image-based rendering. ACM Trans. Graph. 23, 3, 336–342. Google ScholarDigital Library
39. Walter, B., Fernandez, S., Arbree, A., Bala, K., Donikian, M., and Greenberg, D. P. 2005. Lightcuts: A scalable approach to illumination. ACM Trans. Graph. 24, 3 (July), 1098–1107. Google ScholarDigital Library
40. Wang, J., Dong, Y., Tong, X., Lin, Z., and Guo, B. 2009. Kernel nystróm method for light transport. ACM Trans. Graph. 28, 3 (July), 29:1–29:10. Google ScholarDigital Library
41. Wenger, A., Gardner, A., Tchou, C., Unger, J., Hawkins, T., and Debevec, P. 2005. Performance relighting and reflectance transformation with time-multiplexed illumination. ACM Trans. Graph. 24, 3 (July), 756–764. Google ScholarDigital Library
42. Zongker, D. E., Werner, D. M., Curless, B., and Salesin, D. H. 1999. Environment matting and compositing. In Proceedings of SIGGRAPH 99, 205–214. Google ScholarDigital Library