“A machine learning approach for filtering Monte Carlo noise” by Kalantari, Bako and Sen

The most successful approaches for filtering Monte Carlo noise use feature-based filters (e.g., cross-bilateral and cross non-local means filters) that exploit additional scene features such as world positions and shading normals. However, their main challenge is finding the optimal weights for each feature in the filter to reduce noise but preserve scene detail. In this paper, we observe there is a complex relationship between the noisy scene data and the ideal filter parameters, and propose to learn this relationship using a nonlinear regression model. To do this, we use a multilayer perceptron neural network and combine it with a matching filter during both training and testing. To use our framework, we first train it in an offline process on a set of noisy images of scenes with a variety of distributed effects. Then at run-time, the trained network can be used to drive the filter parameters for new scenes to produce filtered images that approximate the ground truth. We demonstrate that our trained network can generate filtered images in only a few seconds that are superior to previous approaches on a wide range of distributed effects such as depth of field, motion blur, area lighting, glossy reflections, and global illumination.

1. Bala, K., Walter, B., and Greenberg, D. P. 2003. Combining edges and points for interactive high-quality rendering. ACM Trans. Graph. 22, 3 (July), 631–640. Google ScholarDigital Library
2. Bauszat, P., Eisemann, M., and Magnor, M. 2011. Guided image filtering for interactive high-quality global illumination. Computer Graphics Forum 30, 4, 1361–1368.Google ScholarDigital Library
3. Buades, A., Coll, B., and Morel, J. M. 2005. A review of image denoising algorithms, with a new one. Multiscale Modeling & Simulation 4, 2, 490–530.Google ScholarCross Ref
4. Burger, H., Schuler, C., and Harmeling, S. 2012. Image denoising: Can plain neural networks compete with BM3D? In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2392–2399. Google ScholarDigital Library
5. Chen, J., Wang, B., Wang, Y., Overbeck, R. S., Yong, J.-H., and Wang, W. 2011. Efficient depth-of-field rendering with adaptive sampling and multiscale reconstruction. Computer Graphics Forum 30, 6, 1667–1680.Google ScholarCross Ref
6. Cook, R. L., Porter, T., and Carpenter, L. 1984. Distributed ray tracing. SIGGRAPH Comput. Graph. 18, 3 (Jan.), 137–145. Google ScholarDigital Library
7. Dachsbacher, C. 2011. Analyzing visibility configurations. IEEE Transactions on Visualization and Computer Graphics 17, 4 (April), 475–486. Google ScholarDigital Library
8. Dammertz, H., Sewtz, D., Hanika, J., and Lensch, H. P. 2010. Edge-avoiding À-trous wavelet transform for fast global illumination filtering. In Proceedings of High Performance Graphics (HPG), 67–75. Google ScholarDigital Library
9. Delbracio, M., Musé, P., Buades, A., Chauvier, J., Phelps, N., and Morel, J.-M. 2014. Boosting Monte Carlo rendering by ray histogram fusion. ACM Trans. Graph. 33, 1 (Feb.), 8:1–8:15. Google ScholarDigital Library
10. Drucker, H., Burges, C. J. C., Kaufman, L., Smola, A., and Vapnik, V. 1997. Support vector regression machines. In Advances in Neural Information Processing Systems 9, MIT Press, 155–161.Google Scholar
11. Dutré, P., Bala, K., Bekaert, P., and Shirley, P. 2006. Advanced Global Illumination. AK Peters Ltd. Google ScholarDigital Library
12. Egan, K., Tseng, Y.-T., Holzschuch, N., Durand, F., and Ramamoorthi, R. 2009. Frequency analysis and sheared reconstruction for rendering motion blur. ACM Trans. Graph. 28, 3 (July), 93:1–93:13. Google ScholarDigital Library
13. Egan, K., Durand, F., and Ramamoorthi, R. 2011. Practical filtering for efficient ray-traced directional occlusion. ACM Trans. Graph. 30, 6 (Dec.), 180:1–180:10. Google ScholarDigital Library
14. Egan, K., Hecht, F., Durand, F., and Ramamoorthi, R. 2011. Frequency analysis and sheared filtering for shadow light fields of complex occluders. ACM Trans. Graph. 30, 2 (Apr.), 9:1–9:13. Google ScholarDigital Library
15. Grzeszczuk, R., Terzopoulos, D., and Hinton, G. 1998. Neuroanimator: Fast neural network emulation and control of physics-based models. In ACM SIGGRAPH ’98, ACM, New York, NY, USA, 9–20. Google ScholarDigital Library
16. Hachisuka, T., Jarosz, W., Weistroffer, R. P., Dale, K., Humphreys, G., Zwicker, M., and Jensen, H. W. 2008. Multidimensional adaptive sampling and reconstruction for ray tracing. ACM Trans. Graph. 27 (Aug.), 33:1–33:10. Google ScholarDigital Library
17. Hastie, T., Tibshirani, R., Friedman, J., Hastie, T., Friedman, J., and Tibshirani, R. 2009. The elements of statistical learning, vol. 2. Springer.Google Scholar
18. Jakob, W., Regg, C., and Jarosz, W. 2011. Progressive expectation-maximization for hierarchical volumetric photon mapping. Computer Graphics Forum 30, 4, 1287–1297.Google ScholarDigital Library
19. Jensen, H. W., and Christensen, N. J. 1995. Optimizing path tracing using noise reduction filters. In Winter School of Computer Graphics (WSCG) 1995, 134–142.Google Scholar
20. Jensen, H. W. 2001. Realistic Image Synthesis Using Photon Mapping. A. K. Peters, Ltd., Natick, MA, USA. Google ScholarDigital Library
21. Kalantari, N. K., and Sen, P. 2013. Removing the noise in Monte Carlo rendering with general image denoising algorithms. Computer Graphics Forum 32, 2pt1, 93–102.Google Scholar
22. Laine, S., Saransaari, H., Kontkanen, J., Lehtinen, J., and Aila, T. 2007. Incremental instant radiosity for real-time indirect illumination. In Proceedings of the 18th Eurographics Conference on Rendering Techniques, EGSR’07, 277–286. Google ScholarDigital Library
23. Lawrence, J., Rusinkiewicz, S., and Ramamoorthi, R. 2004. Efficient BRDF importance sampling using a factored representation. ACM Trans. Graph. 23, 3 (Aug.), 496–505. Google ScholarDigital Library
24. Le Cun, Y., Bottou, L., Orr, G. B., and Müller, K.-R. 1998. Efficient backprop. In Neural Networks, Tricks of the Trade, Lecture Notes in Computer Science LNCS 1524. Springer Verlag.Google Scholar
25. Lee, M., and Redner, R. 1990. A note on the use of nonlinear filtering in computer graphics. IEEE Computer Graphics and Applications 10, 3 (May), 23–29. Google ScholarDigital Library
26. Lehtinen, J., Aila, T., Chen, J., Laine, S., and Durand, F. 2011. Temporal light field reconstruction for rendering distribution effects. ACM Trans. Graph. 30, 4 (Aug.), 55:1–55:12. Google ScholarDigital Library
27. Li, T.-M., Wu, Y.-T., and Chuang, Y.-Y. 2012. Sure-based optimization for adaptive sampling and reconstruction. ACM Trans. Graph. 31, 6 (Nov.), 194:1–194:9. Google ScholarDigital Library
28. McCool, M. D. 1999. Anisotropic diffusion for Monte Carlo noise reduction. ACM Trans. Graph. 18, 2 (Apr.), 171–194. Google ScholarDigital Library
29. Mehta, S., Wang, B., and Ramamoorthi, R. 2012. Axis-aligned filtering for interactive sampled soft shadows. ACM Trans. Graph. 31, 6 (Nov.), 163:1–163:10. Google ScholarDigital Library
30. Mehta, S. U., Wang, B., Ramamoorthi, R., and Durand, F. 2013. Axis-aligned filtering for interactive physically-based diffuse indirect lighting. ACM Trans. Graph. 32, 4 (July), 96:1–96:12. Google ScholarDigital Library
31. Mehta, S. U., Yao, J., Ramamoorthi, R., and Durand, F. 2014. Factored axis-aligned filtering for rendering multiple distribution effects. ACM Trans. Graph. 33, 4 (July), 57:1–57:12. Google ScholarDigital Library
32. Mitchell, D. P. 1987. Generating antialiased images at low sampling densities. SIGGRAPH Comput. Graph. 21, 4 (Aug.), 65–72. Google ScholarDigital Library
33. Moon, B., Carr, N., and Yoon, S.-E. 2014. Adaptive rendering based on weighted local regression. ACM Trans. Graph. 33, 5 (Sept.), 170:1–170:14. Google ScholarDigital Library
34. Nowrouzezahrai, D., Kalogerakis, E., and Fiume, E. 2009. Shadowing dynamic scenes with arbitrary BRDFs. Computer Graphics Forum 28, 2, 249–258.Google ScholarCross Ref
35. Overbeck, R. S., Donner, C., and Ramamoorthi, R. 2009. Adaptive wavelet rendering. ACM Trans. Graph. 28, 5 (Dec.), 140:1–140:12. Google ScholarDigital Library
36. Pharr, M., and Humphreys, G. 2010. Physically Based Rendering: From Theory to Implementation, second ed. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA. Google ScholarDigital Library
37. Ren, P., Wang, J., Gong, M., Lin, S., Tong, X., and Guo, B. 2013. Global illumination with radiance regression functions. ACM Trans. Graph. 32, 4 (July), 130:1–130:12. Google ScholarDigital Library
38. Riedmiller, M., and Braun, H. 1993. A direct adaptive method for faster backpropagation learning: the RPROP algorithm. In IEEE International Conference on Neural Networks, 586–591 vol.1.Google Scholar
39. Rousselle, F., Knaus, C., and Zwicker, M. 2011. Adaptive sampling and reconstruction using greedy error minimization. ACM Trans. Graph. 30, 6 (Dec.), 159:1–159:12. Google ScholarDigital Library
40. Rousselle, F., Knaus, C., and Zwicker, M. 2012. Adaptive rendering with non-local means filtering. ACM Trans. Graph. 31, 6 (Nov.), 195:1–195:11. Google ScholarDigital Library
41. Rousselle, F., Manzi, M., and Zwicker, M. 2013. Robust denoising using feature and color information. Computer Graphics Forum 32, 7, 121–130.Google ScholarCross Ref
42. Rumelhart, D. E., Hinton, G. E., and Williams, R. J. 1986. Learning representations by back-propagating errors. Nature 323 (Oct.), 533–536.Google ScholarCross Ref
43. Rushmeier, H. E., and Ward, G. J. 1994. Energy preserving non-linear filters. In ACM SIGGRAPH ’94, 131–138. Google ScholarDigital Library
44. Segovia, B., Iehl, J. C., Mitanchey, R., and Péroche, B. 2006. Non-interleaved deferred shading of interleaved sample patterns. In Proceedings of the 21st ACM SIGGRAPH/EUROGRAPHICS Symposium on Graphics Hardware, ACM, New York, NY, USA, GH ’06, 53–60. Google ScholarDigital Library
45. Sen, P., and Darabi, S. 2011. Implementation of random parameter filtering. Tech. Rep. EECE-TR-11-0004, University of New Mexico.Google Scholar
46. Sen, P., and Darabi, S. 2012. On filtering the noise from the random parameters in Monte Carlo rendering. ACM Trans. Graph. 31, 3 (June), 18:1–18:15. Google ScholarDigital Library
47. Shirley, P., Aila, T., Cohen, J., Enderton, E., Laine, S., Luebke, D., and McGuire, M. 2011. A local image reconstruction algorithm for stochastic rendering. In Symposium on Interactive 3D Graphics and Games, ACM, New York, NY, USA, I3D ’11, 9–14. Google ScholarDigital Library
48. Shirley, P. 1991. Discrepancy as a quality measure for sample distributions. In Proc. Eurographics, vol. 91, 183–194.Google Scholar
49. Stein, C. M. 1981. Estimation of the mean of a multivariate normal distribution. The Annals of Statistics 9, 6, 1135–1151.Google ScholarCross Ref
50. Suykens, J., and Vandewalle, J. 1999. Least squares support vector machine classifiers. Neural Processing Letters 9, 3, 293–300. Google ScholarDigital Library
51. Tomasi, C., and Manduchi, R. 1998. Bilateral filtering for gray and color images. In Sixth International Conference on Computer Vision, 839–846. Google ScholarDigital Library
52. Veach, E., and Guibas, L. J. 1995. Optimally combining sampling techniques for Monte Carlo rendering. In ACM SIGGRAPH ’95, ACM, New York, NY, USA, 419–428. Google ScholarDigital Library
53. Veach, E., and Guibas, L. J. 1997. Metropolis light transport. In ACM SIGGRAPH ’97, ACM, New York, NY, USA, 65–76. Google ScholarDigital Library
54. Wang, Z., Bovik, A., Sheikh, H., and Simoncelli, E. 2004. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing 13, 4 (April), 600–612. Google ScholarDigital Library
55. Xu, R., and Pattanaik, S. N. 2005. A novel Monte Carlo noise reduction operator. IEEE Computer Graphics and Applications 25, 31–35. Google ScholarDigital Library