“A machine learning approach for filtering Monte Carlo noise” by Kalantari, Bako and Sen

  • ©Nima Khademi Kalantari, Steve Bako, and Pradeep Sen




    A machine learning approach for filtering Monte Carlo noise



    The most successful approaches for filtering Monte Carlo noise use feature-based filters (e.g., cross-bilateral and cross non-local means filters) that exploit additional scene features such as world positions and shading normals. However, their main challenge is finding the optimal weights for each feature in the filter to reduce noise but preserve scene detail. In this paper, we observe there is a complex relationship between the noisy scene data and the ideal filter parameters, and propose to learn this relationship using a nonlinear regression model. To do this, we use a multilayer perceptron neural network and combine it with a matching filter during both training and testing. To use our framework, we first train it in an offline process on a set of noisy images of scenes with a variety of distributed effects. Then at run-time, the trained network can be used to drive the filter parameters for new scenes to produce filtered images that approximate the ground truth. We demonstrate that our trained network can generate filtered images in only a few seconds that are superior to previous approaches on a wide range of distributed effects such as depth of field, motion blur, area lighting, glossy reflections, and global illumination.


    1. Bala, K., Walter, B., and Greenberg, D. P. 2003. Combining edges and points for interactive high-quality rendering. ACM Trans. Graph. 22, 3 (July), 631–640. Google ScholarDigital Library
    2. Bauszat, P., Eisemann, M., and Magnor, M. 2011. Guided image filtering for interactive high-quality global illumination. Computer Graphics Forum 30, 4, 1361–1368.Google ScholarDigital Library
    3. Buades, A., Coll, B., and Morel, J. M. 2005. A review of image denoising algorithms, with a new one. Multiscale Modeling & Simulation 4, 2, 490–530.Google ScholarCross Ref
    4. Burger, H., Schuler, C., and Harmeling, S. 2012. Image denoising: Can plain neural networks compete with BM3D? In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2392–2399. Google ScholarDigital Library
    5. Chen, J., Wang, B., Wang, Y., Overbeck, R. S., Yong, J.-H., and Wang, W. 2011. Efficient depth-of-field rendering with adaptive sampling and multiscale reconstruction. Computer Graphics Forum 30, 6, 1667–1680.Google ScholarCross Ref
    6. Cook, R. L., Porter, T., and Carpenter, L. 1984. Distributed ray tracing. SIGGRAPH Comput. Graph. 18, 3 (Jan.), 137–145. Google ScholarDigital Library
    7. Dachsbacher, C. 2011. Analyzing visibility configurations. IEEE Transactions on Visualization and Computer Graphics 17, 4 (April), 475–486. Google ScholarDigital Library
    8. Dammertz, H., Sewtz, D., Hanika, J., and Lensch, H. P. 2010. Edge-avoiding À-trous wavelet transform for fast global illumination filtering. In Proceedings of High Performance Graphics (HPG), 67–75. Google ScholarDigital Library
    9. Delbracio, M., Musé, P., Buades, A., Chauvier, J., Phelps, N., and Morel, J.-M. 2014. Boosting Monte Carlo rendering by ray histogram fusion. ACM Trans. Graph. 33, 1 (Feb.), 8:1–8:15. Google ScholarDigital Library
    10. Drucker, H., Burges, C. J. C., Kaufman, L., Smola, A., and Vapnik, V. 1997. Support vector regression machines. In Advances in Neural Information Processing Systems 9, MIT Press, 155–161.Google Scholar
    11. Dutré, P., Bala, K., Bekaert, P., and Shirley, P. 2006. Advanced Global Illumination. AK Peters Ltd. Google ScholarDigital Library
    12. Egan, K., Tseng, Y.-T., Holzschuch, N., Durand, F., and Ramamoorthi, R. 2009. Frequency analysis and sheared reconstruction for rendering motion blur. ACM Trans. Graph. 28, 3 (July), 93:1–93:13. Google ScholarDigital Library
    13. Egan, K., Durand, F., and Ramamoorthi, R. 2011. Practical filtering for efficient ray-traced directional occlusion. ACM Trans. Graph. 30, 6 (Dec.), 180:1–180:10. Google ScholarDigital Library
    14. Egan, K., Hecht, F., Durand, F., and Ramamoorthi, R. 2011. Frequency analysis and sheared filtering for shadow light fields of complex occluders. ACM Trans. Graph. 30, 2 (Apr.), 9:1–9:13. Google ScholarDigital Library
    15. Grzeszczuk, R., Terzopoulos, D., and Hinton, G. 1998. Neuroanimator: Fast neural network emulation and control of physics-based models. In ACM SIGGRAPH ’98, ACM, New York, NY, USA, 9–20. Google ScholarDigital Library
    16. Hachisuka, T., Jarosz, W., Weistroffer, R. P., Dale, K., Humphreys, G., Zwicker, M., and Jensen, H. W. 2008. Multidimensional adaptive sampling and reconstruction for ray tracing. ACM Trans. Graph. 27 (Aug.), 33:1–33:10. Google ScholarDigital Library
    17. Hastie, T., Tibshirani, R., Friedman, J., Hastie, T., Friedman, J., and Tibshirani, R. 2009. The elements of statistical learning, vol. 2. Springer.Google Scholar
    18. Jakob, W., Regg, C., and Jarosz, W. 2011. Progressive expectation-maximization for hierarchical volumetric photon mapping. Computer Graphics Forum 30, 4, 1287–1297.Google ScholarDigital Library
    19. Jensen, H. W., and Christensen, N. J. 1995. Optimizing path tracing using noise reduction filters. In Winter School of Computer Graphics (WSCG) 1995, 134–142.Google Scholar
    20. Jensen, H. W. 2001. Realistic Image Synthesis Using Photon Mapping. A. K. Peters, Ltd., Natick, MA, USA. Google ScholarDigital Library
    21. Kalantari, N. K., and Sen, P. 2013. Removing the noise in Monte Carlo rendering with general image denoising algorithms. Computer Graphics Forum 32, 2pt1, 93–102.Google Scholar
    22. Laine, S., Saransaari, H., Kontkanen, J., Lehtinen, J., and Aila, T. 2007. Incremental instant radiosity for real-time indirect illumination. In Proceedings of the 18th Eurographics Conference on Rendering Techniques, EGSR’07, 277–286. Google ScholarDigital Library
    23. Lawrence, J., Rusinkiewicz, S., and Ramamoorthi, R. 2004. Efficient BRDF importance sampling using a factored representation. ACM Trans. Graph. 23, 3 (Aug.), 496–505. Google ScholarDigital Library
    24. Le Cun, Y., Bottou, L., Orr, G. B., and Müller, K.-R. 1998. Efficient backprop. In Neural Networks, Tricks of the Trade, Lecture Notes in Computer Science LNCS 1524. Springer Verlag.Google Scholar
    25. Lee, M., and Redner, R. 1990. A note on the use of nonlinear filtering in computer graphics. IEEE Computer Graphics and Applications 10, 3 (May), 23–29. Google ScholarDigital Library
    26. Lehtinen, J., Aila, T., Chen, J., Laine, S., and Durand, F. 2011. Temporal light field reconstruction for rendering distribution effects. ACM Trans. Graph. 30, 4 (Aug.), 55:1–55:12. Google ScholarDigital Library
    27. Li, T.-M., Wu, Y.-T., and Chuang, Y.-Y. 2012. Sure-based optimization for adaptive sampling and reconstruction. ACM Trans. Graph. 31, 6 (Nov.), 194:1–194:9. Google ScholarDigital Library
    28. McCool, M. D. 1999. Anisotropic diffusion for Monte Carlo noise reduction. ACM Trans. Graph. 18, 2 (Apr.), 171–194. Google ScholarDigital Library
    29. Mehta, S., Wang, B., and Ramamoorthi, R. 2012. Axis-aligned filtering for interactive sampled soft shadows. ACM Trans. Graph. 31, 6 (Nov.), 163:1–163:10. Google ScholarDigital Library
    30. Mehta, S. U., Wang, B., Ramamoorthi, R., and Durand, F. 2013. Axis-aligned filtering for interactive physically-based diffuse indirect lighting. ACM Trans. Graph. 32, 4 (July), 96:1–96:12. Google ScholarDigital Library
    31. Mehta, S. U., Yao, J., Ramamoorthi, R., and Durand, F. 2014. Factored axis-aligned filtering for rendering multiple distribution effects. ACM Trans. Graph. 33, 4 (July), 57:1–57:12. Google ScholarDigital Library
    32. Mitchell, D. P. 1987. Generating antialiased images at low sampling densities. SIGGRAPH Comput. Graph. 21, 4 (Aug.), 65–72. Google ScholarDigital Library
    33. Moon, B., Carr, N., and Yoon, S.-E. 2014. Adaptive rendering based on weighted local regression. ACM Trans. Graph. 33, 5 (Sept.), 170:1–170:14. Google ScholarDigital Library
    34. Nowrouzezahrai, D., Kalogerakis, E., and Fiume, E. 2009. Shadowing dynamic scenes with arbitrary BRDFs. Computer Graphics Forum 28, 2, 249–258.Google ScholarCross Ref
    35. Overbeck, R. S., Donner, C., and Ramamoorthi, R. 2009. Adaptive wavelet rendering. ACM Trans. Graph. 28, 5 (Dec.), 140:1–140:12. Google ScholarDigital Library
    36. Pharr, M., and Humphreys, G. 2010. Physically Based Rendering: From Theory to Implementation, second ed. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA. Google ScholarDigital Library
    37. Ren, P., Wang, J., Gong, M., Lin, S., Tong, X., and Guo, B. 2013. Global illumination with radiance regression functions. ACM Trans. Graph. 32, 4 (July), 130:1–130:12. Google ScholarDigital Library
    38. Riedmiller, M., and Braun, H. 1993. A direct adaptive method for faster backpropagation learning: the RPROP algorithm. In IEEE International Conference on Neural Networks, 586–591 vol.1.Google Scholar
    39. Rousselle, F., Knaus, C., and Zwicker, M. 2011. Adaptive sampling and reconstruction using greedy error minimization. ACM Trans. Graph. 30, 6 (Dec.), 159:1–159:12. Google ScholarDigital Library
    40. Rousselle, F., Knaus, C., and Zwicker, M. 2012. Adaptive rendering with non-local means filtering. ACM Trans. Graph. 31, 6 (Nov.), 195:1–195:11. Google ScholarDigital Library
    41. Rousselle, F., Manzi, M., and Zwicker, M. 2013. Robust denoising using feature and color information. Computer Graphics Forum 32, 7, 121–130.Google ScholarCross Ref
    42. Rumelhart, D. E., Hinton, G. E., and Williams, R. J. 1986. Learning representations by back-propagating errors. Nature 323 (Oct.), 533–536.Google ScholarCross Ref
    43. Rushmeier, H. E., and Ward, G. J. 1994. Energy preserving non-linear filters. In ACM SIGGRAPH ’94, 131–138. Google ScholarDigital Library
    44. Segovia, B., Iehl, J. C., Mitanchey, R., and Péroche, B. 2006. Non-interleaved deferred shading of interleaved sample patterns. In Proceedings of the 21st ACM SIGGRAPH/EUROGRAPHICS Symposium on Graphics Hardware, ACM, New York, NY, USA, GH ’06, 53–60. Google ScholarDigital Library
    45. Sen, P., and Darabi, S. 2011. Implementation of random parameter filtering. Tech. Rep. EECE-TR-11-0004, University of New Mexico.Google Scholar
    46. Sen, P., and Darabi, S. 2012. On filtering the noise from the random parameters in Monte Carlo rendering. ACM Trans. Graph. 31, 3 (June), 18:1–18:15. Google ScholarDigital Library
    47. Shirley, P., Aila, T., Cohen, J., Enderton, E., Laine, S., Luebke, D., and McGuire, M. 2011. A local image reconstruction algorithm for stochastic rendering. In Symposium on Interactive 3D Graphics and Games, ACM, New York, NY, USA, I3D ’11, 9–14. Google ScholarDigital Library
    48. Shirley, P. 1991. Discrepancy as a quality measure for sample distributions. In Proc. Eurographics, vol. 91, 183–194.Google Scholar
    49. Stein, C. M. 1981. Estimation of the mean of a multivariate normal distribution. The Annals of Statistics 9, 6, 1135–1151.Google ScholarCross Ref
    50. Suykens, J., and Vandewalle, J. 1999. Least squares support vector machine classifiers. Neural Processing Letters 9, 3, 293–300. Google ScholarDigital Library
    51. Tomasi, C., and Manduchi, R. 1998. Bilateral filtering for gray and color images. In Sixth International Conference on Computer Vision, 839–846. Google ScholarDigital Library
    52. Veach, E., and Guibas, L. J. 1995. Optimally combining sampling techniques for Monte Carlo rendering. In ACM SIGGRAPH ’95, ACM, New York, NY, USA, 419–428. Google ScholarDigital Library
    53. Veach, E., and Guibas, L. J. 1997. Metropolis light transport. In ACM SIGGRAPH ’97, ACM, New York, NY, USA, 65–76. Google ScholarDigital Library
    54. Wang, Z., Bovik, A., Sheikh, H., and Simoncelli, E. 2004. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing 13, 4 (April), 600–612. Google ScholarDigital Library
    55. Xu, R., and Pattanaik, S. N. 2005. A novel Monte Carlo noise reduction operator. IEEE Computer Graphics and Applications 25, 31–35. Google ScholarDigital Library

ACM Digital Library Publication:

Overview Page: