“Interactive reconstruction of Monte Carlo image sequences using a recurrent denoising autoencoder” by Chaitanya, Kaplanyan, Schied, Salvi, Lefohn, et al. …

  • ©Chakravarty R. Alla Chaitanya, Anton S. Kaplanyan, Christoph Schied, Marco Salvi, Aaron E. Lefohn, Derek Nowrouzezahrai, and Timo Aila

Conference:


Type:


Title:

    Interactive reconstruction of Monte Carlo image sequences using a recurrent denoising autoencoder

Session/Category Title: Rendering Systems

Presenter(s)/Author(s):


Moderator(s):


Abstract:


    We describe a machine learning technique for reconstructing image sequences rendered using Monte Carlo methods. Our primary focus is on reconstruction of global illumination with extremely low sampling budgets at interactive rates. Motivated by recent advances in image restoration with deep convolutional networks, we propose a variant of these networks better suited to the class of noise present in Monte Carlo rendering. We allow for much larger pixel neighborhoods to be taken into account, while also improving execution speed by an order of magnitude. Our primary contribution is the addition of recurrent connections to the network in order to drastically improve temporal stability for sequences of sparsely sampled input images. Our method also has the desirable property of automatically modeling relationships based on auxiliary per-pixel input channels, such as depth and normals. We show significantly higher quality results compared to existing methods that run at comparable speeds, and furthermore argue a clear path for making our method run at realtime rates in the near future.

References:


    1. Bako, S., Vogels, T., McWilliams, B., Meyer, M., Novák, J., Harvill, A., Sen, P., DeRose, T., and Rousselle, F. 2017. Kernel-Predicting Convolutional Networks for Denoising Monte Carlo Renderings. ACM Trans. Graph. (2017).Google Scholar
    2. Bauszat, P., Eisemann, M., John, S., and Magnor, M. 2015. Sample-Based Manifold Filtering for Interactive Global Illumination and Depth of Field. Comp. Graph. Forum 34, 1 (2015).Google ScholarDigital Library
    3. Bauszat, P., Eisemann, M., and Magnor, M. 2011. Guided Image Filtering for Interactive High-quality Global Illumination (Comp. Graph. Forum). 8.Google Scholar
    4. Bikker, J. and van Schijndel, J. 2013. The brigade renderer: A path tracer for real-time games. J. Comp. Games Tech. 8 (2013).Google Scholar
    5. Bitterli, B., Rousselle, F., Moon, B., Iglesias-Guitián, J. A., Adler, D., Mitchell, K., Jarosz, W., and Novák, J. 2016. Nonlinearly Weighted First-order Regression for Denoising Monte Carlo Renderings. Comp. Graph. Forum 35, 4 (2016).Google Scholar
    6. Dammertz, H., Sewtz, D., Hanika, J., and Lensch, H. 2010. Edge-avoiding À-Trous wavelet transform for fast global illumination filtering. In Proc. High Perf. Graph.Google Scholar
    7. Deering, M., Winner, S., Schediwy, B., Duffy, C., and Hunt, N. 1988. The Triangle Processor and Normal Vector Shader: A VLSI System for High Performance Graphics. Proc. SIGGRAPH 22, 4 (1988).Google Scholar
    8. Dieleman, S., Schlüter, J., Raffel, C., Olson, E., Sønderby, S. K., and others. 2015. Lasagne: First release. (2015).Google Scholar
    9. Dosovitskiy, A., Springenberg, J. T., and Brox, T. 2015. Learning to generate chairs with convolutional neural networks. In Proc. CVPR. 1538–1546. Google ScholarCross Ref
    10. Egan, K., Durand, F., and Ramamoorthi, R. 2011a. Practical Filtering for Efficient Ray-traced Directional Occlusion. ACM Trans. Graph. 30, 6 (2011). Google ScholarDigital Library
    11. Egan, K., Hecht, F., Durand, F., and Ramamoorthi, R. 2011b. Frequency Analysis and Sheared Filtering for Shadow Light Fields of Complex Occluders. ACM Trans. Graph. 30, 2 (2011). Google ScholarDigital Library
    12. Egan, K., Tseng, Y.-T., Holzschuch, N., Durand, F., and Ramamoorthi, R. 2009. Frequency Analysis and Sheared Reconstruction for Rendering Motion Blur. ACM Trans. Graph. 28, 3 (2009). Google ScholarDigital Library
    13. Fischer, P., Dosovitskiy, A., Ilg, E., Häusser, P., Hazirbas, C., Golkov, V., van der Smagt, P., Cremers, D., and Brox, T. 2015. FlowNet: Learning Optical Flow with Convolutional Networks. ArXiv e-prints (2015). arXiv:1504.06852.Google Scholar
    14. Flynn, J., Neulander, I., Philbin, J., and Snavely, N. 2016. DeepStereo: Learning to Predict New Views From the World’s Imagery. In Proc. CVPR.Google Scholar
    15. Gastal, E. S. and Oliveira, M. M. 2012. Adaptive manifolds for real-time high-dimensional filtering. ACM Trans. Graph. 31, 4 (2012). Google ScholarDigital Library
    16. Girshick, R., Donahue, J., Darrell, T., and Malik, J. 2014. Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation. In Proc. CVPR. 580–587. Google ScholarDigital Library
    17. Halton, J. H. 1964. Algorithm 247: Radical-inverse Quasi-random Point Sequence. Comm. ACM 7, 12 (1964). Google ScholarDigital Library
    18. Han, S., Liu, X., Mao, H., Pu, J., Pedram, A., Horowitz, M. A., and Dally, W. J. 2016. EIE: Efficient Inference Engine on Compressed Deep Neural Network. In Proc. Comp. Arch.Google Scholar
    19. Hanika, J., Dammertz, H., and Lensch, H. 2011. Edge-Optimized À-Trous Wavelets for Local Contrast Enhancement with Robust Denoising. Comp. Graph. Forum 30, 7 (2011).Google ScholarCross Ref
    20. He, K., Zhang, X., Ren, S., and Sun, J. 2015a. Deep Residual Learning for Image Recognition. ArXiv e-prints (2015). arXiv:1512.03385.Google Scholar
    21. He, K., Zhang, X., Ren, S., and Sun, J. 2015b. Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. In Proc. IEEE Conf. Comp. Vis.Google Scholar
    22. Hill, S., McAuley, S., Burley, B., Chan, D., Fascione, L., Iwanicki, M., Hoffman, N., Jakob, W., Neubelt, D., Pesce, A., and Pettineo, M. 2015. Physically Based Shading in Theory and Practice. In SIGGRAPH Courses. Article 22. Google ScholarDigital Library
    23. Hill, S., McAuley, S., Dupuy, J., Gotanda, Y., Heitz, E., Hoffman, N., Lagarde, S., Langlands, A., Megibben, I., Rayani, F., and de Rousiers, C. 2014. Physically Based Shading in Theory and Practice. In SIGGRAPH Courses. Article 23. Google ScholarDigital Library
    24. Huang, Y., Wang, W., and Wang, L. 2015. Bidirectional recurrent convolutional networks for multi-frame super-resolution. In Proc. NIPS.Google Scholar
    25. Iizuka, S., Simo-Serra, E., and Ishikawa, H. 2016. Joint End-to-end Learning of Global and Local Image Priors for Automatic Image Colorization with Simultaneous Classification. ACM Trans. Graph. 35, 4 (2016). Google ScholarDigital Library
    26. Jensen, H. W. and Christensen, N. J. 1995. Optimizing Path Tracing using Noise Reduction Filters. In Proc. WSCG.Google Scholar
    27. Kajiya, J. T 1986. The rendering equation. In Proc. SIGGRAPH. Google ScholarDigital Library
    28. Kalantari, N. K., Bako, S., and Sen, P. 2015. A machine learning approach for filtering Monte Carlo noise. ACM Trans. Graph. 34, 4 (2015). Google ScholarDigital Library
    29. Kaplanyan, A. S. and Dachsbacher, C. 2013. Path Space Regularization for Holistic and Robust Light Transport. Comp. Graph. Forum 32, 2 (2013).Google ScholarCross Ref
    30. Karis, B. 2014. High-quality temporal supersampling. In Advances in Real-Time Rendering in Games, SIGGRAPH Courses.Google Scholar
    31. Keller, A., Fascione, L., Fajardo, M., Georgiev, I., Christensen, P., Hanika, J., Eisenacher, C., and Nichols, G. 2015. The path tracing revolution in the movie industry. SIGGRAPH Courses (2015).Google Scholar
    32. Kingma, D. and Ba, J. 2014. Adam: A Method for Stochastic Optimization. ArXiv e-prints (2014). arXiv:1412.6980.Google Scholar
    33. Krizhevsky, A., Sutskever, I., and Hinton, G. E. 2012. Imagenet classification with deep convolutional neural networks. In Proc. NIPS.Google ScholarDigital Library
    34. Larsson, G., Maire, M., and Shakhnarovich, G. 2016. Learning Representations for Automatic Colorization. In Proc. ECCV. Google ScholarCross Ref
    35. LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. 1998. Gradient-based learning applied to document recognition. In Proc. IEEE. Google ScholarCross Ref
    36. Ledig, C., Theis, L., Huszar, F., Caballero, J., Aitken, A. P., Tejani, A., Totz, J., Wang, Z., and Shi, W. 2016. Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. ArXiv e-prints (2016). arXiv:1609.04802.Google Scholar
    37. Lee, M. E. and Redner, R. A. 1990. A note on the use of nonlinear filtering in computer graphics. IEEE Comp. Graph. App. 10, 3 (1990). Google ScholarDigital Library
    38. Lehtinen, J., Aila, T., Laine, S., and Durand, F. 2012. Reconstructing the indirect light field for global illumination. ACM Trans. Graph. 31, 4 (2012). Google ScholarDigital Library
    39. Li, T.-M., Wu, Y.-T., and Chuang, Y.-Y. 2012. SURE-based Optimization for Adaptive Sampling and Reconstruction. ACM Trans. Graph. 31, 6 (2012). Google ScholarDigital Library
    40. Maas, A. L., Hannun, A. Y., and Ng, A. Y. 2013. Rectifier nonlinearities improve neural network acoustic models. In International Conference on Machine Learning, Vol. 30.Google Scholar
    41. Mao, X., Shen, C., and Yang, Y. 2016. Image Restoration Using Convolutional Auto-encoders with Symmetric Skip Connections. In Proc. NIPS. arXiv:1606.08921.Google Scholar
    42. McCool, M. D. 1999. Anisotropic diffusion for Monte Carlo noise reduction. ACM Trans. Graph. 18, 2 (1999). Google ScholarDigital Library
    43. Mehta, S. U., Wang, B., and Ramamoorthi, R. 2012. Axis-aligned Filtering for Interactive Sampled Soft Shadows. ACM Trans. Graph. 31, 6 (2012). Google ScholarDigital Library
    44. Mehta, S. U., Wang, B., Ramamoorthi, R., and Durand, F. 2013. Axis-aligned Filtering for Interactive Physically-based Diffuse Indirect Lighting. ACM Trans. Graph. 32, 4 (2013). Google ScholarDigital Library
    45. Mehta, S. U., Yao, J., Ramamoorthi, R., and Durand, F. 2014. Factored Axis-aligned Filtering for Rendering Multiple Distribution Effects. ACM Trans. Graph. 33, 4 (2014). Google ScholarDigital Library
    46. Mishkin, D., Sergievskiy, N., and Matas, J. 2016. Systematic evaluation of CNN advances on the ImageNet. ArXiv e-prints (2016). arXiv:1606.02228.Google Scholar
    47. Moon, B., Iglesias-Guitian, J. A., Yoon, S.-E., and Mitchell, K. 2015. Adaptive rendering with linear predictions. ACM Trans. Graph. 34, 4 (2015). Google ScholarDigital Library
    48. Moon, B., McDonagh, S., Mitchell, K., and Gross, M. 2016. Adaptive polynomial rendering. ACM Trans. Graph. 35, 4 (2016). Google ScholarDigital Library
    49. Munkberg, J., Hasselgren, J., Clarberg, P., Andersson, M., and Akenine-Möller, T 2016. Texture space caching and reconstruction for ray tracing. ACM Trans. Graph. 35, 6 (2016). Google ScholarDigital Library
    50. Parker, S. G., Bigler, J., Dietrich, A., Friedrich, H., Hoberock, J., Luebke, D., McAllister, D., McGuire, M., Morley, K., Robison, A., and Stich, M. 2010. OptiX: A General Purpose Ray Tracing Engine. ACM Trans. Graph. 29, 4 (2010). Google ScholarDigital Library
    51. Pathak, D., Krähenbühl, P., Donahue, J., Darrell, T., and Efros, A. 2016. Context Encoders: Feature Learning by Inpainting. In Proc. CVPR. Google ScholarCross Ref
    52. Patraucean, V., Handa, A., and Cipolla, R. 2015. Spatio-temporal video autoencoder with differentiable memory. ArXiv e-prints (2015). arXiv:1511.06309.Google Scholar
    53. Raiko, T., Valpola, H., and Lecun, Y. 2012. Deep Learning Made Easier by Linear Transformations in Perceptrons, In Proc. Artificial Intelligence and Statistics. J. of Machine Learning Research 22.Google Scholar
    54. Ravishankar, S. and Bresler, Y. 2011. MR Image Reconstruction From Highly Undersampled k-Space Data by Dictionary Learning. IEEE Trans. Med. Imaging 30, 5 (2011), 1028–1041. Google ScholarCross Ref
    55. Robison, A. and Shirley, P. 2009. Image space gathering. In Proc. High Perf Graph. Google ScholarDigital Library
    56. Ronneberger, O., Fischer, P., and Brox, T. 2015. U-Net: Convolutional Networks for Biomedical Image Segmentation. 234–241.Google Scholar
    57. Rousselle, F., Manzi, M., and Zwicker, M. 2013. Robust Denoising using Feature and Color Information. Comp. Graph. Forum 32, 7 (2013). Google ScholarCross Ref
    58. Schmidhuber, J. 2015. Deep Learning in Neural Networks: An Overview. Neural Networks 61 (2015). Google ScholarDigital Library
    59. Schwenk, K. 2013. Filtering techniques for low-noise previews of interactive stochastic ray tracing. Ph.D. Dissertation. Darmstadt University.Google Scholar
    60. Sen, P. and Darabi, S. 2012. On filtering the noise from the random parameters in Monte Carlo rendering. ACM Trans. Graph. 31, 3 (2012). Google ScholarDigital Library
    61. Shi, W., Caballero, J., Huszár, F., Totz, J., Aitken, A. P., Bishop, R., Rueckert, D., and Wang, Z. 2016. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proc. CVPR. Google ScholarCross Ref
    62. Stein, C. M. 1981. Estimation of the mean of a multivariate normal distribution. In Annals of Stat., Vol. 9. Google ScholarCross Ref
    63. Theano Development Team. 2016. Theano: A Python framework for fast computation of mathematical expressions. ArXiv e-prints (2016). arXiv:1605.02688.Google Scholar
    64. Veach, E. 1998. Robust Monte Carlo Methods for Light Transport Simulation. Ph.D. Dissertation. Advisor(s) Guibas, Leonidas J.Google Scholar
    65. Vincent, P., Larochelle, H., Bengio, Y, and Manzagol, P.-A. 2008. Extracting and Composing Robust Features with Denoising Autoencoders. Proc. Machine Learning (2008).Google Scholar
    66. Wang, Z., Bovik, A. C., Sheikh, H. R., and Simoncelli, E. P. 2004. Image Quality Assessment: From Error Visibility to Structural Similarity. Trans. Img. Proc. 13, 4 (2004), 600–612. Google ScholarDigital Library
    67. Werbos, P. J. 1988. Generalization of backpropagation with application to a recurrent gas market model. Neural Networks 1, 4 (1988). Google ScholarCross Ref
    68. Yan, L.-Q., Mehta, S. U., Ramamoorthi, R., and Durand, F. 2015. Fast 4D Sheared Filtering for Interactive Rendering of Distribution Effects. ACM Trans. Graph. 35, 1 (2015). Google ScholarDigital Library
    69. Zhao, H., Gallo, O., Frosio, I., and Kautz, J. 2016. Loss Functions for Image Restoration with Neural Networks. IEEE Transactions on Computational Imaging (2016).Google Scholar
    70. Zimmer, H., Rousselle, F., Jakob, W., Wang, O., Adler, D., Jarosz, W., Sorkine-Hornung, O., and Sorkine-Hornung, A. 2015. Path-space Motion Estimation and Decomposition for Robust Animation Filtering. Comp. Graph. Forum 34, 4 (2015).Google Scholar
    71. Zwicker, M., Jarosz, W., Lehtinen, J., Moon, B., Ramamoorthi, R., Rousselle, F., Sen, P., Soler, C., and Yoon, S.-E. 2015. Recent advances in adaptive sampling and reconstruction for Monte Carlo rendering. Comp. Graph. Forum 34, 2 (2015).Google ScholarCross Ref

ACM Digital Library Publication:


Overview Page: