“DeepToF: off-the-shelf real-time correction of multipath interference in time-of-flight imaging” by Marco, Hernandez, Muñoz, Dong, Jarabo, et al. …
Conference:
Type(s):
Title:
- DeepToF: off-the-shelf real-time correction of multipath interference in time-of-flight imaging
Session/Category Title:
- High Performance Imaging
Presenter(s)/Author(s):
Abstract:
Time-of-flight (ToF) imaging has become a widespread technique for depth estimation, allowing affordable off-the-shelf cameras to provide depth maps in real time. However, multipath interference (MPI) resulting from indirect illumination significantly degrades the captured depth. Most previous works have tried to solve this problem by means of complex hardware modifications or costly computations. In this work, we avoid these approaches and propose a new technique to correct errors in depth caused by MPI, which requires no camera modifications and takes just 10 milliseconds per frame. Our observations about the nature of MPI suggest that most of its information is available in image space; this allows us to formulate the depth imaging process as a spatially-varying convolution and use a convolutional neural network to correct MPI errors. Since the input and output data present similar structure, we base our network on an autoencoder, which we train in two stages. First, we use the encoder (convolution filters) to learn a suitable basis to represent MPI-corrupted depth images; then, we train the decoder (deconvolution filters) to correct depth from synthetic scenes, generated by using a physically-based, time-resolved renderer. This approach allows us to tackle a key problem in ToF, the lack of ground-truth data, by using a large-scale captured training set with MPI-corrupted depth to train the encoder, and a smaller synthetic training set with ground truth depth to train the decoder stage of the network. We demonstrate and validate our method on both synthetic and real complex scenarios, using an off-the-shelf ToF camera, and with only the captured, incorrect depth as input.
References:
1. Ayush Bhandari, Achuta Kadambi, Refael Whyte, Christopher Barsi, Micha Feigin, Adrian Dorrington, and Ramesh Raskar. 2014. Resolving multipath interference in time-of-flight imaging via modulation frequency diversity and sparse regularization. Opt. Lett. 39, 6 (2014), 1705–1708. Cross Ref
2. Jean-Yves Bouguet. 2004. Camera calibration toolbox for Matlab. (2004).
3. Inchang Choi, Daniel S. Jeon, Giljoo Nam, Diego Gutierrez, and Min H. Kim. 2017. High-Quality Hyperspectral Reconstruction Using a Spectral Prior. ACM Transactions on Graphics (SIGGRAPH Asia 2017) 36, 6 (2017).
4. A. A. Dorrington, J. P. Godbaz, M. J. Cree, A. D. Payne, and L. V. Streeter. 2011. Separating true range measurements from multi-path and scattering interference in commercial range cameras. In Proceedings of SPIE, Vol. 7864. 786404-786404-10.
5. B. Du, W. Xiong, J. Wu, L. Zhang, L. Zhang, and D. Tao. 2016. Stacked Convolutional Denoising Auto-Encoders for Feature Representation. IEEE Trans. Cybernetics 99 (2016), 1–11.
6. David Eigen and Rob Fergus. 2015. Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. In Proceedings of ICCV.
7. David Eigen, Christian Puhrsch, and Rob Fergus. 2014. Depth map prediction from a single image using a multi-scale deep network. In Proceedings of NIPS.
8. Dominik Maria Endres and Johannes E Schindelin. 2003. A new metric for probability distributions. IEEE Transactions on Information theory 49, 7 (2003), 1858–1860.
9. M. Feigin, A. Bhandari, S. Izadi, C. Rhemann, M. Schmidt, and R. Raskar. 2016. Resolving Multipath Interference in Kinect: An Inverse Problem Approach. IEEE Sensors Journal 16, 10 (May 2016), 3419–3427. Cross Ref
10. Daniel Freedman, Yoni Smolin, Eyal Krupka, Ido Leichter, and Mirko Schmidt. 2014. SRA: Fast removal of general multipath for ToF sensors. In Proceedings of ECCV. Springer, 234–249.
11. Stefan Fuchs. 2010. Multipath Interference Compensation in Time-of-Flight Camera Images. In Proceedings of the International Conference on Pattern Recognition. 3583–3586.
12. Stefan Fuchs, Michael Suppa, and Olaf Hellwich. 2013. Compensation for Multipath in ToF Camera Measurements Supported by Photometric Calibration and Environment Integration. In Proceedings of the International Conference on Computer Vision Systems (ICVS’13). Springer-Verlag, Berlin, Heidelberg, 31–41.
13. John P. Godbaz, Michael J. Cree, and Adrian A. Dorrington. 2012. Closed-form inverses for the mixed pixel/multipath interference problem in AMCW lidar. In Proceedings of SPIE, Vol. 8296. 829618-829618-15.
14. Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. MIT Press. http://www.deeplearningbook.org.
15. Mohit Gupta, Shree K. Nayar, Matthias B. Hullin, and Jaime Martin. 2015. Phasor Imaging: A Generalization of Correlation-Based Time-of-Flight Imaging. ACM Trans. Graph. 34, 5, Article 156 (Nov. 2015), 18 pages.
16. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of CVPR. 770–778. Cross Ref
17. Felix Heide, Matthias B. Hullin, James Gregson, and Wolfgang Heidrich. 2013. Low-budget Transient Imaging Using Photonic Mixer Devices. ACM Trans. Graph. 32, 4, Article 45 (July 2013), 10 pages.
18. Jinggang Huang, Ann B Lee, and David Mumford. 2000. Statistics of range images. In Proceedings of CVPR, Vol. 1. IEEE, 324–331.
19. Jinggang Huang and David Mumford. 1999. Statistics of natural images and models. In Proceedings of CVPR, Vol. 1. IEEE, 541–547.
20. Adrian Jarabo, Julio Marco, Adolfo Muhoz, Raul Buisan, Wojciech Jarosz, and Diego Gutierrez. 2014. A Framework for Transient Rendering. ACM Trans. Graph. 33, 6, Article 177 (2014).
21. Adrian Jarabo, Belen Masia, Julio Marco, and Diego Gutierrez. 2017. Recent Advances in Transient Imaging: A Computer Graphics and Vision Perspective. Visual Informatics 1, 1 (2017).
22. David Jiménez, Daniel Pizarro, Manuel Mazo, and Sira Palazuelos. 2014. Modeling and correction of multipath interference in time-of-flight cameras. Image and Vision Computing 32, 1 (2014), 1–13.
23. Achuta Kadambi, Refael Whyte, Ayush Bhandari, Lee Streeter, Christopher Barsi, Adrian Dorrington, and Ramesh Raskar. 2013. Coded Time of Flight Cameras: Sparse Deconvolution to Address Multipath Interference and Recover Time Profiles. ACM Trans. Graph. 32, 6, Article 167 (Nov. 2013), 10 pages.
24. Nima Khademi Kalantari, Ting-Chun Wang, and Ravi Ramamoorthi. 2016. Learning-based View Synthesis for Light Field Cameras. ACM Trans. Graph. 35, 6, Article 193 (Nov. 2016), 10 pages.
25. S Karayev, Y Jia, J Barron, M Fritz, K Saenko, and T Darrell. 2011. A category-level 3-D object dataset: putting the Kinect to work. In Proceedings of the IEEE International Conference on Computer Vision Workshops. 1167–1174.
26. Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv.1412.6980 (2014).
27. Ahmed Kirmani, Arrigo Benedetti, and Philip A Chou. 2013. Spumic: Simultaneous phase unwrapping and multipath interference cancellation in time-of-flight cameras using spectral methods. In Proceedings of IEEE International Conference on Multimedia and Expo. 1–6. Cross Ref
28. Nick Knighton and Bruce Bugbee. 2005. A mixture of barium sulfate and white paint is a low-cost substitute reflectance standard for Spectralon®. (2005).
29. Ann B Lee, JG Huang, and DB Mumford. 2000. Random collage model for natural images. Int. J. of Computer Vision (2000).
30. Bo Li, Chunhua Shen, Yuchao Dai, A. van den Hengel, and Mingyi He. 2015. Depth and surface normal estimation from monocular images using regression on deep features and hierarchical CRFs. In Proceedings of CVPR.
31. Jianhua Lin. 1991. Divergence measures based on the Shannon entropy. IEEE Transactions on Information theory 37, 1 (1991), 145–151.
32. Marvin Lindner, Ingo Schiller, Andreas Kolb, and Reinhard Koch. 2010. Time-of-flight sensor calibration for accurate range sensing. Computer Vision and Image Understanding 114, 12 (2010), 1318–1328.
33. Fayao Liu, Chunhua Shen, and Guosheng Lin. 2015. Deep Convolutional Neural Fields for Depth Estimation from a Single Image. In Proceedings of CVPR. Cross Ref
34. Jonathan Masci, Ueli Meier, Dan Cireşan, and Jürgen Schmidhuber. 2011. Stacked convolutional auto-encoders for hierarchical feature extraction. In Proceedings of Int. Conf. Artificial Neural Networks. 52–59.
35. Nikhil Naik, Achuta Kadambi, Christoph Rhemann, Shahram Izadi, Ramesh Raskar, and Sing Bing Kang. 2015. A light transport model for mitigating multipath interference in time-of-flight sensors. In Proceedings of CVPR. 73–81. Cross Ref
36. Ren Ng, Ravi Ramamoorthi, and Pat Hanrahan. 2003. All-frequency shadows using non-linear wavelet lighting approximation. ACM Trans. Graph. 22, 3 (2003).
37. Matthew O’Toole, Felix Heide, Lei Xiao, Matthias B. Hullin, Wolfgang Heidrich, and Kiriakos N. Kutulakos. 2014. Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport. ACM Trans. Graph. 33, 4, Article 87 (July 2014), 11 pages.
38. EM Patterson, CE Shelden, and BH Stockton. 1977. Kubelka-Munk optical properties of a barium sulfate white reflectance standard. Applied Optics 16, 3 (1977), 729–732. Cross Ref
39. Ofir Pele and Michael Werman. 2010. The quadratic-chi histogram distance family. In Proceedings of ECCV. Springer, 749–762.
40. Christoph Peters, Jonathan Klein, Matthias B. Hullin, and Reinhard Klein. 2015. Solving Trigonometric Moment Problems for Fast Transient Imaging. ACM Trans. Graph. 34, 6 (Nov. 2015).
41. Hui Qiao, Jingyu Lin, Yebin Liu, Matthias B Hullin, and Qionghai Dai. 2015. Resolving transient time profile in ToF imaging via log-sum sparse regularization. Opt. Lett. 40, 6 (2015). Cross Ref
42. Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. 2012. Indoor segmentation and support inference from RGBD images. Computer Vision—ECCV 2012 (2012), 746–760.
43. Hao Su, Haoqiang Fan, and Leonidas Guibas. 2017. A Point Set Generation Network for 3D Object Reconstruction from a Single Image. In Proceedings of CVPR.
44. Andreas Velten, Di Wu, Adrian Jarabo, Belen Masia, Christopher Barsi, Chinmaya Joshi, Everett Lawson, Moungi G. Bawendi, Diego Gutierrez, and Ramesh Raskar. 2013. Femto-Photography: Capturing and Visualizing the Propagation of Light. ACM Transactions on Graphics (SIGGRAPH 2013) 32, 4 (2013).
45. Peng Wang, Xiaohui Shen, Zhe Lin, S. Cohen, B. Price, and A. Yuille. 2015. Towards unified depth and semantic prediction from a single image. In Proceedings of CVPR.
46. Di Wu, Andreas Velten, Matthew O’Toole, Belen Masia, Amit Agrawal, Qionghai Dai, and Ramesh Raskar. 2014. Decomposing Global Light Transport Using Time of Flight Imaging. International Journal of Computer Vision 107, 2 (April 2014), 123 — 138.
47. Jianxiong Xiao, Andrew Owens, and Antonio Torralba. 2013. Sun3d: A database of big spaces reconstructed using sfm and object labels. In Proceedings of ICCV. 1625–1632.
48. Dan Xu, Elisa Ricci, Wanli Ouyang, Xiaogang Wang, and Nicu Sebe. 2017. Multi-Scale Continuous CRFs as Sequential Deep Networks for Monocular Depth Estimation. In Proceedings of CVPR. Cross Ref
49. Jianchao Yang, John Wright, Thomas S. Huang, and Yi Ma. 2010. Image Super-Resolution Via Sparse Representation. IEEE TIP 19, 11 (2010).
50. Jure Žbontar and Yann LeCun. 2015. Computing the Stereo Matching Cost with a Convolutional Neural Network. In Proceedings of CVPR. Cross Ref
