“Real-time shading-based refinement for consumer depth cameras” by Wu, Zollhöfer, Nießner, Stamminger, Izadi, et al. …
Conference:
Type(s):
Title:
- Real-time shading-based refinement for consumer depth cameras
Session/Category Title: Data In, Surface Out
Presenter(s)/Author(s):
Abstract:
We present the first real-time method for refinement of depth data using shape-from-shading in general uncontrolled scenes. Per frame, our real-time algorithm takes raw noisy depth data and an aligned RGB image as input, and approximates the time-varying incident lighting, which is then used for geometry refinement. This leads to dramatically enhanced depth maps at 30Hz. Our algorithm makes few scene assumptions, handling arbitrary scene objects even under motion. To enable this type of real-time depth map enhancement, we contribute a new highly parallel algorithm that reformulates the inverse rendering optimization problem in prior work, allowing us to estimate lighting and shape in a temporally coherent way at video frame-rates. Our optimization problem is minimized using a new regular grid Gauss-Newton solver implemented fully on the GPU. We demonstrate results showing enhanced depth maps, which are comparable to offline methods but are computed orders of magnitude faster, as well as baseline comparisons with online filtering-based methods. We conclude with applications of our higher quality depth maps for improved real-time surface reconstruction and performance capture.
References:
1. Ahmed, A. H., and Farag, A. A. 2007. Shape from shading under various imaging conditions. In Proc. CVPR, 1–8.
2. Aodha, O. M., Campbell, N. D. F., Nair, A., and Brostow, G. J. 2012. Patch based synthesis for single depth image superresolution. In Proc. ECCV, 71–84.
3. Barron, J. T., and Malik, J. 2013. Intrinsic scene properties from a single rgb-d image. In Proc. CVPR, IEEE, 17–24.
4. Barron, J. T., and Malik, J. 2013. Shape, illumination, and reflectance from shading. Tech. rep., EECS, UC Berkeley, May.
5. Beder, C., Bartczak, B., and Koch, R. 2007. A combined approach for estimating patchlets from PMD depth images and stereo intensity images. In Proc. DAGM, 11–20.
6. Beeler, T., Bickel, B., Beardsley, P., Sumner, B., and Gross, M. 2010. High-quality single-shot capture of facial geometry. Proc. SIGGRAPH 29, 3.
7. Beeler, T., Bradley, D., Zimmer, H., and Gross, M. 2012. Improved reconstruction of deforming surfaces by cancelling ambient occlusion. In Proc. ECCV, 30–43.
8. Bermano, A., Bradley, D., Zund, T. B. F., Nowrouzezahrai, D., Baran, I., Sorkine-hornung, O., Pfister, H., Sumner, R., Bickel, B., and Gross, M. 2014. Facial performance enhancement using dynamic shape space analysis. ACM Transactions on Graphics 33. to appear.
9. Besl, P. J., and McKay, N. D. 1992. Method for registration of 3-d shapes. In Robotics-DL tentative, International Society for Optics and Photonics, 586–606.
10. Böhme, M., Haker, M., Martinetz, T., and Barth, E. 2008. Shading constraint improves accuracy of time-of-flight measurements. In Proc. CVPR Workshop.
11. Chan, D., Buisman, H., Theobalt, C., and Thrun, S. 2008. A noise-aware filter for real-time depth upsampling. In ECCV Workshop on multi-camera & multi-modal sensor fusion.
12. Cui, Y., Schuon, S., Thrun, S., Stricker, D., and Theobalt, C. 2013. Algorithms for 3d shape scanning with a depth camera. IEEE Trans. PAMI 35, 5, 1039–1050.
13. Debevec, P. 1998. Rendering synthetic objects into real scenes: Bridging traditional and image-based graphics with global illumination and high dynamic range photography. In Proc. SIGGRAPH, 189–198.
14. Debevec, P. 2012. The light stages and their applications to photoreal digital actors. In SIGGRAPH Asia Technical Briefs.
15. Diebel, J., and Thrun, S. 2006. An application of Markov Random Fields to range sensing. In Proc. NIPS, 291–298.
16. Dolson, J., Baek, J., Plagemann, C., and Thrun, S. 2010. Upsampling range data in dynamic environments. In Proc. CVPR.
17. Fanello, S., Keskin, C., Izadi, S., Kohli, P., et al. 2014. Learning to be a depth camera for close-range human capture and interaction. ACM Trans. Graph. 33, 4.
18. Ghosh, A., Fyffe, G., Tunwattanapong, B., Busch, J., Yu, X., and Debevec, P. 2011. Multiview face capture using polarized spherical gradient illumination. ACM Trans. Graph. 30.
19. Gudmundsson, S. A., Aanaes, H., and Larsen, R. 2008. Fusion of stereo vision and time-of-flight imaging for improved 3d estimation. Int. J. Intell. Syst. Technol. Appl. 5, 425–433.
20. Han, Y., Lee, J.-Y., and Kweon, I. S. 2013. High quality shape from a single rgb-d image under uncalibrated natural illumination. In Proc. ICCV.}
21. Hernández, C., Vogiatzis, G., and Cipolla, R. 2008. Multiview photometric stereo. IEEE PAMI 30, 3, 548–554.
22. Horn, B. K. 1974. Determining lightness from an image. Computer graphics and image processing 3, 4, 277–299.
23. Horn, B. K. 1975. Obtaining shape from shading information. The psychology of computer vision, 115–155.
24. Izadi, S., Kim, D., Hilliges, O., Molyneaux, D., Newcombe, R., Kohli, P., Shotton, J., Hodges, S., Freeman, D., Davison, A., and Fitzgibbon, A. 2011. Kinectfusion: Real-time 3d reconstruction and interaction using a moving depth camera. In Proc. UIST, ACM, 559–568.
25. Khan, N., Tran, L., and Tappen, M. 2009. Training many-parameter shape-from-shading models using a surface database. In Proc. ICCV Workshop.
26. Kopf, J., Cohen, M. F., Lischinski, D., and Uyttendaele, M. 2007. Joint bilateral upsampling. ACM Trans. Graph. 26, 3.
27. Lindner, M., Kolb, A., and Hartmann, K. 2007. Data-fusion of PMD-based distance-information and high-resolution RGB-images. In Proc. ISSCS, 121–124.
28. Mulligan, J., and Brolly, X. 2004. Surface determination by photometric ranging. In Proc. CVPR Workshop.
29. Nehab, D., Rusinkiewicz, S., Davis, J., and Ramamoorthi, R. 2005. Efficiently combining positions and normals for precise 3D geometry. Proc. SIGGRAPH 24, 3.
30. Newcombe, R. A., Izadi, S., et al. 2011. Kinectfusion: Real-time dense surface mapping and tracking. In Mixed and augmented reality (ISMAR), IEEE international symposium on, 127–136.
31. Niessner, M., Zollhöfer, M., Izadi, S., and Stamminger, M. 2013. Real-time 3d reconstruction at scale using voxel hashing. ACM Transactions on Graphics (TOG) 32, 6, 169.
32. Park, J., Kim, H., Tai, Y.-W., Brown, M. S., and Kweon, I.-S. 2011. High quality depth map upsampling for 3d-tof cameras. In ICCV, IEEE, 1623–1630.
33. Prados, E., and Faugeras, O. 2005. Shape from shading: a well-posed problem? In Proc. CVPR.
34. Ramamoorthi, R., and Hanrahan, P. 2001. A signal-processing framework for inverse rendering. In Proc. SIGGRAPH, 117–128.
35. Richardt, C., Stoll, C., Dodgson, N. A., Seidel, H.-P., and Theobalt, C. 2012. Coherent spatiotemporal filtering, upsampling and rendering of RGBZ videos. Computer Graphics Forum (Proceedings of Eurographics) 31, 2 (May).
36. Tunwattanapong, B., Fyffe, G., Graham, P., Busch, J., Yu, X., Ghosh, A., and Debevec, P. 2013. Acquiring reflectance and shape from continuous spherical harmonic illumination. ACM Transactions on Graphics (TOG) 32, 4, 109.
37. Valgaerts, L., Wu, C., Bruhn, A., Seidel, H.-P., and Theobalt, C. 2012. Lightweight binocular facial performance capture under uncontrolled lighting. ACM Trans. Graph. 31, 6.
38. Weber, D., Bender, J., Schnoes, M., Stork, A., and Fellner, D. 2013. Efficient gpu data structures and methods to solve sparse linear systems in dynamics applications. Computer Graphics Forum 32, 1, 16–26.Cross Ref
39. Wei, G.-Q., and Hirzinger, G. 1996. Learning shape from shading by a multilayer network. IEEE Trans. Neural Networks.
40. Wu, C., Varanasi, K., Liu, Y., Seidel, H.-P., and Theobalt, C. 2011. Shading-based dynamic shape refinement from multi-view video under general illumination. In Proc. ICCV.
41. Wu, C., Stoll, C., Valgaerts, L., and Theobalt, C. 2013. On-set performance capture of multiple actors with a stereo camera. ACM Transactions on Graphics (TOG) 32, 6, 161.
42. Yang, Q., Yang, R., Davis, J., and Nistr, D. 2007. Spatial-depth super resolution for range images. In Proc. CVPR, IEEE.
43. Yu, L.-F., Yeung, S.-K., Tai, Y.-W., and Lin, S. 2013. Shading-based shape refinement of rgb-d images. In Proc. CVPR.
44. Zhang, Z., Tsa, P.-S., Cryer, J. E., and Shah, M. 1999. Shape from shading: A survey. IEEE PAMI 21, 8, 690–706.
45. Zhu, J., Wang, L., Yang, R., and Davis, J. 2008. Fusion of time-of-flight depth and stereo for high accuracy depth maps. In Proc. CVPR.
46. Zollhöfer, M., Niessner, M., Izadi, S., Rehmann, C., Zach, C., Fisher, M., Wu, C., Fitzgibbon, A., Loop, C., Theobalt, C., and Stamminger, M. 2014. Real-time non-rigid reconstruction using an rgb-d camera. ACM TOG (Proc. SIGGRAPH) 33, 4.
47. Zollhöfer, M., Thies, J., Colaianni, M., Stamminger, M., and Greiner, G. 2014. Interactive model-based reconstruction of the human head using an rgb-d sensor. Computer Animation and Virtual Worlds 25, 3-4, 213–222.
