“Real-time pixel luminance optimization for dynamic multi-projection mapping”
Conference:
Type(s):
Title:
- Real-time pixel luminance optimization for dynamic multi-projection mapping
Session/Category Title: Cinematography and Video Processing
Presenter(s)/Author(s):
Abstract:
Using projection mapping enables us to bring virtual worlds into shared physical spaces. In this paper, we present a novel, adaptable and real-time projection mapping system, which supports multiple projectors and high quality rendering of dynamic content on surfaces of complex geometrical shape. Our system allows for smooth blending across multiple projectors using a new optimization framework that simulates the diffuse direct light transport of the physical world to continuously adapt the color output of each projector pixel. We present a real-time solution to this optimization problem using off-the-shelf graphics hardware, depth cameras and projectors. Our approach enables us to move projectors, depth camera or objects while maintaining the correct illumination, in realtime, without the need for markers on the object. It also allows for projectors to be removed or dynamically added, and provides compelling results with only commodity hardware.
References:
1. Akaoka, E., Ginn, T., and Vertegaal, R. 2010. Display-objects: Prototyping functional physical interfaces on 3d styrofoam, paper or cardboard models. In Proceedings of the Fourth International Conference on Tangible, Embedded, and Embodied Interaction, ACM, New York, NY, USA, TEI ’10, 49–56.
2. Bandyopadhyay, D., Raskar, R., and Fuchs, H. 2001. Dynamic shader lamps: Painting on movable objects. In Augmented Reality, 2001. Proceedings. IEEE and ACM International Symposium on, IEEE, 207–216.
3. Bimber, O., and Raskar, R. 2005. Spatial augmented reality: merging real and virtual worlds. AK Peters Wellesley, MA.
4. Bimber, O., Emmerling, A., and Klemmer, T. 2005. Embedded entertainment with smart projectors. Computer 38, 1.
5. Bimber, O., Wetzstein, G., Emmerling, A., and Nitschke, C. 2005. Enabling view-dependent stereoscopic projection in real environments. In Proceedings of the 4th IEEE/ACM International Symposium on Mixed and Augmented Reality, IEEE Computer Society, 14–23.
6. Bimber, O., Iwai, D., Wetzstein, G., and Grundhöfer, A. 2008. The visual computing of projector-camera systems. In Computer Graphics Forum, vol. 27, Wiley Online Library.
7. Brown, M., Majumder, A., and Yang, R. 2005. Camera-based calibration techniques for seamless multiprojector displays. Visualization and Computer Graphics, IEEE Transactions on 11, 2, 193–206.
8. Cao, X., and Balakrishnan, R. 2006. Interacting with dynamically defined information spaces using a handheld projector and a pen. In Proceedings of the 19th annual ACM symposium on User interface software and technology, ACM, 225–234.
9. Cossairt, O., Nayar, S. K., and Ramamoorthi, R. 2008. Light field transfer: Global illumination between real and synthetic objects. ACM Trans. on Graphics (Aug).
10. Fujii, K., Grossberg, M. D., and Nayar, S. K. 2005. A projector-camera system with real-time photometric adaptation for dynamic environments. In Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on, vol. 1, IEEE, 814–821.
11. Grundhöfer, A., and bimber, O. 2008. Real-time adaptive radiometric compensation. Visualization and Computer Graphics, IEEE Transactions on 14, 1, 97–108.
12. Grundhöfer, A. 2013. Practical non-linear photometric projector compensation. In Computer Vision and Pattern Recognition Workshops (CVPRW), 2013 IEEE Conference on, IEEE.
13. Harrison, C., Benko, H., and Wilson, A. D. 2011. Omnitouch: wearable multitouch interaction everywhere. In Proceedings of the 24th annual ACM symposium on User interface software and technology, ACM, 441–450.
14. Izadi, S., Kim, D., Hilliges, O., Molyneaux, D., Newcombe, R., Kohli, P., Shotton, J., Hodges, S., Freeman, D., Davison, A., et al. 2011. Kinectfusion: real-time 3d reconstruction and interaction using a moving depth camera. In Proc. UIST, ACM, 559–568.
15. Jones, B. R., Benko, H., Ofek, E., and Wilson, A. D. 2013. Illumiroom: Peripheral projected illusions for interactive experiences. In ACM SIGGRAPH 2013 Emerging Technologies, ACM, New York, NY, USA, SIGGRAPH ’13, 7:1–7:1.
16. Jones, B., Sodhi, R., Murdock, M., Mehra, R., Benko, H., Wilson, A., Ofek, E., MacIntyre, B., Raghuvanshi, N., and Shapira, L. 2014. Roomalive: Magical experiences enabled by scalable, adaptive projector-camera units. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology, ACM, NY, USA, UIST ’14.
17. Kaspar, A., and Deng, B. 2015. Real-time deformation of constrained meshes using gpu. In GPU Computing and Applications, Y. Cai and S. See, Eds. Springer Singapore, 15–34.
18. Knecht, M., Traxler, C., Mattausch, O., Purgathofer, W., and Wimmer, M. 2010. Differential instant radiosity for mixed reality. In Proceedings of the 2010 IEEE International Symposium on Mixed and Augmented Reality (ISMAR 2010), 99–107. Best Paper Award!
19. Konieczny, J., and Meyer, G. W. 2006. Material and color design using projectors. In CGIV’06, 438–442.
20. Law, A. J., Aliaga, D. G., Sajadi, B., Majumder, A., and Pizlo, Z. 2011. Perceptually based appearance modification for compliant appearance editing. In Computer Graphics Forum, vol. 30, Wiley Online Library, 2288–2300.
21. Lee, J. C., Dietz, P. H., Maynes-Aminzade, D., Raskar, R., and Hudson, S. E. 2004. Automatic projector calibration with embedded light sensors. In Proceedings of the 17th annual ACM symposium on User interface software and technology, ACM, 123–126.
22. Lee, J. C., Hudson, S. E., and Tse, E. 2008. Foldable interactive displays. In Proceedings of the 21st annual ACMsymposium on User interface software and technology, ACM, 287–290.
23. Majumder, A., and Brown, M. S. 2007. Practical multi-projector display design. AK Peters USA.
24. Molyneaux, D., Izadi, S., Kim, D., Hilliges, O., Hodges, S., Cao, X., Butler, A., and Gellersen, H. 2012. Interactive environment-aware handheld projectors for pervasive computing spaces. In Pervasive Computing. Springer, 197–215.
25. Newcombe, R. A., Davison, A. J., Izadi, S., Kohli, P., Hilliges, O., Shotton, J., Molyneaux, D., Hodges, S., Kim, D., and Fitzgibbon, A. 2011. Kinectfusion: Real-time dense surface mapping and tracking. In Proc. ISMAR, IEEE.
26. Niessner, M., Zollhöfer, M., Izadi, S., and Stamminger, M. 2013. Real-time 3d reconstruction at scale using voxel hashing. ACM TOG 32, 6, 169.
27. Nowrouzezahrai, D., Geiger, S., Mitchell, K., Sumner, R., Jarosz, W., and Gross, M. 2011. Light factorization for mixed-frequency shadows in augmented reality. In 10th IEEE International Symposium on Mixed and Augmented Reality (Proceedings of ISMAR 2011).
28. Pinhanez, C. 2001. The everywhere displays projector: A device to create ubiquitous graphical interfaces. In Ubicomp 2001: Ubiquitous Computing, Springer, 315–331.
29. Raskar, R., Welch, G., Cutts, M., Lake, A., Stesin, L., and Fuchs, H. 1998. The office of the future: A unified approach to image-based modeling and spatially immersive displays. In Proceedings of the 25th annual conference on Computer graphics and interactive techniques, ACM, 179–188.
30. Raskar, R., Welch, G., Low, K.-L., and Bandyopadhyay, D. 2001. Shader lamps: Animating real objects with image-based illumination. In Proceedings of the 12th Eurographics Workshop on Rendering Techniques, Springer-Verlag, London.
31. Reed, S. E., Kreylos, O., Hsi, S., Kellogg, L. H., Schladow, G., Yikilmaz, M. B., Segale, H., Silverman, J., Yalowitz, S., and Sato, E. 2014. Shaping Watersheds Exhibit: An Interactive, Augmented Reality Sandbox for Advancing Earth Science Education. AGU Fall Meeting Abstracts (Dec.), A1.
32. Resch, C., Keitler, P., and Klinker, G. 2014. Sticky projections—a new approach to interactive shader lamp tracking. In Mixed and Augmented Reality (ISMAR), 2014 IEEE International Symposium on, IEEE, 151–156.
33. Sheng, Y., Yapo, T., Young, C., and Cutler, B. 2009. Virtual heliodon: Spatially augmented reality for architectural day-lighting design. In Virtual Reality Conference, 2009. IEEE.
34. Shum, H.-Y., Chan, S.-C., and Kang, S. B. 2008. Image-based rendering. Springer Science & Business Media.
35. Sueishi, T., Oku, H., and Ishikawa, M. 2015. Robust high-speed tracking against illumination changes for dynamic projection mapping. In IEEE Virtual Reality Conference (VR2015).
36. Uday Mehta, S., Kim, K., Pajak, D., Pulli, K., Kautz, J., and Ramamoorthi, R. 2015. Filtering environment illumination for interactive physically-based rendering in mixed reality. Tech. Rep. UCB/EECS-2015-164, EECS Department, University of California, Berkeley, Jun.
37. Weber, D., Bender, J., Schnoes, M., Stork, A., and Fellner, D. 2013. Efficient gpu data structures and methods to solve sparse linear systems in dynamics applications. Computer Graphics Forum 32, 1, 16–26.
38. Wetzstein, G., Bimber, O., et al. 2007. Radiometric compensation through inverse light transport. In Pacific conference on computer graphics and applications, 391–399.
39. Wilson, A. D., and Benko, H. 2010. Combining multiple depth cameras and projectors for interactions on, above and between surfaces. In Proceedings of the 23nd annual ACM symposium on User interface software and technology, ACM.
40. Wu, C., Zollhöfer, M., Niessner, M., Stamminger, M., Izadi, S., and Theobalt, C. 2014. Real-time shading-based refinement for consumer depth cameras. ACM Transactions on Graphics (TOG) 33, 6.
41. Zhang, Z. 2000. A flexible new technique for camera calibration. Pattern Analysis and Machine Intelligence, IEEE Transactions on 22, 11 (Nov), 1330–1334.
42. Zollhöfer, M., Niessner, M., Izadi, S., Rehmann, C., Zach, C., Fisher, M., Wu, C., Fitzgibbon, A., Loop, C., Theobalt, C., and Stamminger, M. 2014. Real-time non-rigid reconstruction using an rgb-d camera. ACM Transactions on Graphics (TOG) 33, 4.
