“Non-photorealistic camera: depth edge detection and stylized rendering using multi-flash imaging” by Raskar, Tan, Feris, Yu and Turk

We present a non-photorealistic rendering approach to capture and convey shape features of real-world scenes. We use a camera with multiple flashes that are strategically positioned to cast shadows along depth discontinuities in the scene. The projective-geometric relationship of the camera-flash setup is then exploited to detect depth discontinuities and distinguish them from intensity edges due to material discontinuities.We introduce depiction methods that utilize the detected edge features to generate stylized static and animated images. We can highlight the detected features, suppress unnecessary details or combine features from multiple images. The resulting images more clearly convey the 3D structure of the imaged scenes.We take a very different approach to capturing geometric features of a scene than traditional approaches that require reconstructing a 3D model. This results in a method that is both surprisingly simple and computationally efficient. The entire hardware/software setup can conceivably be packaged into a self-contained device no larger than existing digital cameras.

1. AKERS, D., LOSASSO, F., KLINGER, J., AGRAWALA, M., RICK, J., AND HANRAHAN, P. 2003. Conveying Shape and Features with Image-Based Relighting. In IEEE Visualization. Google ScholarDigital Library
2. AVENUE AMY, 2002. Curious Pictures.Google Scholar
3. BIRCHFIELD, S. 1999. Depth and Motion Discontinuities. PhD thesis, Stanford University. Google ScholarDigital Library
4. CHUANG, Y.-Y., GOLDMAN, D. B., CURLESS, B., SALESIN, D. H., AND SZELISKI, R. 2003. Shadow matting and compositing. ACM Trans. Graph. 22, 3, 494–500. Google ScholarDigital Library
5. COHEN, M. F., COLBURN, A., AND DRUCKER, S. 2003. Image stacks. Tech. Rep. MSR-TR-2003-40, Microsoft Research.Google Scholar
6. DAUM, M., AND DUDEK, G. 1998. On 3-D Surface Reconstruction using Shape from Shadows. In CVPR, 461–468. Google ScholarDigital Library
7. DECARLO, D., AND SANTELLA, A. 2002. Stylization and Abstraction of Photographs. In Proc. Siggraph 02, ACM Press. Google ScholarDigital Library
8. DECARLO, D., FINKELSTEIN, A., RUSINKIEWICZ, S., AND SANTELLA, A. 2003. Suggestive contours for conveying shape. ACM Trans. Graph. 22, 3, 848–855. Google ScholarDigital Library
9. DURAND, F. 2002. An Invitation to Discuss Computer Depiction. In Proceedings of NPAR 2002. Google ScholarDigital Library
10. ELDER, J. 1999. Are Edges Incomplete? International Journal of Computer Vision 34, 2/3, 97–122. Google ScholarDigital Library
11. FATTAL, R., LISCHINSKI, D., AND WERMAN, M. 2002. Gradient Domain High Dynamic Range Compression. In Proceedings of SIGGRAPH 2002, ACM SIGGRAPH, 249–256. Google ScholarDigital Library
12. FERIS, R., TURK, M., RASKAR, R., TAN, K., AND OHASHI, G. 2004. Exploiting Depth Discontinuities for Vision-based Fingerspelling Recognition. In IEEE Workshop on Real-time Vision for Human-Computer Interaction (in conjunction with CVPR’04). Google ScholarDigital Library
13. FORSYTH, AND PONCE. 2002. Computer Vision, A Modern Approach. Google ScholarDigital Library
14. GEIGER, D., LADENDORF, B., AND YUILLE, A. L. 1992. Occlusions and Binocular Stereo. In European Conference on Computer Vision, 425–433. Google ScholarDigital Library
15. GOOCH, B., AND GOOCH, A. 2001. Non-Photorealistic Rendering. A K Peters, Ltd., Natick. Google ScholarDigital Library
16. HERTZMANN, A. 1998. Painterly Rendering with Curved Brush Strokes of Multiple Sizes. In ACM SIGGRAPH, 453–460. Google ScholarDigital Library
17. HUERTAS, A., AND NEVATIA, R. 1988. Detecting buildings in aerial images. Computer Vision, Graphics and Image Processing 41, 2, 131–152. Google ScholarDigital Library
18. HUGGINGS, P., CHEN, H., BELHUMEUR, P., AND ZUCKER, S. 2001. Finding Folds: On the Appearance and Identification of Occlusion. In IEEE CVPR, vol. 2, 718–725.Google Scholar
19. IRVIN, R., AND MCKEOWN, D. 1989. Methods for exploiting the relationship between buildings and their shadows in aerial imagery. IEEE Transactions on Systems, Man and Cybernetics 19, 6, 1564–1575.Google ScholarCross Ref
20. JOHNSTON, S. F. 2002. Lumo: Illumination for cel animation. In Proceedings of NPAR, ACM Press, 45–52. Google ScholarDigital Library
21. KANG, S. B., SZELISKI, R., AND CHAI, J. 2001. Handling occlusions in dense multi-view stereo. In IEEE CVPR, vol. 1, 102–110.Google Scholar
22. KRIEGMAN, D., AND BELHUMEUR, P. 2001. What Shadows Reveal About Object Structure. Journal of the Optical Society of America, 1804–1813.Google Scholar
23. LANGER, M., DUDEK, G., AND ZUCKER, S. 1995. Space Occupancy using Multiple Shadow Images. International Conference on Intelligent Robots and Systems, 390–396.Google Scholar
24. LIN, C., AND NEVATIA, R. 1998. Building detection and description from a single intensity image. Computer Vision and Image Understanding: CVIU 72, 2, 101–121. Google ScholarDigital Library
25. PAPADEMETRIS, X., AND BELHUMEUR, P. N. 1996. Estimation of motion boundary location and optical flow using dynamic programming. In Proc. Int. Conf. on Image Processing.Google ScholarCross Ref
26. PRESS, W. H., TEUKOLSKY, S., VETTERLING, W. T., AND FLANNERY, B. P. 1992. Numerical Recipes in C: The Art of Scientific Computing. Pearson Education. Google ScholarDigital Library
27. RASKAR, R., ILIE, A., AND YU, J. 2004. Image Fusion for Context Enhancement and Video Surrealism. In Proceedings of NPAR. Google ScholarDigital Library
28. RAVIV, D., PAO, Y., AND LOPARO, K. A. 1989. Reconstruction of Three-dimensional Surfaces from Two-dimensional Binary Images. In IEEE Transactions on Robotics and Automation, vol. 5(5), 701–710.Google ScholarCross Ref
29. SAITO, T., AND TAKAHASHI, T. 1990. Comprehensible Rendering of 3-D Shapes. In ACM SIGGRAPH, 197–206. Google ScholarDigital Library
30. SATO, I., SATO, Y., AND IKEUCHI, K. 2001. Stability issues in recovering illumination distribution from brightness in shadows. IEEE Conf. on CVPR, 400–407.Google ScholarCross Ref
31. SAVARESE, S., RUSHMEIER, H., BERNARDINI, F., AND PERONA, P. 2001. Shadow Carving. In ICCV.Google Scholar
32. SCHARSTEIN, D., AND SZELISKI, R. 2002. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. In International Journal of Computer Vision, vol. 47(1), 7–42. Google ScholarDigital Library
33. SHIRAI, Y., AND TSUJI, S. 1972. Extraction of the Line Drawing of 3-Dimensional Objects by Sequential Illumination from Several Directions. Pattern Recognition 4, 4, 345–351.Google ScholarCross Ref
34. STROTHOTTE, T., AND SCHLECHTWEG, S. 2002. NonPhotorealistic Computer Graphics: Modeling, Rendering and Animation. Morgan Kaufmann, San Francisco. Google ScholarDigital Library
35. TAN, P., LIN, S., QUAN, L., AND SHUM, H.-Y. 2003. Highlight Removal by Illumination-Constrained Inpainting. In Ninth IEEE International Conference on Computer Vision. Google ScholarDigital Library
36. TAN, K., KOBLER, J., DIETZ, P., FERIS, R., AND RASKAR, R. 2004. Shape-Enhanced Surgical Visualizations and Medical Illustrations with Multi-Flash Imaging. In MERL TR/38.Google Scholar
37. TOYAMA, K., KRUMM, J., BRUMITT, B., AND MEYERS, B. 1999. Wallflower: Principles and Practice of Background Maintenance. In ICCV, 255–261.Google Scholar
38. WAKING LIFE, 2001. Waking Life, the movie.Google Scholar
39. WEISS, Y. 2001. Deriving intrinsic images from image sequences. In Proceedings of ICCV, vol. 2, 68–75.Google Scholar
40. YANG, D. K.-M. 1996. Shape from Darkness Under Error. PhD thesis, Columbia University. Google ScholarDigital Library