“Image-based modeling and photo editing” by Oh, Dorsey, Durand and Chen

We present an image-based modeling and editing system that takes a single photo as input. We represent a scene as a layered collection of depth images, where each pixel encodes both color and depth. Starting from an input image, we employ a suite of user-assisted techniques, based on a painting metaphor, to assign depths and extract layers. We introduce two specific editing operations. The first, a “clone brushing tool,” permits the distortion-free copying of parts of a picture, by using a parameterization optimization technique. The second, a “texture-illuminance decoupling filter,” discounts the effect of illumination on uniformly textured areas, by decoupling large- and small-scale features via bilateral filtering. Our system enables editing from different viewpoints, extracting and grouping of image-based objects, and modifying the shape, color, and illumination of these objects.

1. V. Blanz and T. Vetter. A morphable model for the synthesis of 3d faces. Proc. of SIGGRAPH, 1999.
2. Canoma. http://www.canoma.com.
3. E. Chen. Quicktime VR – an image-based approach to virtual environment navigation. Proc. of SIGGRAPH, 1995.
4. E. Chen and L. Williams. View interpolation for image synthesis. In Proc. of SIGGRAPH, 1993.
5. P. Debevec. Rendering synthetic objects into real scenes: Bridging traditional and image-based graphics with global illumination and high dynamic range photography. In Proc. of SIGGRAPH, 1998.
6. P. Debevec and J. Malik. Recovering high dynamic range radiance maps from photographs. Proc. of SIGGRAPH, 1997.
7. G. Drettakis, L. Robert, and S. Bougnoux. Interactive common illumination for computer augmented reality. Eurographics Rendering Workshop, 1997.
8. P. Debevec, C. Taylor, and J. Malik. Modeling and rendering architecture from photographs: A hybrid geometry- and image-based approach. In Proc. of SIGGRAPH 96, 1996.
9. J. Elder and R. Goldberg. Image editing in the contour domain. IEEE Trans. on Pattern Analysis and Machine Intelligence, 23(3), 2001.
10. A. Fournier, A. Gunawan, and C. Romanzin. Common illumination between real and computer generated scenes. Graphics Interface, 1993.
11. O. Faugeras, S. Laveau, L. Robert, G. Csurka, and C. Zeller. 3-d reconstruction of urban scenes from sequences of images. In A. Gruen, O. Kuebler, and P. Agouris, editors, Automatic Extraction of Man-Made Objects from Aerial and Space Images. Birkhauser, 1995.
12. J. Gomes, L. Darsa, B. Costa, and L. Velho. Warping And Morphing Of Graphical Objects. Morgan Kaufman, 1998.
13. S. Gortler, R. Grzeszczuk, R. Szeliski, and M. Cohen. The lumigraph. In Proc. of SIGGRAPH, 1996.
14. Y. Horry, K. Anjyo, and K. Arai. Tour into the picture: Using a spidery mesh interface to make animation from a single image. In Proc. of SIGGRAPH 97, 1997.
15. P. Hanrahan and P. Haeberli. Direct wysiwyg painting and texturing on 3d shapes. Proc. of SIGGRAPH, 1990.
16. T. Igarashi, S. Matsuoka, and H. Tanaka. Teddy: A sketching interface for 3D freeform design. In Siggraph, Los Angeles, 1999.
17. S. Kang. Depth painting for image-based rendering applications. Tech. report, CRL, Compaq Cambridge Research Lab, 1998. http://www.research.microsoft.com/Users/sbkang/publications/index.html.
18. D. Liebowitz, A. Criminisi, and A. Zisserman. Creating architectural models from images. In Proc. of Eurographics, 1999.
19. C. Loscos, G. Drettakis, and L. Robert. Interactive virtual relighting of real scenes. IEEE Trans. on Visualization and Computer Graphics, 6(3), 2000.
20. S. Laveau and O. Faugeras. 3-D scene representation as a collection of images and fundamental matrices. In Proc. of 12th Int. Conf. on Pattern Recognition, volume 1, pages 689-691, 1994.
21. C. Loscos, M.C. Frasson, G. Drettakis, B. Walter, X. Granier, and P. Poulin. Interactive virtual relighting and remodeling of real scenes. Eurographics Rendering Workshop, 1999.
22. M. Levoy and P. Hanrahan. Light field rendering. In Proc. of SIG- GRAPH, 1996.
23. B. Levy and JL Mallet. Non-distorted texture mapping for sheared triangulated meshes. In Proc. of SIGGRAPH, 1998.
24. JL Mallet. Discrete smooth interpolation. ACM Trans. on Graphics, 8(2):121-144, 1989.
25. L. McMillan and G. Bishop. Plenoptic modeling: An image-based rendering system. In Proc. of SIGGRAPH, 1995.
26. L. McMillan. An Image-based Approach to Three-Dimensional Computer Graphics. PhD thesis, U. of North Carolina, Chapel Hill, 1997.
27. MetaCreations. http://www.metacreations.com.
28. W. Mark, L. McMillan, and G. Bishop. Post-rendering 3D warping. In ACM Symp. on Interactive 3D Graphics, 1997.
29. S. K. Nayar and R. M. Bolle. Computing reflectance ratios from an image. Pattern recognition, 7, 1993.
30. S. Palmer. Vision Science : Photons to Phenomenology. MIT Press, 1999.
31. Photomodeler. http://www.photomodeler.com.
32. P. Perona and J. Malik. Scale-space and edge detection using anisotropic diffusion. IEEE Trans. on Pattern Analysis and Machine Intelligence, 12(7):629-639, July 1990.
33. P. Poulin, M. Ouimet, and M.C. Frasson. Interactively modeling with photogrammetry. In Eurographics Workshop on Rendering, 1998.
34. P. Poulin, K. Ratib, and M. Jacques. Sketching shadows and highlights to position lights. In Proc. of Computer Graphics International 97, 1997.
35. W. Press, S.Teukolsky, W. Vetterling, and B. Flannery. Numerical Recipes. Cambridge Univ. Pr., 2nd edition, 1992.
36. Realviz. Image modeler. http://www.realviz.com.
37. F. Sillion and G. Drettakis. Feature-based control of visibility error: A multi-resolution clustering algorithm for global illumination. In Proc. SIGGRAPH, 1995.
38. J. Shade, S. Gortler, L. He, and R. Szeliski. Layered depth images. In Proc. of SIGGRAPH, 1998.
39. S. Seitz and K. Kutulakos. Plenoptic image editing. In Proc. 5th Int. Conf.on Computer Vision, 1998.
40. M. Segal, C. Korobkin, R. van Widenfelt, J. Foran, and P. Haeberli. Fast shadows and lighting effects using texture mapping. Proc. of SIG- GRAPH, 1992.
41. C. Tomasi and R. Manduchi. Bilateral filtering for gray and color images. In IEEE Int. Conf. on Computer Vision, 1998.
42. J. Y. A. Wang and E. H. Adelson. Representing moving images with layers. IEEE Trans. on Image Processing, 3(5):625-638, 1994.
43. L. Williams. Image jets, level sets and silhouettes. Workshop on Image-Based Modeling and Renderingt, http://wwwgraphics. stanford.edu/workshops/ibr98/, March 1998.
44. Y. Yu, P. Debevec, J. Malik, and T. Hawkins. Inverse global illumination: Recovering reflectance models of real scenes from photographs. Proc. of SIGGRAPH, 1999.