“Flow-guided warping for image-based shape manipulation” by Vergne, Barla, Bonneau and Fleming
Conference:
Type:
Title:
- Flow-guided warping for image-based shape manipulation
Presenter(s)/Author(s):
Session Title: IMAGE & SHAPE MANIPULATION
Moderator(s):
Abstract:
We present an interactive method that manipulates perceived object shape from a single input color image thanks to a warping technique implemented on the GPU. The key idea is to give the illusion of shape sharpening or rounding by exaggerating orientation patterns in the image that are strongly correlated to surface curvature. We build on a growing literature in both human and computer vision showing the importance of orientation patterns in the communication of shape, which we complement with mathematical relationships and a statistical image analysis revealing that structure tensors are indeed strongly correlated to surface shape features. We then rely on these correlations to introduce a flow-guided image warping algorithm, which in effect exaggerates orientation patterns involved in shape perception. We evaluate our technique by 1) comparing it to ground truth shape deformations, and 2) performing two perceptual experiments to assess its effects. Our algorithm produces convincing shape manipulation results on synthetic images and photographs, for various materials and lighting environments.
References:
1. Barrow, H. 1978. Recovering intrinsic scene characteristics from images. Computer Vision Systems, 3–26.Google Scholar
2. Ben-Shahar, O., and Zucker, S. 2001. On the perceptual organization of texture and shading flows: from a geometrical model to coherence computation. In CVPR 2001., vol. 1, I-1048–I-1055 vol. 1.Google Scholar
3. Bigun, J., and Granlund, G. H. 1986. Optimal orientation detection of linear symmetry. Tech. Rep. Report LiTH-ISY-I-0828, Computer Vision Laboratory, Linköping University, Sweden.Google Scholar
4. Bousseau, A., Paris, S., and Durand, F. 2009. User assisted intrinsic images. ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia 2009) 28, 5. Google ScholarDigital Library
5. Boyadzhiev, I., Bala, K., Paris, S., and Adelson, E. 2015. Band-sifting decomposition for image-based material editing. ACM Trans. Graph. 34, 5 (Nov.), 163:1–163:16. Google ScholarDigital Library
6. Brainard, D. H. 1997. The psychophysics toolbox. Spatial Vision 10, 4, 433–436.Google ScholarCross Ref
7. Brox, T., Van Den Boomgaard, R., Lauze, F., Van De Weijer, J., Weickert, J., and Kornprobst, P. 2006. Adaptive structure tensors and their applications. In Visualization and image processing of tensor fields, J. Weickert and H. Hagen, Eds., vol. 1. Springer, Jan., ch. 2, 17–47.Google Scholar
8. Caniard, F., and Fleming, R. W. 2007. Distortion in 3d shape estimation with changes in illumination. In Proceedings of the 4th Symposium on Applied Perception in Graphics and Visualization, ACM, New York, NY, USA, APGV ’07, 99–105. Google ScholarDigital Library
9. Carroll, R., Ramamoorthi, R., and Agrawala, M. 2011. Illumination decomposition for material recoloring with consistent interreflections. ACM Trans. Graph. 30, 4 (July), 43:1–43:10. Google ScholarDigital Library
10. Chen, T., Zhu, Z., Shamir, A., Hu, S.-M., and Cohen-Or, D. 2013. 3sweepp: Extracting editable objects from a single photo. ACM Trans. Graph. 32, 6 (Nov.), 195:1–195:10. Google ScholarDigital Library
11. Dekel, T., Michaeli, T., Irani, M., and Freeman, W. T. 2015. Revealing and modifying non-local variations in a single image. ACM Transactions on Graphics (Proc. SIGGRAPH Asia). Google ScholarDigital Library
12. Dong, Y., Tong, X., Pellacini, F., and Guo, B. 2011. Appgen: Interactive material modeling from a single image. ACM Trans. Graph. 30, 6 (Dec.), 146:1–146:10. Google ScholarDigital Library
13. Farbman, Z., Fattal, R., Lischinski, D., and Szeliski, R. 2008. Edge-preserving decompositions for multi-scale tone and detail manipulation. In ACM SIGGRAPH 2008 Papers, ACM, New York, NY, USA, SIGGRAPH ’08, 67:1–67:10. Google ScholarDigital Library
14. Fattal, R., Agrawala, M., and Rusinkiewicz, S. 2007. Multiscale shape and detail enhancement from multi-light image collections. ACM Trans. Graph. 26, 3 (July). Google ScholarDigital Library
15. Fleming, R. W., Torralba, A., and Adelson, E. H. 2004. Specular reflections and the perception of shape. J. Vis. 4, 9 (9), 798–820.Google ScholarCross Ref
16. Fleming, R. W., Holtmann-Rice, D., and Bülthoff, H. H. 2011. Estimation of 3d shape from image orientations. Proceedings of the National Academy of Sciences 108, 51, 20438–20443.Google ScholarCross Ref
17. Freeman, W. T. 1994. The generic viewpoint assumption in a framework for visual perception. Nature 368, 6471 (Apr.), 542–545.Google ScholarCross Ref
18. Gutierrez, D., Seron, F. J., Lopez-Moreno, J., Sanchez, M. P., Fandos, J., and Reinhard, E. 2008. Depicting procedural caustics in single images. ACM Trans. Graph. 27, 5 (Dec.), 120:1–120:9. Google ScholarDigital Library
19. Horn, B. K. P., and Brooks, M. J., Eds. 1989. Shape from Shading. MIT Press, Cambridge, MA, USA. Google ScholarDigital Library
20. Jakob, W., 2010. Mitsuba renderer. http://www.mitsuba-renderer.org.Google Scholar
21. Kajiya, J. T. 1986. The rendering equation. SIGGRAPH Comput. Graph. 20, 4 (Aug.), 143–150. Google ScholarDigital Library
22. Khan, E. A., Reinhard, E., Fleming, R. W., and Bülthoff, H. H. 2006. Image-based material editing. ACM Trans. Graph. 25, 3 (July), 654–663. Google ScholarDigital Library
23. Kholgade, N., Simon, T., Efros, A., and Sheikh, Y. 2014. 3d object manipulation in a single photograph using stock 3d models. ACM Trans. Graph. 33, 4 (July), 127:1–127:12. Google ScholarDigital Library
24. Koenderink, J., and van Doorn, A. 1980. Photometric invariants related to solid shape. Optica Acta: International Journal of Optics 27, 7, 981–996.Google ScholarCross Ref
25. Kyprianidis, J. E., and Döllner, J. 2008. Image abstraction by structure adaptive filtering. In Proc. EG UK Theory and Practice of Computer Graphics, 51–58.Google Scholar
26. Lindeberg, T. 1998. Feature detection with automatic scale selection. Int. J. Comput. Vision 30, 2 (Nov.), 79–116. Google ScholarDigital Library
27. Luft, T., Colditz, C., and Deussen, O. 2006. Image enhancement by unsharp masking the depth buffer. ACM Trans. Graph. 25, 3 (July), 1206–1213. Google ScholarDigital Library
28. Mooney, S. W., and Anderson, B. L. 2014. Specular image structure modulates the perception of three-dimensional shape. Current Biology 24, 22, 2737–2742.Google ScholarCross Ref
29. Oren, M., and Nayar, S. K. 1997. A theory of specular surface geometry. Int. J. Comput. Vision 24, 2 (Sept.), 105–124. Google ScholarDigital Library
30. Osher, S., and Rudin, L. I. 1990. Feature-oriented image enhancement using shock filters. SIAM J. Numer. Anal. 27, 4 (Aug.), 919–940. Google ScholarDigital Library
31. Paris, S., Hasinoff, S. W., and Kautz, J. 2011. Local laplacian filters: Edge-aware image processing with a laplacian pyramid. ACM Trans. Graph. 30, 4 (July), 68:1–68:12. Google ScholarDigital Library
32. Pérez, P., Gangnet, M., and Blake, A. 2003. Poisson image editing. ACM Trans. Graph. 22, 3 (July), 313–318. Google ScholarDigital Library
33. Perona, P., and Malik, J. 1990. Scale-space and edge detection using anisotropic diffusion. IEEE Trans. Pattern Anal. Mach. Intell. 12, 7 (July), 629–639. Google ScholarDigital Library
34. Savarese, S., Fei-Fei, L., and Perona, P. 2004. What do reflections tell us about the shape of a mirror? In Proceedings of the 1st Symposium on Applied Perception in Graphics and Visualization, ACM, New York, NY, USA, APGV ’04, 115–118. Google ScholarDigital Library
35. Sloan, P.-P. J., Martin, W., Gooch, A., and Gooch, B. 2001. The lit sphere: A model for capturing npr shading from art. In Proceedings of Graphics Interface 2001, Canadian Information Processing Society, Toronto, Ont., Canada, Canada, GI ’01, 143–150. Google ScholarDigital Library
36. Stalling, D., and Hege, H.-C. 1995. Fast and resolution independent line integral convolution. In Proceedings of the 22Nd Annual Conference on Computer Graphics and Interactive Techniques, ACM, New York, NY, USA, SIGGRAPH ’95, 249–256. Google ScholarDigital Library
37. Vergne, R., and Barla, P. 2015. Designing Gratin, A GPU-Tailored Node-Based System. Journal of Computer Graphics Techniques 4, 4, 17.Google Scholar
38. Vergne, R., Barla, P., Fleming, R. W., and Granier, X. 2012. Surface flows for image-based shading design. ACM Trans. Graph. 31, 4 (July), 94:1–94:9. Google ScholarDigital Library
39. Wadhwa, N., Rubinstein, M., Durand, F., and Freeman, W. T. 2013. Phase-based video motion processing. ACM Trans. Graph. 32, 4 (July), 80:1–80:10. Google ScholarDigital Library
40. Wadhwa, N., Dekel, T., Wei, D., Durand, F., and Freeman, W. T. 2015. Deviation magnification: Revealing departures from ideal geometries. ACM Trans. Graph. 34, 6 (Oct.), 226:1–226:10. Google ScholarDigital Library
41. Wijntjes, M. W. A., Doerschner, K., Kucukoglu, G., and Pont, S. C. 2012. Relative flattening between velvet and matte 3d shapes: Evidence for similar shape-from-shading computations. Journal of Vision 12, 1, 2.Google ScholarCross Ref
42. Wu, H.-Y., Rubinstein, M., Shih, E., Guttag, J., Durand, F., and Freeman, W. 2012. Eulerian video magnification for revealing subtle changes in the world. ACM Trans. Graph. 31, 4 (July), 65:1–65:8. Google ScholarDigital Library
43. Yeung, S. K., Tang, C.-K., Brown, M. S., and Kang, S. B. 2011. Matting and compositing of transparent and refractive objects. ACM Transactions on Graphics 30, 1, 2. Google ScholarDigital Library
