“Flow-guided warping for image-based shape manipulation” by Vergne, Barla, Bonneau and Fleming

  • ©Romain Vergne, Pascal Barla, Georges-Pierre Bonneau, and Roland W. Fleming




    Flow-guided warping for image-based shape manipulation





    We present an interactive method that manipulates perceived object shape from a single input color image thanks to a warping technique implemented on the GPU. The key idea is to give the illusion of shape sharpening or rounding by exaggerating orientation patterns in the image that are strongly correlated to surface curvature. We build on a growing literature in both human and computer vision showing the importance of orientation patterns in the communication of shape, which we complement with mathematical relationships and a statistical image analysis revealing that structure tensors are indeed strongly correlated to surface shape features. We then rely on these correlations to introduce a flow-guided image warping algorithm, which in effect exaggerates orientation patterns involved in shape perception. We evaluate our technique by 1) comparing it to ground truth shape deformations, and 2) performing two perceptual experiments to assess its effects. Our algorithm produces convincing shape manipulation results on synthetic images and photographs, for various materials and lighting environments.


    1. Barrow, H. 1978. Recovering intrinsic scene characteristics from images. Computer Vision Systems, 3–26.Google Scholar
    2. Ben-Shahar, O., and Zucker, S. 2001. On the perceptual organization of texture and shading flows: from a geometrical model to coherence computation. In CVPR 2001., vol. 1, I-1048–I-1055 vol. 1.Google Scholar
    3. Bigun, J., and Granlund, G. H. 1986. Optimal orientation detection of linear symmetry. Tech. Rep. Report LiTH-ISY-I-0828, Computer Vision Laboratory, Linköping University, Sweden.Google Scholar
    4. Bousseau, A., Paris, S., and Durand, F. 2009. User assisted intrinsic images. ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia 2009) 28, 5. Google ScholarDigital Library
    5. Boyadzhiev, I., Bala, K., Paris, S., and Adelson, E. 2015. Band-sifting decomposition for image-based material editing. ACM Trans. Graph. 34, 5 (Nov.), 163:1–163:16. Google ScholarDigital Library
    6. Brainard, D. H. 1997. The psychophysics toolbox. Spatial Vision 10, 4, 433–436.Google ScholarCross Ref
    7. Brox, T., Van Den Boomgaard, R., Lauze, F., Van De Weijer, J., Weickert, J., and Kornprobst, P. 2006. Adaptive structure tensors and their applications. In Visualization and image processing of tensor fields, J. Weickert and H. Hagen, Eds., vol. 1. Springer, Jan., ch. 2, 17–47.Google Scholar
    8. Caniard, F., and Fleming, R. W. 2007. Distortion in 3d shape estimation with changes in illumination. In Proceedings of the 4th Symposium on Applied Perception in Graphics and Visualization, ACM, New York, NY, USA, APGV ’07, 99–105. Google ScholarDigital Library
    9. Carroll, R., Ramamoorthi, R., and Agrawala, M. 2011. Illumination decomposition for material recoloring with consistent interreflections. ACM Trans. Graph. 30, 4 (July), 43:1–43:10. Google ScholarDigital Library
    10. Chen, T., Zhu, Z., Shamir, A., Hu, S.-M., and Cohen-Or, D. 2013. 3sweepp: Extracting editable objects from a single photo. ACM Trans. Graph. 32, 6 (Nov.), 195:1–195:10. Google ScholarDigital Library
    11. Dekel, T., Michaeli, T., Irani, M., and Freeman, W. T. 2015. Revealing and modifying non-local variations in a single image. ACM Transactions on Graphics (Proc. SIGGRAPH Asia). Google ScholarDigital Library
    12. Dong, Y., Tong, X., Pellacini, F., and Guo, B. 2011. Appgen: Interactive material modeling from a single image. ACM Trans. Graph. 30, 6 (Dec.), 146:1–146:10. Google ScholarDigital Library
    13. Farbman, Z., Fattal, R., Lischinski, D., and Szeliski, R. 2008. Edge-preserving decompositions for multi-scale tone and detail manipulation. In ACM SIGGRAPH 2008 Papers, ACM, New York, NY, USA, SIGGRAPH ’08, 67:1–67:10. Google ScholarDigital Library
    14. Fattal, R., Agrawala, M., and Rusinkiewicz, S. 2007. Multiscale shape and detail enhancement from multi-light image collections. ACM Trans. Graph. 26, 3 (July). Google ScholarDigital Library
    15. Fleming, R. W., Torralba, A., and Adelson, E. H. 2004. Specular reflections and the perception of shape. J. Vis. 4, 9 (9), 798–820.Google ScholarCross Ref
    16. Fleming, R. W., Holtmann-Rice, D., and Bülthoff, H. H. 2011. Estimation of 3d shape from image orientations. Proceedings of the National Academy of Sciences 108, 51, 20438–20443.Google ScholarCross Ref
    17. Freeman, W. T. 1994. The generic viewpoint assumption in a framework for visual perception. Nature 368, 6471 (Apr.), 542–545.Google ScholarCross Ref
    18. Gutierrez, D., Seron, F. J., Lopez-Moreno, J., Sanchez, M. P., Fandos, J., and Reinhard, E. 2008. Depicting procedural caustics in single images. ACM Trans. Graph. 27, 5 (Dec.), 120:1–120:9. Google ScholarDigital Library
    19. Horn, B. K. P., and Brooks, M. J., Eds. 1989. Shape from Shading. MIT Press, Cambridge, MA, USA. Google ScholarDigital Library
    20. Jakob, W., 2010. Mitsuba renderer. http://www.mitsuba-renderer.org.Google Scholar
    21. Kajiya, J. T. 1986. The rendering equation. SIGGRAPH Comput. Graph. 20, 4 (Aug.), 143–150. Google ScholarDigital Library
    22. Khan, E. A., Reinhard, E., Fleming, R. W., and Bülthoff, H. H. 2006. Image-based material editing. ACM Trans. Graph. 25, 3 (July), 654–663. Google ScholarDigital Library
    23. Kholgade, N., Simon, T., Efros, A., and Sheikh, Y. 2014. 3d object manipulation in a single photograph using stock 3d models. ACM Trans. Graph. 33, 4 (July), 127:1–127:12. Google ScholarDigital Library
    24. Koenderink, J., and van Doorn, A. 1980. Photometric invariants related to solid shape. Optica Acta: International Journal of Optics 27, 7, 981–996.Google ScholarCross Ref
    25. Kyprianidis, J. E., and Döllner, J. 2008. Image abstraction by structure adaptive filtering. In Proc. EG UK Theory and Practice of Computer Graphics, 51–58.Google Scholar
    26. Lindeberg, T. 1998. Feature detection with automatic scale selection. Int. J. Comput. Vision 30, 2 (Nov.), 79–116. Google ScholarDigital Library
    27. Luft, T., Colditz, C., and Deussen, O. 2006. Image enhancement by unsharp masking the depth buffer. ACM Trans. Graph. 25, 3 (July), 1206–1213. Google ScholarDigital Library
    28. Mooney, S. W., and Anderson, B. L. 2014. Specular image structure modulates the perception of three-dimensional shape. Current Biology 24, 22, 2737–2742.Google ScholarCross Ref
    29. Oren, M., and Nayar, S. K. 1997. A theory of specular surface geometry. Int. J. Comput. Vision 24, 2 (Sept.), 105–124. Google ScholarDigital Library
    30. Osher, S., and Rudin, L. I. 1990. Feature-oriented image enhancement using shock filters. SIAM J. Numer. Anal. 27, 4 (Aug.), 919–940. Google ScholarDigital Library
    31. Paris, S., Hasinoff, S. W., and Kautz, J. 2011. Local laplacian filters: Edge-aware image processing with a laplacian pyramid. ACM Trans. Graph. 30, 4 (July), 68:1–68:12. Google ScholarDigital Library
    32. Pérez, P., Gangnet, M., and Blake, A. 2003. Poisson image editing. ACM Trans. Graph. 22, 3 (July), 313–318. Google ScholarDigital Library
    33. Perona, P., and Malik, J. 1990. Scale-space and edge detection using anisotropic diffusion. IEEE Trans. Pattern Anal. Mach. Intell. 12, 7 (July), 629–639. Google ScholarDigital Library
    34. Savarese, S., Fei-Fei, L., and Perona, P. 2004. What do reflections tell us about the shape of a mirror? In Proceedings of the 1st Symposium on Applied Perception in Graphics and Visualization, ACM, New York, NY, USA, APGV ’04, 115–118. Google ScholarDigital Library
    35. Sloan, P.-P. J., Martin, W., Gooch, A., and Gooch, B. 2001. The lit sphere: A model for capturing npr shading from art. In Proceedings of Graphics Interface 2001, Canadian Information Processing Society, Toronto, Ont., Canada, Canada, GI ’01, 143–150. Google ScholarDigital Library
    36. Stalling, D., and Hege, H.-C. 1995. Fast and resolution independent line integral convolution. In Proceedings of the 22Nd Annual Conference on Computer Graphics and Interactive Techniques, ACM, New York, NY, USA, SIGGRAPH ’95, 249–256. Google ScholarDigital Library
    37. Vergne, R., and Barla, P. 2015. Designing Gratin, A GPU-Tailored Node-Based System. Journal of Computer Graphics Techniques 4, 4, 17.Google Scholar
    38. Vergne, R., Barla, P., Fleming, R. W., and Granier, X. 2012. Surface flows for image-based shading design. ACM Trans. Graph. 31, 4 (July), 94:1–94:9. Google ScholarDigital Library
    39. Wadhwa, N., Rubinstein, M., Durand, F., and Freeman, W. T. 2013. Phase-based video motion processing. ACM Trans. Graph. 32, 4 (July), 80:1–80:10. Google ScholarDigital Library
    40. Wadhwa, N., Dekel, T., Wei, D., Durand, F., and Freeman, W. T. 2015. Deviation magnification: Revealing departures from ideal geometries. ACM Trans. Graph. 34, 6 (Oct.), 226:1–226:10. Google ScholarDigital Library
    41. Wijntjes, M. W. A., Doerschner, K., Kucukoglu, G., and Pont, S. C. 2012. Relative flattening between velvet and matte 3d shapes: Evidence for similar shape-from-shading computations. Journal of Vision 12, 1, 2.Google ScholarCross Ref
    42. Wu, H.-Y., Rubinstein, M., Shih, E., Guttag, J., Durand, F., and Freeman, W. 2012. Eulerian video magnification for revealing subtle changes in the world. ACM Trans. Graph. 31, 4 (July), 65:1–65:8. Google ScholarDigital Library
    43. Yeung, S. K., Tang, C.-K., Brown, M. S., and Kang, S. B. 2011. Matting and compositing of transparent and refractive objects. ACM Transactions on Graphics 30, 1, 2. Google ScholarDigital Library

ACM Digital Library Publication: