“Band-Sifting Decomposition for Image-Based Material Editing” by Boyadzhiev, Bala, Paris and Adelson

  • ©Ivaylo Boyadzhiev, Kavita Bala, Sylvain Paris, and Edward H. Adelson

Conference:


Type:


Title:

    Band-Sifting Decomposition for Image-Based Material Editing

Session/Category Title: IMAGE & SHAPE MANIPULATION

Presenter(s)/Author(s):


Moderator(s):


Abstract:


    Photographers often “prep” their subjects to achieve various effects; for example, toning down overly shiny skin, covering blotches, etc. Making such adjustments digitally after a shoot is possible, but difficult without good tools and good skills. Making such adjustments to video footage is harder still. We describe and study a set of 2D image operations, based on multiscale image analysis, that are easy and straightforward and that can consistently modify perceived material properties. These operators first build a subband decomposition of the image and then selectively modify the coefficients within the subbands. We call this selection process band sifting.

    We show that different siftings of the coefficients can be used to modify the appearance of properties such as gloss, smoothness, pigmentation, or weathering. The band-sifting operators have particularly striking effects when applied to faces; they can provide “knobs” to make a face look wetter or drier, younger or older, and with heavy or light variation in pigmentation. Through user studies, we identify a set of operators that yield consistent subjective effects for a variety of materials and scenes. We demonstrate that these operators are also useful for processing video sequences.

References:


    1. Arqball. 2014. Shoes. https://arqspin.com/gallery/shoes/.
    2. S. Bae, S. Paris, and F. Durand. 2006. Two-scale tone management for photographic look. ACM Trans. Graph. 25, 3, 637–645. 
    3. S. Bell, P. Upchurch, N. Snavely, and K. Bala. 2013. OpenSurfaces: A richly annotated catalog of surface appearance. ACM Trans. Graph. 32, 4. 
    4. S. Bell, P. Upchurch, N. Snavely, and K. Bala. 2015. Material recognition in the wild with the materials in context database. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR’15).
    5. P. J. Burt and E. H. Adelson. 1983. The Laplacian pyramid as a compact image code. IEEE Trans. Comm. 31, 4, 532–540.
    6. V. Bychkovsky, S. Paris, E. Chan, and F. Durand. 2011. Learning photographic global tonal adjustment with a database of input/output image pairs. In Proceedings of the 24th IEEE Conference on Computer Vision and Pattern Recognition (CVPR’11). 97–104. 
    7. J. Chen, S. Paris, and F. Durand. 2007. Real-time edge-aware image processing with the bilateral grid. ACM Trans. Graph. 26, 3. 
    8. P. Debevec, T. Hawkins, C. Tchou, H.-P. Duiker, W. Sarokin, and M. Sagar. 2000. Acquiring the reflectance field of a human face. In Proceedings of the 27th Annual ACM Congerence on Computer Graphics and Interactive Techniques (SIGGRAPH’00). ACM Press/Addison-Wesley, New York, 145–156. 
    9. P. E. Debevec and J. Malik. 1997. Recovering high dynamic range radiance maps from photographs. In Proceedings of the 24th Annual ACM Conference on Computer Graphics and Interactive Techniques (SIGGRAPH’97). ACM Press/Addison-Wesley, New York, 369–378. 
    10. D. Donoho. 1995. De-noising by soft-thresholding. IEEE Trans. Inf. Theory. 41, 3, 613–627. 
    11. F. Durand and J. Dorsey. 2002. Fast bilateral filtering for the display of high-dynamic rrange images. ACM Trans. Graph. 21, 3. 
    12. T. Ewan. 2009. Orange. https://www.flickr.com/photos/the_ewan/3958779607/.
    13. Z. Farbman, R. Fattal, D. Lischinski, and R. Szeliski. 2008. Edge-preserving decomposition for mult-scale tone and detail manipulation. ACM Trans. Graph. 27, 3. 
    14. R. Fattal. 2009. Edge-avoiding wavelets and their applications. ACM Trans. Graph. 28, 3. 
    15. R. Fattal, M. Agrawala, and S. Rusinkiewicz. 2007. Multiscale shape and detail enhancement from multi-light image collections. ACM Trans. Graph. 26, 3. 
    16. R. W. Fleming and H. H. Bülthoff. 2005. Low-level image cues in the perception of translucent materials. ACM Trans. Appl. Percept. 2, 3. 
    17. R. W. Fleming, C. Wiebel, and K. Gegenfurtner. 2013. Perceptual qualities and material classes. J. Vis. 13, 8, 9.
    18. E. S. L. Gastal and M. M. Oliveira. 2011. Domain transform for edge-aware image and video processing. ACM Trans. Graph. 30, 3. 
    19. E. S. L. Gastal and M. M. Oliveira. 2012. Adaptive manifolds for real-time high-dimensional filtering. ACM Trans. Graph. 31, 4. 
    20. L. Glondu, L. Muguercia, M. Marchal, C. Bosch, H. Rushmeier, G. Dumont, and G. Drettakis. 2012. Example-based fractured appearance. Comput. Graph. Forum 31, 4. 
    21. A. Golovinskiy, W. Matusik, H. Pfister, S. Rusinkiewicz, and T. Funkhouser. 2006. A statistical model for synthesis of detailed facial geometry. ACM Trans. Graph. 25, 3. 
    22. J. Hanika, H. Dammertz, and H. P. A. Lensch. 2011. Edge-optimized a-trous wavelets for local contrast enhancement with robust denoising. Comput. Graph. Forum 30, 7.
    23. M. Harris, S. Sengupta, and J. D. Owens. 2007. Parallel prefix sum (scan) with CUDA. In GPU Gems 3, H. Nguyen, Ed., Addison-Wesley, 851–876.
    24. W. Hartshorn. 2008. Sweet potatoes. https://www.flickr.com/photos/wallyhartshorn/2850269642/.
    25. K. He, J. Sun, and X. Tang. 2010. Guided image filtering. In Proceedings of the European Conference on Computer Vision (ECCV’10). 
    26. D. J. Heeger and J. R. Bergen. 1995. Pyramid-based texture analysis/synthesis. In Proceedings of the Annual ACM Conference on Computer Graphics and Interactive Techniques (SIGGRAPH’95). ACM Press, New York. 
    27. R. Jill. 2006. Self portrait. https://www.flickr.com/photos/98972490N04/958207891.
    28. E. Judy. 2009. Good grape crop. https://www.flickr.com/photos/659247400N00/5140439795/.
    29. L. Karacan, E. Erdem, and A. Erdem. 2013. Structure preserving image smoothing via region covariances. ACM Trans. Graph. 32, 6. 
    30. E. A. Khan, E. Reinhard, R. W. Fleming, and H. H. Bülthoff. 2006. Image-based material editing. ACM Trans. Graph. 25, 3. 
    31. J. Kim, P. Marlow, and B. L. Anderson. 2011. The perception of gloss depends on highlight congruence with surface shading. J. Vis. 11, 9, 4.
    32. E. H. Land and J. Mccann. 1971. Lightness and retinex theory. J. Optic. Soc. Amer. 1–11.
    33. C. Liu, L. Sharan, E. H. Adelson, and R. Rosenholtz. 2010. Exploring features in a Bayesian framework for material recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’10). 239–246.
    34. S. P. Mallick, T. Zickler, P. N. Belhumeur, and D. J. Kriegman. 2006. Specularity removal in images and videos: A PDE approach. In Proceedings of the European Conference on Computer Vision (ECCV’06). 550–563. 
    35. R. Mantiuk, K. J. Kim, A. G. Rempel, and W. Heidrich. 2011. HDR-VDP-2: A calibrated visual metric for visibility and quality predictions in all luminance conditions. ACM Trans. Graph. 30, 4. 
    36. R. Mantiuk, K. Myszkowski, and H.-P. Seidel. 2006. A perceptual framework for contrast processing of high dynamic range images. ACM Trans. Appl. Percept. 3, 3, 286–308. 
    37. P. J. Marlow and B. L. Anderson. 2013. Generative constraints on image cues for perceived gloss. J. Vis. 13, 14, 2.
    38. P. J. Marlow, J. Kim, and B. L. Anderson. 2012. The perception and misperception of specular surface reflectance. Curr. Biol. 22, 20, 1909–1913.
    39. T. Mertens, J. Kautz, J. Chen, P. Bekaert, and F. Durand. 2006. Texture transfer using geometry correlation. In Proceedings of the Eurographics Conference on Rendering Techniques (EGSR’06). 273–284. 
    40. I. Motoyoshi, S. Nishida, L. Sharan, and E. H. Adelson. 2007. Image statistics and the perception of surface qualities. Nature 447, 206–209.
    41. Ngan A, F. Durand, and W. Matusik. 2006. Image-driven navigation of analytical BRDF models. In Proceedings of the Eurographics Conference on Rendering Techniques (EGSR’06). 399–407. 
    42. F. Osu. 2012. How to answer “behavior based interview questions” – Interview tip. https:///www.youtube.com/watch?v=qKBubK0-798.
    43. S. Paris, S. W. Hasinoff, and J. Kautz. 2011. Local Laplacian filters: Edge-aware image processing with a Laplacian pyramid. ACM Trans. Graph. 30, 4. 
    44. G. Patterson, C. Xu, H. Su, and J. Hays. 2014. The sun attribute database: Beyond categories for deeper scene understanding. Int. J. Comput. Vis. 108, 1–2, 59–81. 
    45. J. Portilla and E. P. Simoncelli. 2000. A parametric texture model based on joint statistics of complex wavelet coefficients. Int. J. Comput. Vis. 40, 1, 49–70. 
    46. L. Sharan, Y. Li, I. Motoyoshi, S. Nishida, and E. H. Adelson. 2008. Image statistics for surface reflectance perception. J. Optic. Soc. Amer. A25, 4.
    47. L. Sharan, C. Liu, R. Rosenholtz, and E. H. Adelson. 2013. Recognizing materials using perceptually inspired features. Int. J. Comput. Vis. 103, 3, 348–371.
    48. E. P. Simoncelli and E. H. Adelson. 1996. Noise removal via Bayesian wavelet coring. In Proceedings of the IEEE International Conference on Image Processing (ICIP’96). 379–382.
    49. E. Stochter. 2013. Gargoyle. https://pixabay.com/en/gargoyle-fountain-lion-lion-head-205999/.
    50. C. Tomasi and R. Manduchi. 1998. Bilateral filtering for gray and color images. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’98).
    51. M. Trentacoste, R. Mantuik, and W. Heidrich. 2011. Blur-aware image downsizing. Comput. Graph. Forum 30, 2, 573–582.
    52. M. Trentacoste, R. Mantiuk, W. Heidrich, and F. Dufrot. 2012. Unsharp masking, countershading and halos: Enhancements or artifacts? Comput. Graph. Forum 30, 2, 573–582. 
    53. R. Vergne, P. Barla, R. Fleming, and X. Granier. 2012. Surface flows for image-based shading design. ACM Trans. Graph. 31, 3. 
    54. Videvo. 2014. Fresh grapes close up. http://www.videvo.net/video/fresh-grapes-close-up/2350.
    55. J. Von Der Pahlen, J. Jimenez, E. Danvoye, P. Devevec, G. Fyffe, and O. Alexander. 2014. Digital IRA and beyond: Creating real-time photoreal digital actors. In ACM SIGGRAPH Courses. ACM Press, New York, 1:1–1:384.
    56. G. J. Ward. 1992. Measuring and modeling anisotropic reflection. ACM Trans. Comput. Graph. 26, 2, 265–272. 
    57. L. West. 2006. Onion. https://www.flickr.com/photos/calliope/104314753/
    58. L. Xu, C. Lu, Y. Xu, and J. Jia. 2011. Image smoothing via L0 gradient minimization. ACM Trans. Graph. 30, 6. 
    59. L. Xu, Q. Yan, Y. Xia, and J. Jia. 2012. Structure extraction from texture via relative total variation. ACM Trans. Graph. 31, 6.

ACM Digital Library Publication:


Overview Page: