“Transient attributes for high-level understanding and editing of outdoor scenes” by Laffont, Ren, Tao, Qian and Hays

  • ©Pierre-Yves Laffont, Zhile Ren, Xiaofeng Tao, Chao Qian, and James Hays




    Transient attributes for high-level understanding and editing of outdoor scenes

Session/Category Title: Changing Your Perception




    We live in a dynamic visual world where the appearance of scenes changes dramatically from hour to hour or season to season. In this work we study “transient scene attributes” — high level properties which affect scene appearance, such as “snow”, “autumn”, “dusk”, “fog”. We define 40 transient attributes and use crowdsourcing to annotate thousands of images from 101 webcams. We use this “transient attribute database” to train regressors that can predict the presence of attributes in novel images. We demonstrate a photo organization method based on predicted attributes. Finally we propose a high-level image editing method which allows a user to adjust the attributes of a scene, e.g. change a scene to be “snowy” or “sunset”. To support attribute manipulation we introduce a novel appearance transfer technique which is simple and fast yet competitive with the state-of-the-art. We show that we can convincingly modify many transient attributes in outdoor scenes.


    1. An, X., and Pellacini, F. 2010. User-controllable color transfer. Comput. Graph. Forum 29, 2. Google ScholarDigital Library
    2. Bell, S., Upchurch, P., Snavely, N., and Bala, K. 2013. Opensurfaces: A richly annotated catalog of surface appearance. ACM Trans. Graph. (proc. SIGGRAPH) 32, 4. Google ScholarDigital Library
    3. Berthouzoz, F., Li, W., Dontcheva, M., and Agrawala, M. 2011. A framework for content-adaptive photo manipulation macros. ACM Trans. Graph. 30, 5. Google ScholarDigital Library
    4. Bonneel, N., Sunkavalli, K., Paris, S., and Pfister, H. 2013. Example-based video color grading. ACM Trans. Graph. (proc. SIGGRAPH) 32. Google ScholarDigital Library
    5. Bychkovsky, V., Paris, S., Chan, E., and Durand, F. 2011. Learning photographic global tonal adjustment with a database of input / output image pairs. In CVPR. Google ScholarDigital Library
    6. Caicedo, J. C., Kapoor, A., and Kang, S. B. 2011. Collaborative personalization of image enhancement. In CVPR. Google ScholarDigital Library
    7. Chen, J., Paris, S., and Durand, F. 2007. Real-time edge-aware image processing with the bilateral grid. ACM Trans. Graph. (proc. SIGGRAPH) 26, 3. Google ScholarDigital Library
    8. Cheng, M.-M., Zheng, S., Lin, W.-Y., Vineet, V., Sturgess, P., Crook, N., Mitra, N., and Torr, P. 2014. Imagespirit: Verbal guided image parsing. ACM Trans. Graph..Google ScholarDigital Library
    9. Comaniciu, D., and Meer, P. 2002. Mean shift: a robust approach toward feature space analysis. IEEE Trans. PAMI 24. Google ScholarDigital Library
    10. Cusano, C., Gasparini, F., and Schettini, R. 2012. Color transfer using semantic image annotation. In SPIE, vol. 8299. Google ScholarDigital Library
    11. Dale, K., Johnson, M. K., Sunkavalli, K., Matusik, W., and Pfister, H. 2009. Image restoration using online photo collections. In ICCV.Google Scholar
    12. Dhar, S., Ordonez, V., and Berg, T. L. 2011. High level describable attributes for predicting aesthetics and interestingness. In CVPR. Google ScholarDigital Library
    13. Eitz, M., Hays, J., and Alexa, M. 2012. How do humans sketch objects? ACM Trans. Graph. (proc. SIGGRAPH) 31, 4. Google ScholarDigital Library
    14. Farhadi, A., Endres, I., Hoiem, D., and Forsyth, D. 2009. Describing objects by their attributes. In CVPR. Google ScholarDigital Library
    15. Fattal, R. 2008. Single image dehazing. ACM Trans. Graph. (proc. SIGGRAPH) 27, 3. Google ScholarDigital Library
    16. Ferrari, V., and Zisserman, A. 2007. Learning visual attributes. In NIPS.Google Scholar
    17. Garg, R., Du, H., Seitz, S. M., and Snavely, N. 2009. The dimensionality of scene appearance. In ICCV.Google Scholar
    18. Hertzmann, A., Jacobs, C. E., Oliver, N., Curless, B., and Salesin, D. H. 2001. Image analogies. In SIGGRAPH. Google ScholarDigital Library
    19. Hoiem, D., Efros, A. A., and Hebert, M. 2007. Recovering surface layout from an image. Int. J. Comput. Vision 75, 1. Google ScholarDigital Library
    20. Jacobs, N., Roman, N., and Pless, R. 2007. Consistent temporal variations in many outdoor scenes. In CVPR.Google Scholar
    21. Johnson, M. K., Dale, K., Avidan, S., Pfister, H., Freeman, W. T., and Matusik, W. 2011. Cg2real: Improving the realism of computer generated images using a large collection of photographs. IEEE Trans. Vis. Comput. Graph. 17, 9. Google ScholarDigital Library
    22. Kang, S. B., Kapoor, A., and Lischinski, D. 2010. Personalization of image enhancement. In CVPR.Google Scholar
    23. Kopf, J., Cohen, M. F., Lischinski, D., and Uyttendaele, M. 2007. Joint bilateral upsampling. ACM Trans. Graph. (proc. SIGGRAPH) 26, 3. Google ScholarDigital Library
    24. Kovashka, A., Parikh, D., and Grauman, K. 2012. Whittle-search: Image search with relative attribute feedback. In CVPR. Google ScholarDigital Library
    25. Kumar, N., Berg, A., Belhumeur, P., and Nayar, S. 2011. Describable visual attributes for face verification and image search. IEEE Trans. PAMI 33, 10. Google ScholarDigital Library
    26. Laffont, P.-Y., Bousseau, A., Paris, S., Durand, F., and Drettakis, G. 2012. Coherent intrinsic images from photo collections. ACM Trans. Graph. (proc. SIGGRAPH Asia) 31, 6. Google ScholarDigital Library
    27. Lalonde, J.-F., Hoiem, D., Efros, A. A., Rother, C., Winn, J., and Criminisi, A. 2007. Photo clip art. ACM Trans. Graph. (proc. SIGGRAPH) 26, 3. Google ScholarDigital Library
    28. Lalonde, J.-F., Efros, A., and Narasimhan, S. 2009. Web-cam clip art: Appearance and illuminant transfer from time-lapse sequences. ACM Trans. Graph. (proc. SIGGRAPH Asia) 28, 5. Google ScholarDigital Library
    29. Liu, Q., Ihler, A., and Steyvers, M. 2013. Scoring workers in crowdsourcing: how many control questions are enough? In NIPS.Google Scholar
    30. Matusik, W., Pfister, H., Brand, M., and McMillan, L. 2003. A data-driven reflectance model. ACM Trans. Graph. (proc. SIGGRAPH) 22, 3. Google ScholarDigital Library
    31. Murphy, K. P. 2012. Machine Learning: A Probabilistic Perspective. The MIT Press. Google ScholarDigital Library
    32. Narasimhan, S., Wang, C., and Nayar, S. 2002. All the images of an outdoor scene. In ECCV. Google ScholarDigital Library
    33. Parikh, D., and Grauman, K. 2011. Relative attributes. In ICCV. Google ScholarDigital Library
    34. Patterson, G., and Hays, J. 2012. Sun attribute database: Discovering, annotating, and recognizing scene attributes. In CVPR. Google ScholarDigital Library
    35. Perronnin, F., Sánchez, J., and Mensink, T. 2010. Improving the fisher kernel for large-scale image classification. In ECCV. Google ScholarDigital Library
    36. Pitié, F., Kokaram, A., and Dahyot, R. 2005. N-Dimensional Probability Density Function Transfer and its Application to Colour Transfer. In ICCV. Google ScholarDigital Library
    37. Pouli, T., and Reinhard, E. 2011. Progressive color transfer for images of arbitrary dynamic range. Computers & Graphics 35. Google ScholarDigital Library
    38. Reinhard, E., Ashikhmin, M., Gooch, B., and Shirley, P. 2001. Color transfer between images. IEEE Comput. Graph. Appl. 21, 5. Google ScholarDigital Library
    39. Scholkopf, B., Smola, A., Williamson, R., and Bartlett, P. 2000. New support vector algorithms. Neural Computation 12. Google ScholarDigital Library
    40. Shih, Y., Paris, S., Durand, F., and Freeman, W. T. 2013. Data-driven hallucination of different times of day from a single outdoor photo. ACM Trans. Graph. (proc. SIGGRAPH Asia) 32, 6. Google ScholarDigital Library
    41. Snavely, N., Seitz, S. M., and Szeliski, R. 2006. Photo tourism: exploring photo collections in 3D. ACM Trans. Graph. (proc. SIGGRAPH) 25, 3. Google ScholarDigital Library
    42. Sunkavalli, K., Matusik, W., Pfister, H., and Rusinkiewicz, S. 2007. Factored time-lapse video. ACM Trans. Graph. (proc. SIGGRAPH) 26, 3. Google ScholarDigital Library
    43. Tao, L., Yuan, L., and Sun, J. 2009. Skyfinder: Attribute-based sky image search. ACM Trans. Graph. (proc. SIGGRAPH) 28, 3. Google ScholarDigital Library
    44. Wu, F., Dong, W., Kong, Y., Mei, X., Paul, J.-C., and Zhang, X. 2013. Content-Based Colour Transfer. Comput. Graph. Forum 32, 1.Google ScholarDigital Library
    45. Xiao, J., Hays, J., Ehinger, K. A., Oliva, A., and Torralba, A. 2010. Sun database: Large-scale scene recognition from abbey to zoo. In CVPR.Google Scholar
    46. Yu, Y., and Malik, J. 1998. Recovering photometric properties of architectural scenes from photographs. In SIGGRAPH. Google ScholarDigital Library

ACM Digital Library Publication:

Overview Page: