“What makes Paris look like Paris?” by Doersch, Singh, Gupta, Sivic and Efros

Given a large repository of geotagged imagery, we seek to automatically find visual elements, e. g. windows, balconies, and street signs, that are most distinctive for a certain geo-spatial area, for example the city of Paris. This is a tremendously difficult task as the visual features distinguishing architectural elements of different places can be very subtle. In addition, we face a hard search problem: given all possible patches in all images, which of them are both frequently occurring and geographically informative? To address these issues, we propose to use a discriminative clustering approach able to take into account the weak geographic supervision. We show that geographically representative image elements can be discovered automatically from Google Street View imagery in a discriminative manner. We demonstrate that these elements are visually interpretable and perceptually geo-informative. The discovered visual elements can also support a variety of computational geography tasks, such as mapping architectural correspondences and influences within and across cities, finding representative elements at different geo-spatial scales, and geographically-informed image retrieval.

1. Berg, T., and Berg, A. 2009. Finding iconic images. In The 2nd Internet Vision Workshop at Conference on Computer Vision and Pattern Recognition (CVPR).Google Scholar
2. Bourdev, L., and Malik, J. 2009. Poselets: Body part detectors trained using 3D human pose annotations. In IEEE 12th International Conference on Computer Vision (ICCV), 1365–1372.Google Scholar
3. Chen, D., Baatz, G., Koser, K., Tsai, S., Vedantham, R., Pylvanainen, T., Roimela, K., Chen, X., Bach, J., Pollefeys, M., Girod, B., and Grzeszczuk, R. 2011. City-scale landmark identification on mobile devices. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 737–744. Google ScholarDigital Library
4. Chum, O., Perdoch, M., and Matas, J. 2009. Geometric min-hashing: Finding a (thick) needle in a haystack. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 17–24.Google Scholar
5. Crandall, D., Backstrom, L., Huttenlocher, D., and Kleinberg, J. 2009. Mapping the world’s photos. In Proceedings of the 18th International Conference on World Wide Web (WWW), 761–770. Google ScholarDigital Library
6. Dalal, N., and Triggs, B. 2005. Histograms of oriented gradients for human detection. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), vol. 1, 886–893. Google ScholarDigital Library
7. Fiss, J., Agarwala, A., and Curless, B. 2011. Candid portrait selection from video. ACM Transactions on Graphics (SIGGRAPH Asia) 30, 6, 128. Google ScholarDigital Library
8. Fulkerson, B., Vedaldi, A., and Soatto, S. 2008. Localizing objects with smart dictionaries. In European Conference on Computer Vision (ECCV), 179–192. Google ScholarDigital Library
9. Gong, Y., and Lazebnik, S. 2011. Iterative quantization: A procrustean approach to learning binary codes. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 817–824. Google ScholarDigital Library
10. Gronat, P., Havlena, M., Sivic, J., and Pajdla, T. 2011. Building streetview datasets for place recognition and city reconstruction. Tech. Rep. CTU-CMP-2011-16, Czech Tech Univ.Google Scholar
11. Hays, J., and Efros, A. 2008. Im2gps: estimating geographic information from a single image. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 1–8.Google Scholar
12. Kalogerakis, E., Vesselova, O., Hays, J., Efros, A., and Hertzmann, A. 2009. Image sequence geolocation with human travel priors. In IEEE 12th International Conference on Computer Vision (ICCV), 253–260.Google Scholar
13. Karlinsky, L., Dinerstein, M., and Ullman, S. 2009. Unsupervised feature optimization (ufo): Simultaneous selection of multiple features with their detection parameters. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 1263–1270.Google Scholar
14. Knopp, J., Sivic, J., and Pajdla, T. 2010. Avoiding confusing features in place recognition. In European Conference on Computer Vision (ECCV), 748–761. Google ScholarDigital Library
15. Lee, Y., and Grauman, K. 2009. Foreground focus: Unsupervised learning from partially matching images. International Journal of Computer Vision (IJCV) 85, 2, 143–166. Google ScholarDigital Library
16. Li, X., Wu, C., Zach, C., Lazebnik, S., and Frahm, J.-M. 2008. Modeling and recognition of landmark image collections using iconic scene graphs. In European Conference on Computer Vision (ECCV), 427–440. Google ScholarDigital Library
17. Li, Y., Crandall, D., and Huttenlocher, D. 2009. Landmark classification in large-scale image collections. In IEEE 12th International Conference on Computer Vision (ICCV), 1957–1964.Google Scholar
18. Li, L., Su, H., Xing, E., and Fei-Fei, L. 2010. Object bank: A high-level image representation for scene classification and semantic feature sparsification. In Advances in Neural Information Processing Systems (NIPS), vol. 24.Google Scholar
19. Loyer, F. 1988. Paris nineteenth century: architecture and urbanism, 1st american ed. Abbeville Press, New York.Google Scholar
20. Moosmann, F., Triggs, B., and Jurie, F. 2007. Fast discriminative visual codebooks using randomized clustering forests. In Advances in Neural Information Processing Systems (NIPS), vol. 19.Google Scholar
21. Mueller, P., Wonka, P., Haegler, S., Ulmer, A., and Van Gool, L. 2006. Procedural modeling of buildings. ACM Transactions on Graphics (SIGGRAPH) 25, 3, 614–623. Google ScholarDigital Library
22. Oliva, A., and Torralba, A. 2006. Building the gist of a scene: The role of global image features in recognition. Progress in brain research 155, 23–36.Google Scholar
23. Paik, K. 2006. The Art of Ratatouille. Chronicle Books.Google Scholar
24. Quack, T., Leibe, B., and Van Gool, L. 2008. Worldscale mining of objects and events from community photo collections. In Proceedings of the International Conference on Content-based Image and Video Retrieval (CIVR), 47–56. Google ScholarDigital Library
25. Russell, B. C., Efros, A. A., Sivic, J., Freeman, W. T., and Zisserman, A. 2006. Using multiple segmentations to discover objects and their extent in image collections. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 1605–1614. Google ScholarDigital Library
26. Schindler, G., Brown, M., and Szeliski, R. 2007. City-scale location recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 1–7.Google Scholar
27. Shotton, J., Johnson, M., and Cipolla, R. 2008. Semantic texton forests for image categorization and segmentation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 1–8.Google Scholar
28. Shrivastava, A., Malisiewicz, T., Gupta, A., and Efros, A. A. 2011. Data-driven visual similarity for cross-domain image matching. ACM Transactions on Graphics (SIGGRAPH Asia) 30, 6, 154. Google ScholarDigital Library
29. Simon, I., Snavely, N., and Seitz, S. M. 2007. Scene summarization for online image collections. In IEEE 11th International Conference on Computer Vision (ICCV), 1–8.Google Scholar
30. Singh, S., Gupta, A., and Efros, A. A. 2012. Unsupervised discovery of mid-level discriminative patches. arXiv:1205.3137 {cs.CV}. Google ScholarDigital Library
31. Sivic, J., and Zisserman, A. 2003. Video google: A text retrieval approach to object matching in videos. In IEEE 9th International Conference on Computer Vision (ICCV), 1470–1477. Google ScholarDigital Library
32. Sutcliffe, A. 1996. Paris: an architectural history. Yale University Press.Google Scholar
33. Teboul, O., Simon, L., Koutsourakis, P., and Paragios, N. 2010. Segmentation of building facades using procedural shape priors. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 3105–3112.Google Scholar
34. Torralba, A., and Oliva, A. 2003. Statistics of natural image categories. Network: Computation in Neural Systems, 391–412.Google Scholar
35. Zheng, Y.-T., Zhao, M., Song, Y., Adam, H., Buddemeier, U., Bissacco, A., Brucher, F., Chua, T.-S., and Neven, H. 2009. Tour the world: building a web-scale landmark recognition engine. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 1085–1092.Google Scholar