“BiggerPicture: data-driven image extrapolation using graph matching” by Wang, Lai, Liang, Martin and Hu
Conference:
Type(s):
Title:
- BiggerPicture: data-driven image extrapolation using graph matching
Session/Category Title: Paintings, Sketches and Buildings
Presenter(s)/Author(s):
Abstract:
Filling a small hole in an image with plausible content is well studied. Extrapolating an image to give a distinctly larger one is much more challenging—a significant amount of additional content is needed which matches the original image, especially near its boundaries. We propose a data-driven approach to this problem. Given a source image, and the amount and direction(s) in which it is to be extrapolated, our system determines visually consistent content for the extrapolated regions using library images. As well as considering low-level matching, we achieve consistency at a higher level by using graph proxies for regions of source and library images. Treating images as graphs allows us to find candidates for image extrapolation in a feasible time. Consistency of subgraphs in source and library images is used to find good candidates for the additional content; these are then further filtered. Region boundary curves are aligned to ensure consistency where image parts are joined using a photomontage method. We demonstrate the power of our method in image editing applications.
References:
1. Arbelaez, P., Maire, M., Fowlkes, C., and Malik, J. 2011. Contour detection and hierarchical image segmentation. IEEE Trans. Pat. Anal. Mach. Intell. 33, 5, 898–916.
2. Avidan, S., and Shamir, A. 2007. Seam carving for content-aware image resizing. ACM Trans. Graph. 26, 3 (July).
3. Baeza-Yates, R., and Valiente, G. 2000. An image similarity measure based on graph matching. In Proc. IEEE Symp. String Processing and Information Retrieval, 28–38.
4. Barnes, C., Shechtman, E., Finkelstein, A., and Goldman, D. B. 2009. PatchMatch: A randomized correspondence algorithm for structural image editing. ACM Trans. Graph. 28, 3, 24.
5. Chen, T., Cheng, M.-M., Tan, P., Shamir, A., and Hu, S.-M. 2009. Sketch2Photo: internet image montage. ACM Trans. Graph. 28, 5, 124.
6. Chen, T., Tan, P., Ma, L.-Q., Cheng, M.-M., Shamir, A., and Hu, S.-M. 2013. Poseshop: human image database construction and personalized content synthesis. IEEE Trans. Vis. Comp. Graph. 19, 5, 824–837.
7. Chia, A. Y.-S., Zhuo, S., Gupta, R. K., Tai, Y.-W., Cho, S.-Y., Tan, P., and Lin, S. 2011. Semantic colorization with internet images. ACM Trans. Graph. 30, 6, 156.
8. Conte, D., Foggia, P., Sansone, C., and Vento, M. 2004. Thirty years of graph matching in pattern recognition. Int. J. Pat. Recog. Art. Intell. 18, 03, 265–298.Cross Ref
9. Criminisi, A., Perez, P., and Toyama, K. 2003. Object removal by exemplar-based inpainting. In Proc. CVPR, vol. 2, 721–728.
10. Dale, K., Johnson, M. K., Sunkavalli, K., Matusik, W., and Pfister, H. 2009. Image restoration using online photo collections. In Proc. ICCV, 2217–2224.
11. Darabi, S., Shechtman, E., Barnes, C., Goldman, D. B., and Sen, P. 2012. Image melding: Combining inconsistent images using patch-based synthesis. ACM Trans. Graph. 31, 4, 82.
12. Farbman, Z., Hoffer, G., Lipman, Y., Cohen-Or, D., and Lischinski, D. 2009. Coordinates for instant image cloning. ACM Trans. Graph. 28, 3, 67.
13. Gould, S., Fulton, R., and Koller, D. 2009. Decomposing a scene into geometric and semantically consistent regions. In Proc. ICCV, 1–8.
14. HaCohen, Y., Shechtman, E., Goldman, D. B., and Lischinski, D. 2013. Optimizing color consistency in photo collections. ACM Trans. Graph. 32, 4, 85:1–85:9.
15. Hays, J., and Efros, A. A. 2007. Scene completion using millions of photographs. ACM Trans. Graph. 26, 3, 4.
16. Hlaoui, A., and Wang, S. 2002. A new algorithm for graph matching with application to content-based image retrieval. In Structural, Syntactic, and Statistical Pattern Recognition. 291–300.
17. Hu, S.-M., Chen, T., Xu, K., Cheng, M.-M., and Martin, R. R. 2013. Internet visual media processing: a survey with graphics and vision applications. The Visual Computer 29, 5, 393–405.
18. Hu, S.-M., Zhang, F.-L., Wang, M., Martin, R. R., and Wang, J. 2013. PatchNet: A patch-based image representation for interactive library-driven image editing. ACM Trans. Graph. 32, 6, 196.
19. Huang, H., Yin, K., Gong, M., Lischinski, D., Cohen-Or, D., Ascher, U., and Chen, B. 2013. Mind the gap: Tele-registration for structure-driven image completion. ACM Trans. Graph. 32, 6, 174.
20. Jia, J., and Tang, C.-K. 2008. Image stitching using structure deformation. IEEE Trans. Pat. Anal. Mach. Intell. 30, 4, 617–631.
21. Johnson, M. K., Dale, K., Avidan, S., Pfister, H., Freeman, W. T., and Matusik, W. 2011. CG2Real: Improving the realism of computer generated images using a large collection of photographs. IEEE Trans. Vis. Comp. Graph. 17, 9, 1273–1285.
22. Kaneva, B., Sivic, J., Torralba, A., Avidan, S., and Freeman, W. T. 2010. Infinite images: Creating and exploring a large photorealistic virtual space. In Proceedings of the IEEE.
23. Kopf, J., Neubert, B., Chen, B., Cohen, M. F., Cohen-Or, D., Deussen, O., Uyttendaele, M., and Lischinski, D. 2008. Deep photo: Model-based photograph enhancement and viewing. ACM Trans. Graph. 27, 5, 116:1–116:10.
24. Kopf, J., Kienzle, W., Drucker, S., and Kang, S. B. 2012. Quality prediction for image completion. ACM Trans. Graph. 31, 6, 131.
25. Kwatra, V., Schödl, A., Essa, I., Turk, G., and Bobick, A. 2003. Graphcut textures: image and video synthesis using graph cuts. ACM Trans. Graph. 22, 3, 277–286.
26. Lazebnik, S., Schmid, C., and Ponce, J. 2006. Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. In Proc. CVPR, vol. 2, 2169–2178.
27. Lee, Y. J., and Grauman, K. 2010. Object-graphs for context-aware category discovery. In Proc. CVPR, 1–8.
28. Levin, A., Zomet, A., and Weiss, Y. 2003. Learning how to inpaint from global image statistics. In Proc. ICCV, 305–312.
29. Li, F.-F., and Perona, P. 2005. A Bayesian hierarchical model for learning natural scene categories. In Proc. CVPR, vol. 2, 524–531.
30. Liu, Y., and Yu, Y. 2012. Interactive image segmentation based on level sets of probabilities. IEEE Trans. Vis. Comp. Graph. 18, 2, 202–213.
31. Liu, C., Yuen, J., and Torralba, A. 2009. Nonparametric scene parsing: Label transfer via dense scene alignment. In Proc. CVPR, 1972–1979.
32. Malisiewicz, T., and Efros, A. 2009. Beyond categories: The visual memex model for reasoning about object relationships. In Proc. NIPS, 1222–1230.
33. Martin, D., Fowlkes, C., Tal, D., and Malik, J. 2001. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proc. ICCV, vol. 2, 416–423.
34. Oliva, A., and Torralba, A. 2001. Modeling the shape of the scene: A holistic representation of the spatial envelope. Int. J. Comp. Vis. 42, 3, 145–175.
35. Pérez, P., Gangnet, M., and Blake, A. 2003. Poisson image editing. ACM Trans. Graph. 22, 3, 313–318.
36. Poleg, Y., and Peleg, S. 2012. Alignment and mosaicing of non-overlapping images. In Proc. ICCP, IEEE, 1–8.
37. Shih, Y., Paris, S., Durand, F., and Freeman, W. T. 2013. Data-driven hallucination of different times of day from a single outdoor photo. ACM Trans. Graph. 32, 6 (Nov.), 200:1–200:11.
38. Sivic, J., and Zisserman, A. 2003. Video google: A text retrieval approach to object matching in videos. In Proc. ICCV, 1470–1477.
39. Sivic, J., Kaneva, B., Torralba, A., Avidan, S., and Freeman, W. T. 2008. Creating and exploring a large photorealistic virtual space. In Proc. CVPR Workshop, 1–8.
40. Tighe, J., and Lazebnik, S. 2013. Superparsing. Int. J. Comp. Vis. 101, 2, 329–349.
41. Torralba, A., Fergus, R., and Freeman, W. T. 2008. 80 million tiny images: A large data set for nonparametric object and scene recognition. IEEE Trans. Pat. Anal. Mach. Intell. 30, 11, 1958–1970.
42. Wexler, Y., Shechtman, E., and Irani, M. 2007. Space-time completion of video. IEEE Trans. Pat. Anal. Mach. Intell. 29, 3, 463–476.
43. Xiao, J., Hays, J., Ehinger, K. A., Oliva, A., and Torralba, A. 2010. Sun database: Large-scale scene recognition from abbey to zoo. In Proc. CVPR, IEEE, 3485–3492.
44. Yang, F., and Li, B. 2012. Unsupervised learning of spatial structures shared among images. The Visual Computer 28, 2, 175–180.
45. Zhang, Y., Xiao, J., Hays, J., and Tan, P. 2013. Frame-break: dramatic image extrapolation by guided shift-maps. In Proc. CVPR, 1171–1178.
46. Zhou, F., and De la Torre, F. 2012. Factorized graph matching. In Proc. CVPR, 127–134.
