“Discovering pattern structure using differentiable compositing” by Chintala, Guerrero, Fisher, Li and Mitra
Conference:
Type(s):
Title:
- Discovering pattern structure using differentiable compositing
Session/Category Title: Shape Analysis
Presenter(s)/Author(s):
Abstract:
Patterns, which are collections of elements arranged in regular or near-regular arrangements, are an important graphic art form and widely used due to their elegant simplicity and aesthetic appeal. When a pattern is encoded as a flat image without the underlying structure, manually editing the pattern is tedious and challenging as one has to both preserve the individual element shapes and their original relative arrangements. State-of-the-art deep learning frameworks that operate at the pixel level are unsuitable for manipulating such patterns. Specifically, these methods can easily disturb the shapes of the individual elements or their arrangement, and thus fail to preserve the latent structures of the input patterns. We present a novel differentiable compositing operator using pattern elements and use it to discover structures, in the form of a layered representation of graphical objects, directly from raw pattern images. This operator allows us to adapt current deep learning based image methods to effectively handle patterns. We evaluate our method on a range of patterns and demonstrate superiority in the context of pattern manipulations when compared against state-of-the-art pixel- or point-based alternatives.
References:
1. Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas. 2018. Learning Representations and Generative Models for 3D Point Clouds. In Proceedings of the 35th International Conference on Machine Learning, Vol. 80. 40–49.Google Scholar
2. Min Bai, Wenjie Luo, Kaustav Kundu, and Raquel Urtasun. 2016. Exploiting semantic information and deep matching for optical flow. In European Conference on Computer Vision. Springer, 154–170.Google ScholarCross Ref
3. Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman. 2009. Patch-Match: A randomized correspondence algorithm for structural image editing. In ACM Transactions on Graphics (ToG), Vol. 28. ACM, 24.Google ScholarDigital Library
4. Kai Briechle and Uwe D Hanebeck. 2001. Template matching using fast normalized cross correlation. In Optical Pattern Recognition XII, Vol. 4387. International Society for Optics and Photonics, 95–102.Google ScholarCross Ref
5. Jiaxin Cheng, Yue Wu, Wael AbdAlmageed, and Premkumar Natarajan. 2019. Qatm: quality-aware template matching for deep learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 11553–11562.Google ScholarCross Ref
6. Ming-Ming Cheng, Fang-Lue Zhang, Niloy J Mitra, Xiaolei Huang, and Shi-Min Hu. 2010. Repfinder: finding approximately repeated scene elements for image editing. ACM Transactions on Graphics (TOG) 29, 4 (2010), 1–8.Google ScholarDigital Library
7. Tali Dekel, Shaul Oron, Michael Rubinstein, Shai Avidan, and William T Freeman. 2015. Best-buddies similarity for robust template matching. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2021–2029.Google ScholarCross Ref
8. Alexei A Efros and William T Freeman. 2001. Image quilting for texture synthesis and transfer. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques. 341–346.Google ScholarDigital Library
9. Leon A Gatys, Alexander S Ecker, and Matthias Bethge. 2016. Image style transfer using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2414–2423.Google ScholarCross Ref
10. Paul Guerrero, Gilbert Bernstein, Wilmot Li, and Niloy J. Mitra. 2016. PATEX: Exploring Pattern Variations. ACM Trans. Graph. 35, 4 (2016), 48:1–48:13. Google ScholarDigital Library
11. Shi Guo, Zifei Yan, Kai Zhang, Wangmeng Zuo, and Lei Zhang. 2019. Toward Convolutional Blind Denoising of Real Photographs. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google Scholar
12. Xufeng Han, Thomas Leung, Yangqing Jia, Rahul Sukthankar, and Alexander C Berg. 2015. Matchnet: Unifying feature and metric learning for patch-based matching. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3279–3286.Google Scholar
13. Muhammad Shehzad Hanif. 2019. Patch match networks: Improved two-channel and Siamese networks for image patch matching. Pattern Recognition Letters 120 (2019), 54–61.Google ScholarCross Ref
14. Daniel Heck, Thomas Schlomer, and Oliver Deussen. 2013. Blue noise sampling with controlled aliasing. ACM Transactions on Graphics 32, 3 (6 2013). Google ScholarDigital Library
15. Yacov Hel-Or and Hagit Hel-Or. 2005. Real-time pattern matching using projection kernels. IEEE transactions on pattern analysis and machine intelligence 27, 9 (2005), 1430–1445.Google ScholarDigital Library
16. Aaron Hertzmann, Charles E Jacobs, Nuria Oliver, Brian Curless, and David H Salesin. 2001. Image analogies. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques. 327–340.Google ScholarDigital Library
17. T. Hurtut, P. E. Landes, J. Thollot, Y. Gousseau, R. Drouillhet, and J. F. Coeurjolly. 2009. Appearance-guided synthesis of element arrangements by example. In NPAR Symposium on Non-Photorealistic Animation and Rendering. 51–60. Google ScholarDigital Library
18. Janine. Illian. 2008. Statistical analysis and modelling of spatial point patterns. John Wiley. 534 pages.Google Scholar
19. Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. 2015. Spatial transformer networks. In Advances in neural information processing systems. 2017–2025.Google Scholar
20. Y. Jing, Y. Yang, Z. Feng, J. Ye, Y. Yu, and M. Song. 2019. Neural Style Transfer: A Review. IEEE Transactions on Visualization and Computer Graphics (2019), 1–1.Google Scholar
21. Tero Karras, Samuli Laine, and Timo Aila. 2019. A Style-Based Generator Architecture for Generative Adversarial Networks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google ScholarCross Ref
22. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. arXiv:cs.LG/1412.6980Google Scholar
23. Simon Korman, Daniel Reichman, Gilad Tsur, and Shai Avidan. 2013. Fast-match: Fast affine template matching. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2331–2338.Google ScholarDigital Library
24. Vivek Kwatra, Arno Schödl, Irfan Essa, Greg Turk, and Aaron Bobick. 2003. Graphcut Textures: Image and Video Synthesis Using Graph Cuts. ACM Trans. Graph. 22, 3 (July 2003), 277–286. Google ScholarDigital Library
25. Pierre-Edouard Landes, Bruno Galerne, and Thomas Hurtut. 2013. A Shape-Aware Model for Discrete Texture Synthesis. Computer Graphics Forum 32, 4 (7 2013), 67–76. Google ScholarDigital Library
26. Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miika Aittala, and Timo Aila. 2018. Noise2Noise: Learning Image Restoration without Clean Data. In Proceedings of the 35th International Conference on Machine Learning, Vol. 80. 2965–2974.Google Scholar
27. Thomas Leimkühler, Gurprit Singh, Karol Myszkowski, Hans-Peter Seidel, and Tobias Ritschel. 2019. Deep Point Correlation Design. ACM Trans. Graph. 38, 6, Article Article 226 (Nov. 2019), 17 pages. Google ScholarDigital Library
28. J.P. Lewis. 1995. Fast Template Matching. In Proc. Vision Interface. 120–123.Google Scholar
29. Chun-Liang Li, Manzil Zaheer, Yang Zhang, Barnabas Poczos, and Ruslan Salakhutdinov. 2018. Point Cloud GAN. (10 2018). http://arxiv.org/abs/1810.05795Google Scholar
30. Lin Liang, Ce Liu, Ying-Qing Xu, Baining Guo, and Heung-Yeung Shum. 2001. Real-Time Texture Synthesis by Patch-Based Sampling. ACM Trans. Graph. 20, 3 (July 2001), 127–150. Google ScholarDigital Library
31. Hsueh-Ti Derek Liu, Michael Tao, and Alec Jacobson. 2018. Paparazzi: surface editing by way of multi-view image processing. ACM Trans. Graph. 37, 6 (2018), 221–1.Google Scholar
32. Shichen Liu, Tianye Li, Weikai Chen, and Hao Li. 2019. Soft rasterizer: A differentiable renderer for image-based 3d reasoning. In Proceedings of the IEEE International Conference on Computer Vision. 7708–7717.Google ScholarCross Ref
33. Guillaume Loubet, Nicolas Holzschuch, and Wenzel Jakob. 2019. Reparameterizing discontinuous integrands for differentiable rendering. ACM Transactions on Graphics (TOG) 38, 6 (2019), 1–14.Google ScholarDigital Library
34. Wenjie Luo, Alexander G Schwing, and Raquel Urtasun. 2016. Efficient deep learning for stereo matching. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5695–5703.Google ScholarCross Ref
35. Chongyang Ma, Li-Yi Wei, and Xin Tong. 2011. Discrete element textures. ACM Transactions on Graphics (TOG) 30, 4 (2011), 1–10.Google ScholarDigital Library
36. Stefano Mattoccia, Federico Tombari, and Luigi Di Stefano. 2008. Fast full-search equivalent template matching by enhanced bounded correlation. IEEE transactions on image processing 17, 4 (2008), 528–538.Google ScholarDigital Library
37. James McCann and Nancy Pollard. 2009. Local Layering. In ACM SIGGRAPH 2009 Papers. Article 84, 7 pages.Google Scholar
38. Iaroslav Melekhov, Juho Kannala, and Esa Rahtu. 2016. Image patch matching using convolutional descriptors with euclidean distance. In Asian Conference on Computer Vision. Springer, 638–653.Google Scholar
39. Thomas Müller, Brian McWilliams, Fabrice Rousselle, Markus Gross, and Jan Novák. 2018. Neural Importance Sampling. (8 2018). http://arxiv.org/abs/1808.03856Google Scholar
40. Wanli Ouyang, Federico Tombari, Stefano Mattoccia, Luigi Di Stefano, and Wai-Kuen Cham. 2011. Performance evaluation of full search equivalent pattern matching algorithms. IEEE transactions on pattern analysis and machine intelligence 34, 1 (2011), 127–143.Google Scholar
41. A. Cengiz Öztireli and Markus Grossy. 2012. Analysis and synthesis of point distributions based on pair correlation. In ACM Transactions on Graphics, Vol. 31. Google ScholarDigital Library
42. Ignacio Rocco, Relja Arandjelovic, and Josef Sivic. 2017. Convolutional neural network architecture for geometric matching. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 6148–6157.Google ScholarCross Ref
43. Amir Rosenberger, Daniel Cohen-Or, and Dani Lischinski. 2009. Layered shape synthesis: automatic generation of control maps for non-stationary textures. ACM Transactions on Graphics (TOG) 28, 5 (2009), 1–9.Google ScholarDigital Library
44. Carsten Rother, Vladimir Kolmogorov, and Andrew Blake. 2004. ” GrabCut” interactive foreground extraction using iterated graph cuts. ACM transactions on graphics (TOG) 23, 3 (2004), 309–314.Google Scholar
45. Riccardo Roveri, A. Cengiz Öztireli, and Markus Gross. 2017. General Point Sampling with Adaptive Density and Correlations. Computer Graphics Forum 36, 2 (5 2017), 107–117. Google ScholarDigital Library
46. Riccardo Roveri, A Cengiz Öztireli, Sebastian Martin, Barbara Solenthaler, and Markus Gross. 2015. Example based repetitive structure synthesis. In Computer Graphics Forum, Vol. 34. Wiley Online Library, 39–52.Google Scholar
47. Tamar Rott Shaham, Tali Dekel, and Tomer Michaeli. 2019. SinGAN: Learning a Generative Model From a Single Natural Image. In The IEEE International Conference on Computer Vision (ICCV).Google ScholarCross Ref
48. Assaf Shocher, Shai Bagon, Phillip Isola, and Michal Irani. 2019. InGAN: Capturing and Retargeting the “DNA” of a Natural Image. In The IEEE International Conference on Computer Vision (ICCV).Google ScholarCross Ref
49. Edgar Simo-Serra, Eduard Trulls, Luis Ferraz, Iasonas Kokkinos, Pascal Fua, and Francesc Moreno-Noguer. 2015. Discriminative learning of deep convolutional feature point descriptors. In Proceedings of the IEEE International Conference on Computer Vision. 118–126.Google ScholarDigital Library
50. Karen Simonyan and Andrew Zisserman. 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition. In International Conference on Learning Representations.Google Scholar
51. Ondrej Št’ava, Bedrich Beneš, Radomir Měch, Daniel G Aliaga, and Peter Krištof. 2010. Inverse procedural modeling by automatic generation of L-systems. In Computer Graphics Forum, Vol. 29. Wiley Online Library, 665–674.Google Scholar
52. Yongbin Sun, Yue Wang, Ziwei Liu, Joshua E. Siegel, and Sanjay E. Sarma. 2018. Point-Grow: Autoregressively Learned Point Cloud Generation with Self-Attention. (10 2018). http://arxiv.org/abs/1810.05591Google Scholar
53. Itamar Talmi, Roey Mechrez, and Lihi Zelnik-Manor. 2017. Template matching with deformable diversity similarity. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 175–183.Google ScholarCross Ref
54. Jerry O Talton, Yu Lou, Steve Lesser, Jared Duke, Radomír Měch, and Vladlen Koltun. 2011. Metropolis procedural modeling. ACM Transactions on Graphics (TOG) 30, 2 (2011), 1–14.Google ScholarDigital Library
55. Siyu Tang, Bjoern Andres, Mykhaylo Andriluka, and Bernt Schiele. 2016. Multi-person tracking by multicut and deep matching. In European Conference on Computer Vision. Springer, 100–111.Google ScholarCross Ref
56. James Thewlis, Shuai Zheng, Philip HS Torr, and Andrea Vedaldi. 2016. Fully-trainable deep matching. In British Machine Vision Conference.Google ScholarCross Ref
57. Peihan Tu, Dani Lischinski, and Hui Huang. 2019. Point Pattern Synthesis via Irregular Convolution. In Computer Graphics Forum, Vol. 38. Wiley Online Library, 109–122.Google Scholar
58. Dmitry Ulyanov, Vadim Lebedev, Andrea Vedaldi, and Victor Lempitsky. 2016. Texture Networks: Feed-forward Synthesis of Textures and Stylized Images. (3 2016). http://arxiv.org/abs/1603.03417Google Scholar
59. Shou-Der Wei and Shang-Hong Lai. 2008. Fast template matching based on normalized cross correlation with adaptive multilevel winner update. IEEE Transactions on Image Processing 17, 11 (2008), 2227–2235.Google ScholarDigital Library
60. Yue Wu, Wael Abd-Almageed, and Prem Natarajan. 2017. Deep matching and validation network: An end-to-end solution to constrained image splicing localization and detection. In Proceedings of the 25th ACM international conference on Multimedia. 1480–1502.Google ScholarDigital Library
61. Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S. Huang. 2018. Generative Image Inpainting With Contextual Attention. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google Scholar
62. Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S. Huang. 2019. Free-Form Image Inpainting With Gated Convolution. In The IEEE International Conference on Computer Vision (ICCV).Google Scholar
63. Yahan Zhou, Haibin Huang, Li Yi Wei, and Rui Wang. 2012. Point sampling with general noise spectrum. ACM Transactions on Graphics 31, 4 (7 2012). Google ScholarDigital Library
64. Yang Zhou, Hui Huang, Zhen Zhu, Xiang Bai, Dani Lischinski, and Daniel Cohen-Or. 2018. Non-Stationary Texture Synthesis by Adversarial Expansion Additional Key Words and Phrases: Example-based texture synthesis, non-stationary textures, generative adversarial networks ACM Reference Format. ACM Trans. Graph 37 (2018), 13. Google ScholarDigital Library
