“Differentiable vector graphics rasterization for editing and learning” by Li, Lukáč, Gharbi and Ragan-Kelley
Conference:
Type(s):
Title:
- Differentiable vector graphics rasterization for editing and learning
Session/Category Title: Differentiable Graphics
Presenter(s)/Author(s):
Abstract:
We introduce a differentiable rasterizer that bridges the vector graphics and raster image domains, enabling powerful raster-based loss functions, optimization procedures, and machine learning techniques to edit and generate vector content. We observe that vector graphics rasterization is differentiable after pixel prefiltering. Our differentiable rasterizer offers two prefiltering options: an analytical prefiltering technique and a multisampling anti-aliasing technique. The analytical variant is faster but can suffer from artifacts such as conflation. The multisampling variant is still efficient, and can render high-quality images while computing unbiased gradients for each pixel with respect to curve parameters.We demonstrate that our rasterizer enables new applications, including a vector graphics editor guided by image metrics, a painterly rendering algorithm that fits vector primitives to an image by minimizing a deep perceptual loss function, new vector graphics editing algorithms that exploit well-known image processing methods such as seam carving, and deep generative models that generate vector content from raster-only supervision under a VAE or GAN training objective.
References:
1. Adobe Inc. 2006. PDF Reference, six edition. https://www.adobe.com/content/dam/acom/en/devnet/pdf/pdf_reference_archive/pdf_reference_1-7.pdf.Google Scholar
2. Alessandro Artusi, Francesco Banterle, Alejandro Moreo, and Fabio Carrara. 2019. Efficient Evaluation of Image Quality via Deep-Learning Approximation of Perceptual Metrics. IEEE Trans. Image Process. 29 (2019), 1843–1855.Google ScholarDigital Library
3. Shai Avidan and Ariel Shamir. 2007. Seam Carving for Content-Aware Image Resizing. ACM Trans. Graph. (Proc. SIGGRAPH) 26, 3 (2007), 10.Google ScholarDigital Library
4. Samaneh Azadi, Matthew Fisher, Vladimir G Kim, Zhaowen Wang, Eli Shechtman, and Trevor Darrell. 2018. Multi-content GAN for few-shot font style transfer. In Computer Vision and Pattern Recognition. 7564–7573.Google Scholar
5. Dejan Azinović, Tzu-Mao Li, Anton Kaplanyan, and Matthias Nießner. 2019. Inverse Path Tracing for Joint Material and Lighting Estimation. In Computer Vision and Pattern Recognition.Google Scholar
6. Francesco Banterle, Massimiliano Corsini, Paolo Cignoni, and Roberto Scopigno. 2012. A low-memory, straightforward and fast bilateral filter through subsampling in spatial domain. In Comput. Graph. Forum, Vol. 31. Wiley Online Library, 19–32.Google Scholar
7. Vineet Batra, Mark J. Kilgard, Harish Kumar, and Tristan Lorach. 2015. Accelerating Vector Graphics Rendering Using the Graphics Hardware Pipeline. ACM Trans. Graph. (Proc. SIGGRAPH) 34, 4 (2015), 146:1–146:15.Google ScholarDigital Library
8. Bradley M Bell and James V Burke. 2008. Algorithmic differentiation of implicit functions and optimal values. In Advances in Automatic Differentiation. Springer.Google Scholar
9. Gilbert Louis Bernstein and Wilmot Li. 2015. Lillicon: Using Transient Widgets to Create Scale Variations of Icons. ACM Trans. Graph. (Proc. SIGGRAPH) 34, 4 (2015).Google ScholarDigital Library
10. John C Bowers, Jonathan Leahey, and Rui Wang. 2011. A ray tracing approach to diffusion curves. 30, 4 (2011), 1345–1352.Google Scholar
11. Edwin Catmull. 1978. A Hidden-Surface Algorithm with Anti-Aliasing. Comput. Graph. (Proc. SIGGRAPH) 12, 3 (1978), 6–11.Google ScholarDigital Library
12. Ravi Chugh, Brian Hempel, Mitchell Spradlin, and Jacob Albers. 2016. Programmatic and direct manipulation, together at last. 51, 6 (2016), 341–354.Google Scholar
13. Adrian Colomitchi. 2006. Approximating cubic Bezier curves by quadratic ones. http://caffeineowl.com/graphics/2d/vectorial/cubic2quad01.html#pseudoQuadDist. Accessed: 2020-05-21.Google Scholar
14. Robert L. Cook. 1986. Stochastic Sampling in Computer Graphics. ACM Trans. Graph. 5, 1 (1986), 51–72.Google ScholarDigital Library
15. Joe Cridge. 2015. Approximating Arcs Using Cubic Bézier Curves. https://www.joecridge.me/content/pdf/bezier-arcs.pdfGoogle Scholar
16. Martin de La Gorce, David J Fleet, and Nikos Paragios. 2011. Model-based 3D hand pose estimation from monocular video. IEEE Trans. Pattern Anal. Mach. Intell. 33, 9 (2011), 1793–1805.Google ScholarDigital Library
17. Mark A. Z. Dippé and Erling Henry Wold. 1985. Antialiasing through Stochastic Sampling. Comput. Graph. (Proc. SIGGRAPH) 19, 3 (1985), 69–78.Google ScholarDigital Library
18. Tom Duff. 1989. Polygon scan conversion by exact convolution.. In International Conference On Raster Imaging and Digital Typography. 154–168.Google Scholar
19. David Eberly. 2011. Distance from a point to an ellipse, an ellipsoid, or a hyperellipsoid. Geometric Tools, LLC (2011).Google Scholar
20. Kevin Ellis, Daniel Ritchie, Armando Solar-Lezama, and Joshua B. Tenenbaum. 2018. Learning to Infer Graphics Programs from Hand-drawn Images. In Advances in Neural Information Processing Systems. Curran Associates Inc., 6062–6071.Google Scholar
21. A. E. Fabris and A. R. Forrest. 1997. Antialiasing of Curves by Discrete Pre-Filtering. In SIGGRAPH. ACM, 317–326.Google Scholar
22. Rida T Farouki and C Andrew Neff. 1990. Algebraic properties of plane offset curves. Computer Aided Geometric Design 7, 1–4 (1990), 101–127.Google Scholar
23. Harley Flanders. 1973. Differentiation under the integral sign. The American Mathematical Monthly 80, 6 (1973), 615–627.Google ScholarCross Ref
24. Francisco Ganacim, Rodolfo S. Lima, Luiz Henrique de Figueiredo, and Diego Nehab. 2014. Massively-parallel Vector Graphics. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 33, 6 (2014), 229:1–229:14.Google Scholar
25. Yaroslav Ganin, Tejas Kulkarni, Igor Babuschkin, S. M. Ali Eslami, and Oriol Vinyals. 2018. Synthesizing Programs for Images using Reinforced Adversarial Learning. In International Conference on Machine Learning. 1666–1675.Google Scholar
26. Leon A Gatys, Alexander S Ecker, and Matthias Bethge. 2016. Image style transfer using convolutional neural networks. In Computer Vision and Pattern Recognition. IEEE, 2414–2423.Google Scholar
27. Michael Gleicher. 1992. Briar: A Constraint-Based Drawing Program. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM.Google ScholarDigital Library
28. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in Neural Information Processing Systems. 2672–2680.Google Scholar
29. Andreas Griewank and Andrea Walther. 2008. Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation (second ed.). Society for Industrial and Applied Mathematics, Philadelphia, PA, USA.Google ScholarDigital Library
30. Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. 2017. Improved training of wasserstein gans. In Advances in Neural Information Processing Systems. 5767–5777.Google Scholar
31. Satish Gupta and Robert F. Sproull. 1981. Filtering Edges for Gray-Scale Displays. Comput. Graph. (Proc. SIGGRAPH) 15, 3 (1981), 1–5.Google ScholarDigital Library
32. David Ha and Douglas Eck. 2018. A Neural Representation of Sketch Drawings. In International Conference on Learning Representations.Google Scholar
33. Aaron Hertzmann. 1998. Painterly rendering with curved brush strokes of multiple sizes. In SIGGRAPH. ACM, 453–460.Google Scholar
34. Aaron Hertzmann. 2003. A survey of stroke-based rendering. IEEE Comput. Graph. Appl. 4 (2003), 70–81.Google ScholarDigital Library
35. Zhewei Huang, Wen Heng, and Shuchang Zhou. 2019. Learning to paint with modelbased deep reinforcement learning. In International Conference on Computer Vision. 8709–8718.Google Scholar
36. Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2017. Image-to-image translation with conditional adversarial networks. In Computer Vision and Pattern Recognition. 1125–1134.Google Scholar
37. Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. 2018. Neural 3D Mesh Renderer. In Computer Vision and Pattern Recognition. IEEE, 3907–3916.Google Scholar
38. Mark J. Kilgard. 2020. Polar Stroking: New Theory and Methods for Stroking Paths. ACM Trans. Graph. (Proc. SIGGRAPH) 39, 4 (2020).Google ScholarDigital Library
39. Mark J. Kilgard and Jeff Bolz. 2012. GPU-accelerated Path Rendering. ACM Trans. Graph. (Proc. SIGGRAPH) 31, 6 (2012), 172:1–172:10.Google ScholarDigital Library
40. Diederick P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations.Google Scholar
41. Diederik P Kingma and Max Welling. 2014. Auto-encoding variational Bayes. In International Conference on Learning Representations.Google Scholar
42. Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. 2017. Self-normalizing neural networks. In Advances in Neural Information Processing Systems. 971–980.Google Scholar
43. Yoshiyuki Kokojima, Kaoru Sugita, Takahiro Saito, and Takashi Takemoto. 2006. Resolution Independent Rendering of Deformable Vector Objects Using Graphics Hardware. In ACM SIGGRAPH 2006 Sketches. ACM, 118.Google Scholar
44. Christian Lauterbach, Michael Garland, Shubhabrata Sengupta, David Luebke, and Dinesh Manocha. 2009. Fast BVH construction on GPUs. Comput. Graph. Forum (Proc. Eurographics) 28, 2 (2009), 375–384.Google ScholarCross Ref
45. Gregory Lecot and Bruno Levy. 2006. Ardeco: Automatic Region DEtection and COnversion. (2006).Google Scholar
46. Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-based learning applied to document recognition. Proc. IEEE 86, 11 (1998), 2278–2324.Google ScholarCross Ref
47. Raph Levien. 2020. Fast 2D rendering on GPU. https://raphlinus.github.io/rust/graphics/gpu/2020/06/13/fast-2d-rendering.html Accessed: 2020-08-24.Google Scholar
48. Rui Li, Qiming Hou, and Kun Zhou. 2016. Efficient GPU Path Rendering Using Scanline Rasterization. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 35, 6 (2016), 228:1–228:12.Google Scholar
49. Tzu-Mao Li. 2019. Differentiable Visual Computing. Ph.D. Dissertation. Massachusetts Institute of Technology. Advisor(s) Durand, Frédo.Google Scholar
50. Tzu-Mao Li, Miika Aittala, Frédo Durand, and Jaakko Lehtinen. 2018. Differentiable Monte Carlo Ray Tracing through Edge Sampling. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 37, 6 (2018), 222:1–222:11.Google Scholar
51. Hsueh-Ti Derek Liu, Michael Tao, and Alec Jacobson. 2018. Paparazzi: Surface Editing by Way of Multi-view Image Processing. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 37, 6 (2018), 221:1–221:11.Google Scholar
52. Shichen Liu, Tianye Li, Weikai Chen, and Hao Li. 2019. Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning. International Conference on Computer Vision (2019).Google ScholarCross Ref
53. Charles Loop and Jim Blinn. 2005. Resolution independent curve rendering using programmable graphics hardware. ACM Trans. Graph. (Proc. SIGGRAPH) 24, 3 (2005), 1000–1009.Google ScholarDigital Library
54. Matthew M. Loper and Michael J. Black. 2014. OpenDR: An Approximate Differentiable Renderer. In European Conference on Computer Vision, Vol. 8695. ACM, 154–169.Google Scholar
55. Raphael Gontijo Lopes, David Ha, Douglas Eck, and Jonathon Shlens. 2019. A Learned Representation for Scalable Vector Graphics. In International Conference on Computer Vision.Google ScholarCross Ref
56. Josiah Manson and Scott Schaefer. 2013. Analytic rasterization of curves with polynomial filters. Comput. Graph. Forum (Proc. Eurographics) 32, 24 (2013), 499–507.Google ScholarCross Ref
57. Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. 2018. Spectral Normalization for Generative Adversarial Networks. In International Conference on Learning Representations.Google Scholar
58. Reiichiro Nakano. 2019. Neural Painters: A learned differentiable constraint for generating brushstroke paintings. arXiv preprint arXiv:1904.08410 (2019).Google Scholar
59. Diego Nehab. 2020. Converting Stroked Primitives to Filled Primitives. ACM Trans. Graph. (Proc. SIGGRAPH) 39, 4 (2020).Google ScholarDigital Library
60. Diego Nehab and Hugues Hoppe. 2008. Random-access Rendering of General Vector Graphics. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 27, 5 (2008), 135:1–135:10.Google Scholar
61. Alexandrina Orzan, Adrien Bousseau, Holger Winnemöller, Pascal Barla, Joëlle Thollot, and David Salesin. 2008. Diffusion Curves: A Vector Representation for Smooth-shaded Images. ACM Trans. Graph. (Proc. SIGGRAPH) 27, 3 (2008), 92:1–92:8.Google ScholarDigital Library
62. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems. 8024–8035.Google Scholar
63. Thomas Porter and Tom Duff. 1984. Compositing digital images. (1984), 253–259.Google Scholar
64. William H. Press, Saul A. Teukolsky, William T. Vetterling, and Brian P. Flannery. 2007. Numerical Recipes 3rd Edition: The Art of Scientific Computing (3 ed.). Cambridge University Press, USA.Google ScholarDigital Library
65. Osborne Reynolds, Arthur William Brightmore, and William Henry Moorby. 1903. The sub-mechanics of the universe. Vol. 3. University Press.Google Scholar
66. Oswaldo Rio Branco de Oliveira. 2012. The Implicit and the Inverse Function theorems: easy proofs. arXiv preprint arXiv:1212.2066 (2012).Google Scholar
67. Thomas W Sederberg and Geng-Zhe Chang. 1994. Isolator polynomials. In Algebraic Geometry and Its Applications. Springer, 507–512.Google Scholar
68. Peter Selinger. 2003. Potrace: a polygon-based tracing algorithm. http://potrace.sourceforge.net/potrace.pdfGoogle Scholar
69. Jian Sun, Lin Liang, Fang Wen, and Heung-Yeung Shum. 2007. Image Vectorization Using Optimized Gradient Meshes. ACM Trans. Graph. (Proc. SIGGRAPH) 26, 3 (2007).Google ScholarDigital Library
70. Ivan E. Sutherland. 1964. Sketch Pad: a Man-machine Graphical Communication System. In Proceedings of the SHARE Design Automation Workshop. 6.329–6.346.Google ScholarDigital Library
71. K. Turkowski. 1982. Anti-Aliasing through the Use of Coordinate Transformations. ACM Trans. Graph. 1, 3 (1982), 215–234.Google ScholarDigital Library
72. Zhou Wang, Alan C Bovik, Hamid R Sheikh, Eero P Simoncelli, et al. 2004. Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13, 4 (2004), 600–612.Google ScholarDigital Library
73. Tian Xia, Binbin Liao, and Yizhou Yu. 2009. Patch-Based Image Vectorization with Automatic Curvilinear Feature Alignment. ACM Trans. Graph. (Proc. SIGGRAPH Asia) (2009).Google Scholar
74. Guofu Xie, Xin Sun, Xin Tong, and Derek Nowrouzezahrai. 2014. Hierarchical Diffusion Curves for Accurate Automatic Image Vectorization. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 33, 6 (2014), 230:1–230:11.Google Scholar
75. Ming Yang, Hongyang Chao, Chi Zhang, Jun Guo, Lu Yuan, and Jian Sun. 2015. Effective clipart image vectorization through direct optimization of bezigons. IEEE Trans. Vis. Comput. Graph. 22, 2 (2015), 1063–1075.Google ScholarDigital Library
76. Gao Yue, Guo Yuan, Lian Zhouhui, Tang Yingmin, and Xiao Jianguo. 2019. Artistic Glyph Image Synthesis via One-Stage Few-Shot Learning. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 38, 6 (2019).Google Scholar
77. Cheng Zhang, Lifan Wu, Changxi Zheng, Ioannis Gkioulekas, Ravi Ramamoorthi, and Shuang Zhao. 2019. A differential theory of radiative transfer. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 38, 6 (2019), 227.Google Scholar
78. Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. 2018. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In Computer Vision and Pattern Recognition. 586–595.Google Scholar
79. Shuang Zhao, Frédo Durand, and Changxi Zheng. 2018. Inverse diffusion curves using shape optimization. IEEE Trans. Vis. Comput. Graph. 24, 7 (2018), 2153–2166.Google ScholarCross Ref
80. Ningyuan Zheng, Yifan Jiang, and Dingjiang Huang. 2019. StrokeNet: A Neural Painting Environment. In International Conference on Learning Representations.Google Scholar
81. Chenyang Zhu, Kai Xu, Siddhartha Chaudhuri, Renjiao Yi, and Hao Zhang. 2018. SCORES: Shape Composition with Recursive Substructure Priors. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 37, 6 (2018), 14.Google Scholar
