“Example-Based Sampling with Diffusion Models” by Bonneel, Coeurjolly, Digne, Doignies, Paulin, et al. …
Conference:
Type(s):
Title:
- Example-Based Sampling with Diffusion Models
Session/Category Title: Magic Diffusion Model
Presenter(s)/Author(s):
Abstract:
Much effort has been put into developing samplers with specific properties, such as producing blue noise, low-discrepancy, lattice or Poisson disk samples. These samplers can be slow if they rely on optimization processes, may rely on a wide range of numerical methods, are not always differentiable. The success of recent diffusion models for image generation suggests that these models could be appropriate for learning how to generate point sets from examples. However, their convolutional nature makes these methods impractical for dealing with scattered data such as point sets. We propose a generic way to produce 2-d point sets imitating existing samplers from observed point sets using a diffusion model. We address the problem of convolutional layers by leveraging neighborhood information from an optimal transport matching to a uniform grid, that allows us to benefit from fast convolutions on grids, and to support the example-based learning of non-uniform sampling patterns. We demonstrate how the differentiability of our approach can be used to optimize point sets to enforce properties.
References:
[1]
Abdalla G. M. Ahmed, Till Niese, Hui Huang, and Oliver Deussen. 2017. An Adaptive Point Sampler on a Regular Lattice. ACM Trans. Graph. 36, 4 (July 2017), 138:1–138:13. https://doi.org/10.1145/3072959.3073588
[2]
Abdalla G. M. Ahmed, Hélène Perrier, David Coeurjolly, Victor Ostromoukhov, Jianwei Guo, Dong-Ming Yan, Hui Huang, and Oliver Deussen. 2016. Low-Discrepancy Blue Noise Sampling. ACM Trans. on Graphics (SIGGRAPH Asia) 35, 6 (2016), 247:1–247:13. https://doi.org/f9cpt2
[3]
Abdalla G. M. Ahmed, Jing Ren, and Peter Wonka. 2022. Gaussian Blue Noise. ACM Trans. Graph. 41, 6, Article 260 (nov 2022), 15 pages. https://doi.org/jtp8
[4]
Abdalla G. M. Ahmed and Peter Wonka. 2021. Optimizing Dyadic Nets. ACM Trans. on Graphics (SIGGRAPH) 40, 4 (2021), 141:1–141:17. https://doi.org/hn22
[5]
Michael Balzer, Thomas Schlömer, and Oliver Deussen. 2009. Capacity-Constrained Point Distributions: A Variant of Lloyd’s Method. ACM Trans. Graph. 28, 3, Article 86 (jul 2009), 8 pages. https://doi.org/10.1145/1531326.1531392
[6]
Nicolas Bonneel, Michiel Van De Panne, Sylvain Paris, and Wolfgang Heidrich. 2011. Displacement interpolation using Lagrangian mass transport. In Proceedings of the 2011 SIGGRAPH Asia conference. 1–12. https://doi.org/gkcqgt
[7]
Robert Bridson. 2007. Fast Poisson disk sampling in arbitrary dimensions.SIGGRAPH sketches 10, 1 (2007), 1. https://doi.org/gf8tsr
[8]
R. Dennis Cook. 1986. Assessment of Local Influence. Journal of the Royal Statistical Society: Series B (Methodological) 48, 2 (1986), 133–155. https://doi.org/10.1111/j.2517-6161.1986.tb01398.x arXiv:https://rss.onlinelibrary.wiley.com/doi/pdf/10.1111/j.2517-6161.1986.tb01398.x
[9]
Fernando De Goes, Katherine Breeden, Victor Ostromoukhov, and Mathieu Desbrun. 2012. Blue noise through optimal transport. ACM Trans. Graph. 31, 6 (2012), 171:1–171:10. https://doi.org/gbb6n9
[10]
Oliver Deussen, Stefan Hiller, Cornelius Overveld, and Thomas Strothotte. 2000. Floating Points: A Method for Computing Stipple Drawings. Computer Graphics Forum (EG’00) 19, 3 (2000), 40–51. https://doi.org/fg9w98
[11]
Mark A. Z. Dippé and Erling Henry Wold. 1985. Antialiasing through Stochastic Sampling. In Proceedings of the 12th Annual Conference on Computer Graphics and Interactive Techniques(SIGGRAPH ’85). Association for Computing Machinery, New York, NY, USA, 69–78. https://doi.org/10.1145/325334.325182
[12]
Daniel Dunbar and Greg Humphreys. 2006. A spatial data structure for fast Poisson-disk sample generation. ACM Transactions on Graphics (TOG) 25, 3 (2006), 503–508.
[13]
Raanan Fattal. 2011. Blue-Noise Point Sampling Using Kernel Density Model. ACM Trans. Graph. 30 (2011), 48:1–48:12. https://doi.org/cv7pbv
[14]
Manuel N Gamito and Steve C Maddock. 2009. Accurate multidimensional Poisson-disk sampling. ACM Trans. Graph. 29, 1 (2009), 8:1–8:19. https://doi.org/dr8646
[15]
Fabian Groh, Patrick Wieschollek, and Hendrik PA Lensch. 2019. Flex-Convolution: Million-scale point-cloud learning beyond grid-worlds. In Computer Vision–ACCV 2018: 14th Asian Conference on Computer Vision, Perth, Australia, December 2–6, 2018, Revised Selected Papers, Part I 14. Springer, 105–122.
[16]
Stefan Heinrich. 1996. Efficient algorithms for computing the L2-discrepancy. Math. Comp. 65, 216 (1996), 1621–1633.
[17]
Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems 33 (2020), 6840–6851.
[18]
Binh-Son Hua, Minh-Khoi Tran, and Sai-Kit Yeung. 2018. Pointwise convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. 984–993.
[19]
Xingchang Huang, Pooran Memari, Hans-Peter Seidel, and Gurprit Singh. 2022. Point-Pattern Synthesis using Gabor and Random Filters. In Computer Graphics Forum, Vol. 41. Wiley Online Library, 169–179.
[20]
Wenzel Jakob, Sébastien Speierer, Nicolas Roussel, Merlin Nimier-David, Delio Vicini, Tizian Zeltner, Baptiste Nicolet, Miguel Crespo, Vincent Leroy, and Ziyi Zhang. 2022b. Mitsuba 3 renderer. https://mitsuba-renderer.org
[21]
Wenzel Jakob, Sébastien Speierer, Nicolas Roussel, and Delio Vicini. 2022a. Dr.Jit: A Just-In-Time Compiler for Differentiable Rendering. ACM Trans. on Graphics (SIGGRAPH) 41, 4 (2022), 124:1–124:19. https://doi.org/gqjn7p
[22]
Alexander Keller. 2004. Stratification by rank-1 lattices. In Monte Carlo and Quasi-Monte Carlo Methods 2002, Harald Niederreiter (Ed.). Springer, 299–313. https://doi.org/fks8z8
[23]
Johannes Kopf, Daniel Cohen-Or, Oliver Deussen, and Dani Lischinski. 2006. Recursive Wang Tiles for Real-Time Blue Noise. ACM Trans. Graph. 25, 3 (2006), 509–518. https://doi.org/dgvw52
[24]
Pierre L’Ecuyer and David Munger. 2016. LatticeBuilder: A General Software Tool for Constructing Rank-1 Lattice Rules. ACM Transactions on Mathematical Software 42 (2016), 1–30. https://doi.org/10.1145/2754929
[25]
Thomas Leimkühler, Gurprit Singh, Karol Myszkowski, Hans-Peter Seidel, and Tobias Ritschel. 2019. Deep point correlation design. ACM Trans. on Graphics (SIGGRAPH Asia) 38, 6 (2019), 1–17. https://doi.org/ggfg2x
[26]
Christiane Lemieux. 2009. Monte Carlo and Quasi-Monte Carlo Sampling. Springer. https://doi.org/b8r4z5
[27]
Shitong Luo and Wei Hu. 2021. Diffusion probabilistic models for 3d point cloud generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2837–2845.
[28]
Quentin Mérigot. 2011. A multiscale approach to optimal transport. In Computer Graphics Forum, Vol. 30. Wiley Online Library, 1583–1592. https://doi.org/cjh4q8
[29]
Thomas Müller, Brian McWilliams, Fabrice Rousselle, Markus Gross, and Jan Novák. 2019. Neural importance sampling. ACM Trans. on Graphics 38, 5 (2019), 1–19. https://doi.org/jtrf
[30]
Thomas Müller, Fabrice Rousselle, Alexander Keller, and Jan Novák. 2020. Neural control variates. ACM Transactions on Graphics (TOG) 39, 6 (2020), 1–19. https://doi.org/jtrj
[31]
Georges Nader and Gael Guennebaud. 2018. Instant transport maps on 2D grids. ACM Trans. Graph. 37, 6 (2018), 249:1–249:13. https://doi.org/jtrg
[32]
Harald Niederreiter. 1992. Random Number Generation and Quasi-Monte Carlo Methods. Society for Industrial and Applied Mathematics (SIAM), Philadelphia, PA, USA. https://doi.org/fd5fjw
[33]
Victor Ostromoukhov. 2007. Sampling with polyominoes. In ACM Trans. Graph., Vol. 26. ACM, 78.
[34]
Victor Ostromoukhov, Charles Donohue, and Pierre-Marc Jodoin. 2004. Fast Hierarchical Importance Sampling with Blue Noise Properties. ACM Trans. on Graphics (SIGGRAPH) 23, 3 (2004), 488–495.
[35]
Art B. Owen. 1998. Scrambling Sobol’ and Niederreiter–Xing Points. Journal of Complexity 14, 4 (1998), 466–489.
[36]
Cengiz Öztireli and Markus Gross. 2012. Analysis and synthesis of point distributions based on pair correlation. ACM Transactions on Graphics (TOG) 31, 6 (2012), 1–10. https://doi.org/gbb6qr
[37]
Loïs Paulin, Nicolas Bonneel, David Coeurjolly, Jean-Claude Iehl, Alexander Keller, and Victor Ostromoukhov. 2022. MatBuilder: Mastering Sampling Uniformity over Projections. ACM Trans. on Graphics (SIGGRAPH) 41, 4 (2022), 84:1–84:13. https://github.com/loispaulin/matbuilder
[38]
Loïs Paulin, Nicolas Bonneel, David Coeurjolly, Jean-Claude Iehl, Antoine Webanck, Mathieu Desbrun, and Victor Ostromoukhov. 2020. Sliced optimal transport sampling. ACM Trans. on Graphics (SIGGRAPH) 39, 4 (2020), 99:1–99:17. https://doi.org/gg8xfj
[39]
Hélène Perrier, David Coeurjolly, Feng Xie, Matt Pharr, Pat Hanrahan, and Victor Ostromoukhov. 2018. Sequences with Low-Discrepancy Blue-Noise 2-D Projections. 37, 2 (2018), 339–353. https://doi.org/gd2j2d
[40]
Matt Pharr, Wenzel Jakob, and Greg Humphreys. 2016. Physically Based Rendering: From Theory to Implementation (3 ed.). Morgan-Kaufmann.
[41]
Adrien Pilleboue, Gurprit Singh, David Coeurjolly, Michael Kazhdan, and Victor Ostromoukhov. 2015. Variance Analysis for Monte Carlo Integration. ACM Trans. Graph. 34, 4 (2015), 124:1–124:14. https://doi.org/f7m28c
[42]
Hongxing Qin, Yi Chen, Jinlong He, and Baoquan Chen. 2017. Wasserstein Blue Noise Sampling. ACM Trans. Graph. 36, 4, Article 137a (Oct. 2017). https://doi.org/gcj3d3
[43]
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-Resolution Image Synthesis with Latent Diffusion Models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 10684–10695.
[44]
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention. Springer, 234–241. https://doi.org/gcgk7j
[45]
Riccardo Roveri, A Cengiz Öztireli, Sebastian Martin, Barbara Solenthaler, and Markus Gross. 2015. Example based repetitive structure synthesis. In Computer Graphics Forum, Vol. 34. Wiley Online Library, 39–52.
[46]
Corentin Salaün, Iliyan Georgiev, Hans-Peter Seidel, and Gurprit Singh. 2022. Scalable Multi-Class Sampling via Filtered Sliced Optimal Transport. ACM Trans. Graph. 41, 6, Article 261 (nov 2022), 14 pages. https://doi.org/10.1145/3550454.3555484
[47]
Adrian Secord. 2002. Weighted Voronoi Stippling. In Proceedings of the 2nd International Symposium on Non-Photorealistic Animation and Rendering (Annecy, France) (NPAR ’02). Association for Computing Machinery, New York, NY, USA, 37–43. https://doi.org/10.1145/508530.508537
[48]
Martin Simonovsky and Nikos Komodakis. 2017. Dynamic edge-conditioned filters in convolutional neural networks on graphs. In Proceedings of the IEEE conference on computer vision and pattern recognition. 3693–3702.
[49]
Gurprit Singh, Cengiz Öztireli, Abdalla G. M. Ahmed, David Coeurjolly, Kartic Subr, Oliver Deussen, Victor Ostromoukhov, Ravi Ramamoorthi, and Wojciech Jarosz. 2019. Analysis of sample correlations for Monte Carlo rendering. In Computer Graphics Forum, Vol. 38. Wiley Online Library, 473–491.
[50]
Ilya M. Sobol’. 1967. On the distribution of points in a cube and the approximate evaluation of integrals. Zhurnal Vychislitel’noi Matematiki i Matematicheskoi Fiziki 7, 4 (1967), 784–802. https://doi.org/crdj6j
[51]
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. 2015. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning. PMLR, 2256–2265.
[52]
Kartic Subr and Jan Kautz. 2013. Fourier analysis of stochastic sampling strategies for assessing bias and variance in integration. ACM Trans. Graph. 32, 4, Article 128 (2013), 12 pages. https://doi.org/gbdg7c
[53]
Peihan Tu, Dani Lischinski, and Hui Huang. 2019. Point Pattern Synthesis via Irregular Convolution. In Computer Graphics Forum, Vol. 38. Wiley Online Library, 109–122.
[54]
Robert Ulichney. 1987. Digital Halftoning. MIT Press, Cambridge, MA, USA.
[55]
Florent Wachtel, Adrien Pilleboue, David Coeurjolly, Katherine Breeden, Gurprit Singh, Gaël Cathelin, Fernando De Goes, Mathieu Desbrun, and Victor Ostromoukhov. 2014. Fast tile-based adaptive sampling with user-specified Fourier spectra. ACM Trans. on Graphics (SIGGRAPH) 33, 4 (2014), 1–11. https://doi.org/f6cz6k
[56]
Li-Yi Wei. 2008. Parallel Poisson disk sampling. In ACM Trans. Graph., Vol. 27. ACM, 20. https://doi.org/cs3jjv
[57]
John I. Yellott. 1982. Spectral analysis of spatial sampling by photoreceptors: Topological disorder prevents aliasing. Vision Research 22, 9 (1982), 1205 – 1210. https://doi.org/fsgtr4
[58]
Cem Yuksel. 2015. Sample elimination for generating Poisson disk sample sets. In Computer Graphics Forum, Vol. 34. Wiley Online Library, 25–32. https://doi.org/f7k7c7
[59]
Xiaohui Zeng, Arash Vahdat, Francis Williams, Zan Gojcic, Or Litany, Sanja Fidler, and Karsten Kreis. 2022. LION: Latent Point Diffusion Models for 3D Shape Generation. In Advances in Neural Information Processing Systems (NeurIPS).
[60]
Linqi Zhou, Yilun Du, and Jiajun Wu. 2021. 3d shape generation and completion through point-voxel diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 5826–5835.
[61]
Yahan Zhou, Haibin Huang, Li-Yi Wei, and Rui Wang. 2012. Point sampling with general noise spectrum. ACM Trans. Graph. 31, 4 (2012), 76.
