“Demystifying Noise: Role of Randomness in Generative AI” by Singh, Huang, Vandersanden, Öztireli and Mitra
Conference:
Type(s):
Title:
- Demystifying Noise: Role of Randomness in Generative AI
Organizer(s):
Presenter(s)/Author(s):
Abstract:
This course offers a thorough exploration of the role of randomness in generative AI, leveraging foundational knowledge from statistical physics, stochastic differential equations, and computer graphics. By connecting these disciplines, the course aims to provide participants with a deep understanding of how noise impacts generative modeling and introduce state-of-the-art techniques and applications of noise in AI. First, we revisit the mathematical concepts essential for understanding diffusion and the integral role of noise in diffusion-based generative modeling. In the second part of the course, we introduce the various types of noises studied within the computer graphics community and present their impact on rendering, texture synthesis and content creation. In the last part, we will look at how different noise correlations and noise schedulers impact the expressive power of image and video generation models. By the end of the course, participants will gain an in-depth understanding of the mathematical constructs for diffusion models and how noise correlations can play an important role in enhancing the diversity and expressiveness of these models. The audience will also learn to code these noises developed in the graphics literature and their impact on generative modeling. The course is aimed for students, researchers and practitioners, with our instructors bringing insights from both academia and industry. We will release all the jupyter notebooks with demos in public domain after the course.
Additional Information:
-
- Motivation: Importance of (Gausssian) noise in generative AI
- Where it all begins: statistical physics
- Generative models Diffusion-based models Score-based models Flow-based models
- Discrete-time Diffusion models Brownian motion is red noise Langevin Dynamics Early variational models (VAEs, HVAEs and ELBO)
- Continuous-time Diffusion models Connecting score-matching, diffusion and flow-based models
-
- Colored noises From random to correlated noise (white-, blue-, red-, Pink-noise)
- Analysis and synthesis of noise Fourier stastics: power spectrum Spatial satistics: Pair correlation function
- Procedural noise Perlin noise, Worley noise, Anisotropic noise, Wavelet noise, Gabor noise Fractional Brownian noise
- The art of noise Blue noise for offline and real-time path traced rendering Procedural noise for generating terrain, clouds, fire, and other natural phenomena
- Coherent noise for animation rendering
-
- White noise for diffusion models Analyzing the impact of white noise in image editing Role of noise in distillation for fast sampling
- Correlated noise for diffusion models Negative correlations are good for diffusion models Color-of-noise can generate hybrid images Edge-preserving noise helps shape-guided diffusion
- Task-specific training or fine-tuning [Xingchang: Noise-aware image composition] Noise optimization for diffusion models Diffusion models for noise patterns
- Open question: How to tailor procedural noise for generative AI?
-
- Video generation and beyond Coherent noise for spatio-temporal animation Training-free noise rescheduling for video generation Warping noise to preserve temporal correlations Noise optimization for motion generation
- Open question: How to tailor procedural noise for generative AI?
Beginner
Prerequisite: Basic understanding of probability, linear algebra, and machine learning concepts is important to follow the course. While advanced expertise is not required, a general background in AI, graphics, or computational methods will help attendees fully engage with the material, especially the hands-on coding exercises.
Topics: AI Diffusion, Generative AI
List of topics and approximate times:
Part I: Overview and theoretical background (35 mins)
Part II: Introduction to different noise profiles (45 mins)
Part III: Impact of Noise on Image diffusion models (40 mins)
Part IV: Impact of Noise on Video Diffusion Models (30 mins)
Additional Info: This course explores the role of randomness in generative AI, drawing from statistical physics, stochastic differential equations, and computer graphics. It provides a deep understanding of how noise affects generative modeling and introduces advanced techniques and applications in AI.
