Let's look at the problem of image blending first then work up to diffusion models. Image blending in a nutshell tries to mix two images S1 & S2 to produce an image S3. Like: S3 = S1*M1 + S2*M2 Where M1 & M2 are blending masks.

The masks are doing a weighted averaging of the two images to produce the result. The idea is to mix the source images so there are no discontinuos visible seam between them. And weighted averaging makes transitions between images smooth.

You can extend this to mixing N number of images from a dataset D. Such that S(N+1) = S1*M1 +... + SN*MN. This naive approach will of course produce "ghosting" and weird color mixing etc. So it's not ideal approach.

So to mix images better requires moving from the pixel domain to gradient domain, why? People gradient model the flow of pixels on an image, so if you what a result as a mix of two images, you can shift pixel values of one image guided by the gradients of the other.

That is I can model the mixing as: S_(i+1) <— S_(i) + e * d(S2) Where d() is the gradient operator on image S2, e = small number, the underscore is indexing the time step such that S_{0} = S1, the first image. If i-> infinite and e -> zero, S = will be a mixture of S1 & S2.

The gradient guided mixing doesn't necessarily average the two images but uses the other image to guide where pixels of the final image end up ie it guides the diffusion of pixels resulting in one image morphed like the other without much ghosting.

You can get this with Laplacian blending too ie given an image. S3 = S1 + Laplacian of (S2). The Laplacian is the second order derive ie repeat the gradient operator twice on the image. Mixing Laplacians like this yeild the Laplacian blending algorithm.

So far the intuition here is the gradients of one image guide the pixel of the other image to form a blended version of the the two ie one gradients guide the diffusion of the pixels to form the mixture, the blended version of the two images without ghosting effects.

In diffusion models you want to guide the blending process not by one image but by all possible images in the training set D & you don't want to necessarily re-generate the exact training set D. So what's the intuition?

To ensure diverse mixing you need to introduce randomness in the mixing process. S_{i+1} <— S_{i} + e * d(S2) + e2 * r Here e2 is some small value which tends to zero has i increases & r is random noise to nudge the process into different random directions.

This creates the stochastic gradient blending process lets call it SGB process. But we also don't want to use one source image but all of them. How? We replace S2 with a deep neural network P(S) that models the probability of image S.

This model P(X) can guide the blending process using the score function in place of S as: S_{i+1} = S_{i} + e * d(log P(S_{i}) + e2* r Where delta of log probability is the score function. This models the Langevin dynamics in Physics.

Essentially, the pixels are diffusing as guided by the gradient field of the score function of the model P() that condensed all training examples, the result is that the output is more of a blending of different images in the source D.

Langevin dynamics are essentially modeling random motion of particles, in this case pixels, such that in the limiting case one is effectively sampling from a probability distribution P(). That is, the output is a random interpolation of neighboring images in the space of D.

A diffusion model isn't coming up with new images by intelligence but by sampling interpolated images from the distribution covered by D, it can't extrapolate outside that distribution. The output is effectively a blending of many different images.

This process can be conditioned on text etc to guide it even further to generate particular types of images ie draw X in the style of Y. In that case it will use Y artist pool of images as gradient guides to form a mixture that looks like Y artistic style.

Of course the model doesn't store individual images of artist Y, but compressed versions of them and are used to guide the diffusion process to output what would be a stochastic blend of that artists work.

For more on the formulation of score based diffusion models using Langevin dynamics from physics see this intro video: https://youtu.be/wMmqCMwuM2Q?s...

Related: https://en.wikipedia.org/wiki/...

See also: https://en.wikipedia.org/wiki/...

Stochastic differential equations (SDE): https://en.wikipedia.org/wiki/...

And gradient based image processing: https://en.wikipedia.org/wiki/...

PS: I work with advanced image processing, computational photography and computer vision algorithms. Tweets are too cumbersome to go too deep but diffusion models are a form of gradient based image processing algorithms.

It's all about large scale blending not intelligence.

At the end of the day. The output from a diffusion model is a weighted averaging of all images in D. Whether this constitutes copyright infringement or not is out of scope of this thread. But my opinion is that people who contributed to D must be compensated.

@ChombaBupe This is basically the Chinese room argument, with the philosophical book replaced by a real deep net. Whether this is "intelligent" probably closely tracks your views on that argument. You could generate any image by an appropriate local weighting of a solid background.

@ChombaBupe Thanks for writing all this out, appreciate you 🫡

@EZE3D Thanks.

@ChombaBupe Very nice explanation

@ChombaBupe I'm buckled in

@ChombaBupe Thanks for sharing this

@SprintPlanner You welcome.

@ChombaBupe @threadreaderapp unroll

@ChombaBupe This is like describing human illustrators with the physics of how graphite rubs off on paper

@ChombaBupe stochastic gradient blending? sounds like a fancy way to say pixel soup. let’s see how deep this rabbit hole goes.