Texture Synthesis and Transfer

Texture Transfer

The texture transfer algorithm which was implemented was the same algorithm described in the Efros and Freeman paper. The general approach can be seen as texture synthesis combined with choosing image patches with a constraint in matching patches from a guidance image based on some weighted cost.

Instead of the SSD calculation performed in texture synthesis, SSD is now composed of the two previous described terms:

$$SSD = \alpha SSD_{overlap} + (1 - \alpha) SSD_{guidance}$$

Note that the greater the \(\alpha\) value, the more the calculations will steer towards choosing a patch that is similar to texture synthesis. The smaller the \(\alpha\) the more the cost will steer towards the guidance image.

To calculate the \(SSD_{overlap}\), we use the input texture, and image patch to be filled.

To calculate the \(SSD_{guidance}\), we need to first create a correspondence map of the input texture, and the guidance image. The paper descibes as taking the intensity values of these images, and applying a Gaussian blur to them. We use these as inputs to compute the SSD between the input texture and guidance.

There are a few caveats to texture transfer. One must take into consideration that:

• The very first patch placed in the image at (0, 0) must be the patch with the minimal SSD value. That is, we select the patch from the input texture where \(SSD_{overlap} = 0\).
• Unlike texture synthesis, with texture transfer, we always select the minimal value to sample within the texture image. That is equivalently, in synthesis terms, tol = 1.
• Smaller block size and larger proportion of overlap will lead to finer results. This is the basis for iterative texture transfer.

Sketch + Feynman

• patch_size = 9
• overlap = 3
• tol = 1
• alpha = 0.1

Fire + Earth

• patch_size = 9
• overlap = 3
• tol = 1
• alpha = 0.1

Painting + Riley

• patch_size = 9
• overlap = 3
• tol = 1
• alpha = 0.1

Iterative Texture Transfer

The method for iterative texture transfer is similar to a single iteration of texture transfer, except that we introduce a third term into the calculation for the SSD.

Instead of simply computing \(SSD_{overlap}\) with just the input texture and image patch, we must also sum up the previous SSD image in computation.

Therefore, the algorithm must remember the SSD for the particular pixel falling into a patch from the previous iteration. We use this to augment the cost value of the overlap to give more influence in values selected before.

Thus computing the SSD becomes:

$$SSD = \alpha(SSD_{overlap} + SSD_{prev}) + (1 - \alpha)SSD_{guidance}$$

For each iteration \(i\) of \(N\), we must modify the following:

• patch_size
• overlap
• alpha

For each iteration, the patch_size and overlap are reduced by \(\frac{1}{3}\) each time. By default we start with patch size \(31 \times 31\) with an overlap region of \(21\) pixels.

The \(\alpha\) value is initially \(0.1\) and is altered with the following expression at each iteration:

$$\alpha_{new} = 0.8 * \frac{i - 1}{N - 1} + 0.1$$

As the number of iteration increases, the less blocky the result seems to be, and fidelity in parts of image increases.

For the iterative version of the Sketch + Feynman example, we see that by \(N = 5\), the output image looks like a sketch. The initial parameters were:

• patch_size = 31
• overlap = 21
• tol = 1
• alpha = 0.1
• N = 5

For my method the smallest granular patch and overlap size was \(9\) and \(3\) respectively.

Face in Toast

To build a "face in toast" image as seen in internet urban legends, it is a multi-step process.

1. We choose a texture that resembles that of toast.
2. Select the guidance image (the face)
3. Apply texture transfer on the images from (1) and (2)
4. Save this result
5. Select an image of toast (the image for which the face should appear on)
6. Apply Laplacian blending on (5) and (6)

Laplacian blending involves constructing the Laplacian pyramids for both the face and toast image. Before combination, we create a mask which dictates where the face will be placed in the toast image. We then use the mask to create a Gaussian pyramid.

Using the Gaussian pyramid of masks, we perform multiplication on the images at each level of their respective Laplacian pyramids and collapse into a single image. This creates the effect of "face in toast".