Generative adversarial networks

Generative Adversarial Networks (GANs) are deep networks introduced by Ian Goodfellow and his colleagues in 2014. They comprise two parts: a generator and a discriminator. Generator produces data that is similar to the existing real data. The discriminator is a binary classifier that tries to detect whether the input is from the real dataset or fake (produced by the generator). The generator and the discriminator are trained simultaneously, such that generator learns to fool the discriminator, and discriminator learns to distinguish between real and fake samples, creating this adversarial setting.

In this topic, we will look at the theoretical foundations of Generative Adversarial Networks (GANs).

The high-level overview

As we mentioned in the introduction, GAN consists of two parts: the discriminator and the generator. The two models are trained simultaneously.

A generator, denoted by $G$, captures the distribution of the training data (the training data is denoted by $x$). A discriminator model, denoted by $D$, estimates the probability that the given sample came from the training data, rather than from $G$. We can present a GAN on an abstract level as follows:

The generator ($G$) receives a random noise vector as input (denoted by $z$ in the illustration below). As $z$ is passed to $G$, $G(z)$ produces an output that looks noisy at the beginning of the training (but hopefully, becomes to resemble the training data more and more as the training progresses). It's important to note that there is no direct connection between the training data and the generator (that is to say, the generator is not trained on the data). Instead, the discriminator ($D$) trains the generator. The generator's objective is producing inputs that make the discriminator mistake it for real data. In turn, the discriminator's objective is to maximize the accuracy of classifying the real samples as real. In the next section, we will define this setting more formally.

GANs transforms the unsupervised problem into a supervised one, because the discriminator is a binary classifier that distinguishes between the real (coming from the dataset) and the fake (being the generator's output) data, and the labels are created during training (we have the information on whether a sample comes from the real or the fake set).

The objective

In this section, we will try to properly decipher a bit intimidating objective for training the GANs:

To start off, the value function for the GAN is given as

where

• $D(x)$ — the probability that the input ($x$) is real;

• $\mathbb{E}_{x \sim p_{\text{data}}(x)}[\log D(x)]$ — the average of discriminator's predictions when the input is from the real data;

• $\mathbb{E}_{z \sim p_{z}(x)}[\log (1 - D(G(z))]$ — the average of of discriminator's predictions the when the inputs are from the noise vector $z$;

• $D(G(z))$ — the output of the discriminator when the data is from the generator (i.e., fake).

The first term can be interpreted as the discriminator's prediction on the real data, and second as the discriminator's prediction on the fake data.

The value function can be interpreted as a payoff. We want to maximize it's value w.r.t. the discriminator, $D$, because the discriminator wants $D(x)$ to be a large number (because it represents high confidence that the real sample is actually real). At the same time, the generator wants to keep $D(x)$ as small as possible (minimize), so that it can successfully fool the discriminator. At the same time, the discriminator wants to minimize $D(G(z))$ (the probability that a fake sample belongs to the real dataset), and generator wants to maximize it.

There is a $1 - D(G(z))$ in the second term, because it aligns the optimization objectives for the discriminator and the generator. Recall that discriminator wants to maximize the first term and minimize the $D(g(z))$, so, by making the second term into $1-D(g(z))$, the discriminator now wants both terms to be maximized. Same argument applies to the generator, that wants to minimize both the $D(x)$ and the $1-D(g(z))$. And we have arrived at the objective, presented in the beginning of this section:

Stepping into the training process

After grasping what the objective for the GAN is, let's see how the training actually takes place. A training step of a GAN with this objective consists of two optimization steps:

1. Maximizing the payoff for the discriminator, $D$ (ascending the gradient);

2. Minimizing the payoff for the generator, $G$ (descending the gradient).

These two steps are alternated during training: the parameters of one network are frozen, and the weights of the other one are optimized, and then the second network is frozen, and the weights of the first one are optimized. This is repeated for each training iteration.

Assuming the generator is frozen, the discriminator is optimized. Both terms in this case contribute, the first term corresponds to the loss associated with real inputs, and the second term is associated with the fake samples. Thus, the objective is to maximize the value function, which is aimed at making the discriminator better at classifying the real samples as real and fake samples as fake.

If the discriminator is frozen, only the second term of the value function ($\mathbb{E}_{z \sim p_{z}(x)}[\log (1 - D(G(z))]$) contributes to the gradients of the generator (since the first term does not interact with the generator in this case). In this case, the objective is to minimize the value function (specifically, the second term).

The training procedure can be summarized as follows:

The training is done akin to how other networks are trained, but in a single epoch, the discriminator and the generator are alternated, as previously described.

Once the training has been completed, the discriminator part is discarded, and the random noise vector is passed to the generator to produce an output (e.g., an image).

A few considerations for GANs

Up until this point, we have ignored the details on what the discriminator and the generator actually are in terms of their architecture. The vanilla GAN typically uses the following settings:

It's just two fully connected networks. If the architecture is capable of classifying (for the discriminator) or turning a vector into a tensor for the generator (of predefined dimensions, e.g., for the MNIST dataset, one would need $28 \times 28 \times 1$ output, because each image in MNIST is 28 by 28 pixels, with a single channel), it can be used as a respective part of the GAN. For example, CNNs and transformers have been used as the components.

Next, let's look at a few limitations that GANs experience:

• Vanishing gradients. This might occur when the discriminator becomes too strong.

• Failure to converge. Occurs when the generator becomes too strong, thus, the discriminator is giving random feedback, and the generator starts to train on the meaningless feedback.

• Mode collapse occurs because the generator is trying to find the output that is the most plausible for the discriminator, and might end up producing the same output.

Conclusion

As a result, you are now familiar with the general layout of GANs, the training procedure, the intuition behind the objective function, and some of the limitations.