Underfitting and Overfitting in Deep Learning

Not sure if good model… or just overfitting?

Source: Authors own image.

In applied Deep Learning, we very often face the problem of overfitting and underfitting. This is a detailed guide that should answer the questions of what is Overfitting and Underfitting in Deep Learning and how to prevent these phenomena.

In Short: Overfitting means that the neural network performs very well on training data, but fails as soon it sees some new data from the problem domain. Underfitting, on the other hand, means, that the model performs poorly on both datasets.

Table of Content

1. Generalization in Deep Learning
2. Overfitting
3. Underfitting
4. Variance Bias Tradeoff
5. Identifying Overfitting and Underfitting during Training
6. How to avoid Overfitting?
7. How to avoid Underfitting?
8. Take-Home-Message

1. Generalization in Deep Learning

When training a neural network we are optimizing the weights and biases so the network can perform a mathematical mapping of input values to output values, dependent on the given objective.

Besides the given objective, our second aim is to build a model that is capable to generalize well from training data to any data from the problem domain. In other words, after the training is complete we want the neural network to perform on yet-unseen data as well as on the training data. Unfortunately, in reality, it’s easier said than done. Often once a model has trained its performance on yet-unseen data can be much worse although the performance on the training data was good. This phenomenon is called overfitting.

Overfitting means that the neural network models the training data too well. Overfitting suggests that the neural network has a good performance. But it fact the model fails when it faces new and yet unseen data from the problem domain. Overfitting is happening for two main reasons.

• The data samples in the training data have noise and fluctuations.
• The model has very high complexity. Before we go any…