Deep Learning is to approximate the data distribution by learning the best-fit probability distribution from training data using a neural network.
Supervised Learning
What we have is a training set $\mathcal D_{\text{train}} = \{(x_i, y_i)\}_{i=1}^N$ sampled from $\mathcal D$. The empirical risk is defined as
$$R_{\text{emp}}(h) = \frac{1}{N} \sum_{i=1}^n L(h(x_i), y_i),$$where $L$ is the loss function, $h$ is the hypothesis function in the hypothesis space $\mathcal H$. And the expected risk is defined as, can not be computed directly since we do not know the data distribution $\mathcal D$,
$$R(h) = \mathbb E_{(x,y)\sim \mathcal D}[L(h(x), y)].$$A best hypothesis function $h^*$ is wanted to minimize the expected risk, but we can only minimize the empirical risk. The generalization error is defined as
$$E_{\text{gen}}(h) = R(h) - R_{\text{emp}}(h).$$This method is called Empirical Risk Minimization (ERM). The generalization error can be bounded by the VC dimension of the hypothesis space $\mathcal H$ which will be introduced in the future.