Description

This definition is used to compute how the loss is affected with a change in the potential of a neuron in a given layer. It is used in the backward pass in backpropagation, where the gradient is computed for updating the model parameters. The error term is computed differently, depending on whether the neuron is a hidden neuron or an output neuron.

Symbols Used:

Example

1. Let \(\hat{y}\) denote the model prediction \(\htmlClass{sdt-0000000001}{\mathcal{N}}_{\htmlClass{sdt-0000000083}{\theta}}(\htmlClass{sdt-0000000103}{u})\). Suppose that we have a binary classification task with a single output neuron. We use binary crossentropy loss, which is given by \[\htmlClass{sdt-0000000072}{L}\left(\hat{y},\htmlClass{sdt-0000000037}{y}\right)=−(\htmlClass{sdt-0000000037}{y}\log(\hat{y})+(1−\htmlClass{sdt-0000000037}{y})\log(1−\hat{y}​))\]
2. Suppose that we compute the error term for the output neuron, which is the first neuron in the \(k\)-th layer, assuming that there are \(k\) layers. Then, the potential of the output neuron is equal to the model prediction, so \(\htmlClass{sdt-0000000099}{a}_1^k=\hat{y}\). That is, we are interested in the quantity \[\htmlClass{sdt-0000000075}{\delta}_1^k=\frac{\partial \htmlClass{sdt-0000000072}{L}\left(\hat{y},\htmlClass{sdt-0000000037}{y}\right)}{\partial \htmlClass{sdt-0000000099}{a}_1^k}=-\frac{\partial}{\partial \hat{y}} \left(\htmlClass{sdt-0000000037}{y}\log(\hat{y})+(1−\htmlClass{sdt-0000000037}{y})\log(1−\hat{y}​)\right).\]
3. Applying the sum rule gives \[\htmlClass{sdt-0000000075}{\delta}_1^k=-\left(\frac{\partial}{\partial \hat{y}}\htmlClass{sdt-0000000037}{y}\log(\hat{y})+\frac{\partial}{\partial \hat{y}}(1−\htmlClass{sdt-0000000037}{y})\log(1−\hat{y}​)\right).\]
4. Using the derivative of the logarithm and the chain rule gives \[\htmlClass{sdt-0000000075}{\delta}_1^k=-\left(\frac{\htmlClass{sdt-0000000037}{y}}{\hat{y}}-\frac{1-\htmlClass{sdt-0000000037}{y}}{1-\hat{y}}\right)=\frac{1-\htmlClass{sdt-0000000037}{y}}{1-\hat{y}}-\frac{\htmlClass{sdt-0000000037}{y}}{\hat{y}}.\]
5. Since \(\htmlClass{sdt-0000000037}{y}\in\{0,1\}\), we can also represent \(\htmlClass{sdt-0000000075}{\delta}_1^k\) as a piecewise function: \[\htmlClass{sdt-0000000075}{\delta}_1^k=\begin{cases}-\frac{1}{\hat{y}}&\text{if }\htmlClass{sdt-0000000037}{y}=1\\\frac{1}{1-\hat{y}}&\text{otherwise.}\end{cases}\]