Description

Contrary to Feedforward Neural Networks, Recurrent Neural Networks (RNNs) have an internal state. Given its current state, the RNN's next state can be computed using an update equation. This gives rise to a recurrence relation between states.

Symbols Used:

Derivation

\( \htmlClass{sdt-0000000059}{\mathbf{W}} \) is an \(\htmlClass{sdt-0000000117}{n} \times \htmlClass{sdt-0000000117}{n}\) matrix and \(\htmlClass{sdt-0000000125}{\mathbf{h}}\) is an \(\htmlClass{sdt-0000000117}{n}\)-dimensional vector. This ensures that the matrix product is defined, and that the hidden state retains the same dimensions over time. Often we use \(\tanh\) as the activation function \(\htmlClass{sdt-0000000051}{\sigma}\) to induce a non-linear relationship between states.

Example

1. Suppose we have an initial state \(\htmlClass{sdt-0000000125}{\mathbf{h}}(0)=\begin{bmatrix}-1\\1\end{bmatrix}\), a weight matrix \(\htmlClass{sdt-0000000059}{\mathbf{W}} =\begin{bmatrix}0&1\\1&0\end{bmatrix}\), and the activation function \(\htmlClass{sdt-0000000051}{\sigma} =\tanh\).
2. The matrix product \(\htmlClass{sdt-0000000059}{\mathbf{W}}\htmlClass{sdt-0000000125}{\mathbf{h}}(0)\) can be computed with matrix multiplication\[\htmlClass{sdt-0000000059}{\mathbf{W}} \htmlClass{sdt-0000000125}{\mathbf{h}}(0)=\begin{bmatrix}0&1\\1&0\end{bmatrix}\begin{bmatrix}-1\\1\end{bmatrix}=\begin{bmatrix}1\\-1\end{bmatrix}.\]
3. By applying the activation function \(\htmlClass{sdt-0000000051}{\sigma}\) elementwise to the matrix product, we can compute the next state \[\begin{align*}\htmlClass{sdt-0000000125}{\mathbf{h}}(1)&=\htmlClass{sdt-0000000051}{\sigma}\left(\htmlClass{sdt-0000000059}{\mathbf{W}} \htmlClass{sdt-0000000125}{\mathbf{h}}(0)\right)\\&=\tanh\left(\begin{bmatrix}1\\-1\end{bmatrix}\right)\\&=\begin{bmatrix}\tanh(1)\\\tanh(-1)\end{bmatrix}\\&\approx \begin{bmatrix}0.762\\-0.762\end{bmatrix}.\end{align*}\]