Linear Layer

• A linear layer gives linear transformation on its inputs.
• A linear transformation is a function which maps \(\mathbb{R}^n \rightarrow \mathbb{R}^m\).

\[\left [ \begin{array}{rrr} 1 & 2 & 0 \\ 2 & 1 & 0 \end{array} \right ] \left [ \begin{array}{r} x \\ y \\ z \end{array} \right ] = \left [ \begin{array}{c} x+2y \\ 2x+y \end{array} \right ]\nonumber\]

• This is a linear transformation because the left-side matrix transforms $[x, y, z]^T \in \mathbb{R}^3$ to $\mathbb{R}^2$ .
• All matrix multiplications are linear transformations.
• A linear layer in neural networks does the same thing and then adds a bias vector to the resultant vector.

\[y = Wx + b \tag{1}\]

• Equation (1) is how mathematicians write it, treating vectors as column vectors by default. In neural networks, we also need to consider batches of samples.
• So, $x$ is not a simple column vector, rather a matrix of row vector. Each row is a sample. Hence, $x \in \mathbb{R}^{B \times Dx}$ , where $B$ is the batch size and $Dx$ is the dimension of each input sample.
• The shape of matrix $W \in \mathbb{R}^{Dy \times Dx}$ is determined by the required output dimension and the input dimension.
• The equation then becomes,

\[y = xW^T + b\]