Multivariate linear regression

Multivariate linear regression (MLR) is the problem of finding a set of basis vectors $\bhw_{xi}$ and corresponding regressors $\beta_i$ in order to minimize the mean square error of the vector $\by$:

$$\epsilon^2 = E\left[\Vert y_i - \sum_{i=1}^M\beta_i\bhw_{xi}^T {\bf x} \Vert^2 \right]$$ (31)

where $M = \text{dim}(\by)$. The basis vectors are described by the matrix $\bC{xx}^{-1}\bC{xy}$ which is also known as the Wiener filter. A low-rank approximation to this problem can be defined by minimizing

$$\epsilon^2 = E\left[\Vert\by - \sum_{i=1}^N\beta_i\bhw_{xi}^T {\bf x} \bhw_{yi} \Vert^2 \right]$$ (32)

where N<M and the orthogonal basis $\bhw_{yi}$s span the subspace of $\by$which gives the smallest mean square error given the rank N. The bases $\{\bhw_{wi}\}$ and $\{\bhw_{yi}\}$ are given by the solutions to

$$\begin{cases} {\bf C}_{xy}\bhw_y = \beta {\bf C}_{xx} \bhw... ... {\bf C}_{yx}\bhw_x = \frac{\rho^2}{\beta} \bhw_y, \end{cases}$$ (33)

which can be recognized from equation 11 with $\bA$ and $\bB$ from the lower row in table 1.