Math and science::Algebra

Preloaded dot product

Matrix products \( W^{T}W \) and \( (WW^{T})^{-1} \) can be thought of as preloading a dot product of vectors after both vectors are transformed by \( W \) or \( W^{-1} \).

\( W^{T}W \)

Consider the dot product \( x^{T}y \) of two vectors \( x \) and \( y \). Now consider the transformed vectors \( Wx \) and \( Wy \). The dot product of the transformed vectors is:

\[ \begin{align*} (Wx)^{T}(Wy) &= x^{T}W^{T}Wy &= x^{T}(W^{T}W)y \end{align*} \]

So when looking at a matrix product \( W^{T}W \), know that it will carry out the transformation and then dot product of any two vectors if they are placed on either side of the product.

Special case: length² of transformed vector

Let \( b = Wx \). What is the of \( b \)? Let's make this a function that takes in any vector \( x \) and returns the length-squared of the transformed vector \( Wx \). This function is simply \( f(x) = x^{T}W^{T}Wx \). We can see \( W^{T}W \) as being the implementation of this function.

\( WW^{T} \)

\( W W^{T} \) acts similarly, but with an inverse transformation.Again, consider the dot product \( x^{T}y \) of two vectors \( x \) and \( y \). Now consider the transformed vectors \( W^{-1}x \) and \( W^{-1}y \), where we are now inverting the transformation \( W \). The dot product of the transformed vectors is:

\[ \begin{align*} (W^{-1}x)^{T}(W^{-1}y) &= x^{T}(W^{-1})^{T}W^{-1}y &= x^{T}(WW^{T})^{-1}y \end{align*} \]

So when looking at a matrix product \( WW^{T} \), know that when this matrix is inverted, \( (WW^{T})^{-1} \), it will carry out the inverse transformation and then dot product of any two vectors if they are placed on either side of the product.

\( QQ^{T} = (QQ^{T})^{-1} \) for orthonormal \( Q \)

When \( Q \) is orthonormal, \( Q^{-1} = Q^{T} \), so it's easy to see that:

\[ \begin{align*} (QQ^{T})^{-1} &= (Q^{T})^{-1}Q^{-1} \\ &= QQ^{T} \end{align*} \]

So orthonormal \( QQ^{T} \) could be thought of as preloading the inverse then dot product of two vectors, even without the matrix inverse operation appearing in the formula. However, more pertinently, we have \( QQ^{T} = QQ^{-1} = I \), so \( QQ^{T} \) is the identity matrix. And so we have arrived at the intuitive idea that a rotation transformation, \( Q \), doesn't change the length of a vector or the angle between two vectors.

\( QQ^{T} \) when not full rank (projection)

If \( Q \) has say 2 orthonormal columns in \( \mathbb{R}^3 \), then it doesn't have an inverse, but we can ask for the projection of a vector \( b \) onto the column space of \( Q \) by computing \( QQ^{T}b \). Here, \( Q^{T}b \) projects \( b \) onto the column space of \( Q \), and then \( Q (Q^{T}b) \) rehydrates the projection so that it's expressed in the original space.

Example

Consider a random vector \( X : \mathbb{R}^n \to \mathbb{R}^n \) distributed as a standard normal distribution, \( X \sim N(\mathbf{0}, I) \). We can derive a random variable \( Y = WX \) for some matrix \( W \). Using the change of variable formula for distributions, we can derive the distribution of \( Y \).

If \( X \) and \( Y \) were 1 dimensional random variables, with \( Y = g(X) \), the change of variable formula would be:

\[ \mathbf{P}_Y(t) = \mathbf{P}_X(g^{-1}(t))\left|\frac{d}{dt}g^{-1}(t)\right| \]

The multidimensional version, with \( Y = WX \), of this formula is:

\[ \mathbf{P}_Y(t) = \mathbf{P}_X(W^{-1}t)\left|\det\left(\frac{d}{dt}W^{-1}t\right)\right| \]

and applying this to the standard normal distribution, we get:

\[ \begin{align*} \mathbf{P}_Y(t) &= \mathbf{P}_X(W^{-1}t)\left|\det(W^{-1})\right| \\ &= \mathrm{Constant} \times \exp\left(-\frac{1}{2}(W^{-1}t)^{T}(W^{-1}t)\right) \\ &= \mathrm{Constant} \times \exp\left(-\frac{1}{2}t^{T}(W^{-1})^{T}W^{-1}t\right) \\ &= \mathrm{Constant} \times \exp\left(-\frac{1}{2}t^{T}(WW^{T})^{-1}t\right) \\ &= N(\mathbf{0}, (WW^{T})^{-1}) \\ &= N(\mathbf{0}, S) \end{align*} \]

where \( S \) is the covariance matrix of \( S \) (aka the cross-covariance matrix \( S = \operatorname{Cov}(Y, Y) \).

So while the covariance matrix \( \Sigma \) of \( Y \) is specified to parameterize the multivariate normal distribution, under the hood, this matrix is preloading the dot product of two vectors that are brought back to the original space by inverting the transformation \( W \).

Note: the fact that \( WW^{T} \) is the covariance matrix of \( Y \) is another separate result that is a special case of the more general theorem:

\[ \operatorname{Cov}(WZ) = W\operatorname{Cov}(Z)W^{T} \]

for which there is another page.