Viewing matrices through SVD

SVD allows any $m$ times $n$ matrix $A$ to be viewed as a projection into a coordinate space, a scaling, and then a multiplication by another set of basis vectors. To demonstrate, consider the following three cases:

$A^T$ as [what?]

$$\begin{array}{rl}{A}^{-1}& =(U\mathrm{\Sigma }{V}^T{)}^{-1}\\ & =(V^T{)}^{-1}{\mathrm{\Sigma }}^{-1}{U}^{-1}\\ & =V{\mathrm{\Sigma }}^{-1}{U}^T\end{array}$$

$A^{-1}$ has the same structure as $A^T$, but with the singular values inverted.

$A^{T}A$ as an input-space→input-space scaling along vectors in [what?].

$A^{T}A$ corresponds to converting an input space vector to the coordinate system of $V$ basic vectors, scaling these $V$ coordinates, then converting back to the input space.

$$\begin{array}{rl}{A}^{T}A& =(U\mathrm{\Sigma }{V}^T{)}^T(U\mathrm{\Sigma }{V}^T)\\ & =V\mathrm{\Sigma }{U}^{T}U\mathrm{\Sigma }{V}^T\\ & =V{\mathrm{\Sigma }}^2{V}^T\end{array}$$

Interestingly, the output space and $U$ are not involved.

$A{A}^T$ as an output-space→output-space scaling of vectors in [what?].

$A{A}^T$ corresponds to converting an output space vector to the coordinate system of $U$ basic vectors, scaling these $U$ coordinates, then converting back to the output space.

$$\begin{array}{rl}A{A}^T& =(U\mathrm{\Sigma }{V}^T)(U\mathrm{\Sigma }{V}^T{)}^T\\ & =U\mathrm{\Sigma }{V}^{T}V\mathrm{\Sigma }{U}^T\\ & =U{\mathrm{\Sigma }}^2{U}^T\end{array}$$

Again, the input space and $V$ are not involved.