Projection Matrices

Let {u1 .... uk}be an orthonormal basis for a subspace W of Rn. Form the n x k matrix

U = [U1 U2 ... Uk]

Then projWv = UU^Tv.

The matrix UU^T is called the projection matrix for the subspace W. It does not depend

on the choice of orthonormal basis.

What if we do not start with an orthonormal basis of W?

Theorem. Let {a1 ... ak}be any basis for a subspace W of Rn. Form the n k matrix
A= [a1 a2 ... ak]

Then the projection matrix for W is A(A^TA)^-1AT .

To see why this formula is true, we need a lemma.

Lemma. Suppose A is an n k matrix whose columns are linearly independent. Then

A^TA is invertible.

To see why this lemma is true, consider the transformation A : ->Rn determined

by A. Since the columns of A are linearly independent, this transformation is one-to-one.

Moreover, the null space of A^T is orthogonal to the column space of A. Consequently, A^T

is one-to-one on the column space of A, and as a result, A^TA : Rk ->Rk is one-to-one. By

the Invertible Matrix Theorem, A^TA is invertible.

Now we can compute the projection matrix for the column space of A. (Note that

W = Col A.) Any element of the column space of the matrix A is a linear combination of

the columns of A, that is,

x1a1 + x2a2 + ... + xkak:

If we let

x= [x1 ... xk ]

then

x1a1 + x2a2 + : : : + xkak = Ax:

Given v in Rn, we denote by xp the x that corresponds to the projection of v onto W. In

other words, let

projWv = Axp:

We find the projection matrix by calculating xp.

The projection of v onto W is characterized by the fact that

v - projWv

is orthogonal to each vector w in W, that is,

w .(v -projWv) = 0

for all w in W. Since w = Ax for some x, we have

Ax . (v - Axp) = 0

for all x in Rk. Writing this dot product in terms of matrices yields

(Ax)^T (v - Axp) = 0;

which is equivalent to

(x^TA^T )(v -Axp) = 0:

Converting back to dot products, we have

x . AT (v -Axp) = 0:

In other words, the vector AT (v - Axp) is orthogonal to every vector x in Rk. The only

vector in Rk with this property is the zero vector, so we may conclude that

AT (v -Axp) = 0:

We get

A^Tv = ATAxp:

From the lemma, we know that ATA is invertible, and we have

(A^TA)^-1ATv = xp:

Since Axp is the desired projection, we have

A(A^TA)^-1ATv = projWv:

We conclude that the projection matrix for W is

A(A^TA)^-1A^T