5. A geometrical interpretation

Recall that the distance between two vectors is the length of their difference. Therefore $\vert \epsilon \vert$ is the distance from $A$   x to b. Moreover, $E ($x$)= \epsilon _ 1 ^ 2 + \dots + \epsilon _ m ^ 2 = \vert \epsilon \vert ^ 2$. For a nonnegative function, minimizing the square of the function is the same as minimizing the function. Therefore

Moreover, the vectors of the form $A$   x, for all possible column vectors x, form a subspace $W$ of R$^ m$. Therefore the problem is really one of finding the point in a subspace that is closest to a given point; the error vector $\epsilon$ is the vector from the given point to the point in the subspace. In Figure , the parallelogram represents part of the subspace $W$. The error vector ``err'' goes from $b$ to $A$x.

Figure: Geometrical interpretation

book/12dir/geometric.eps

To see why $W$ is a subspace, notice that $W$ is just the image of the linear transformation given by $T($x$) = A$x. Or, notice that $A$   x can be expanded as $A$   x$= A _ 1 x _ 1 + \dots + A _ n x _ n$, where $A _ j$ means the $j$-th column of $A$; therefore $W$ is the column space of $A$, i.e., the subspace spanned by the columns of $A$. In Figure , the columns of $A$ are the sides of the parallelogram that touch the origin.

You might think that the way to get from b to the closest point of $W$ is to go along a normal to $W$--i.e., a line perpendicular to $W$--and that's exactly right, as indicated by the right angle in Figure . First let's discuss the computational method that produces the answer, and then then derivation of the method.