3. Solution using normal equations

The least-squares solution can be found by a simple method that almost seems like magic before you know its derivation:

This system of equations is usually called the system of normal equations. For short, they can be written as $A^t A$x$= A^t$   b, which you can think of as the result of multiplying both sides of $A$   x$\approx$   b on the left by $A^t$ and putting $=$ for $\approx$. The matrix $N$ will always be symmetric, i.e., $N ^ t = N$.

Example (continued from above).

$N = A ^ t A = \left[\begin{array}{rrrr} 1&2& 1&2\\ 2&1& -2&-1\end{array}\righ... ...-1 \end{array}\right] = \left[\begin{array}{rr} 10&0\\ 0&10 \end{array}\right]$ ,

d$= A^t$   b$= \left[\begin{array}{rrrr} 1&2& 1&2\\ 2&1& -2&-1\end{array}\right] \left[\b... ... 1\\ 2 \end{array}\right] = \left[\begin{array}{r} 14\\ 5 \end{array}\right]$, so the normal equations are $\left[\begin{array}{rr} 10&0\\ 0&10 \end{array}\right]\left[\begin{array}{c}x\\ y\end{array}\right] = \left[\begin{array}{r}14\\ 5\end{array}\right]$.

Thus the solution is $x = 1.4, y = 0.5$, as mentioned above. (As you see, this example was specially chosen for easy arithmetic; in general $N$ is symmetric but not diagonal.)

Notice that $N$ needs to be nonsingular. In practice this is very likely to be the case. The reason why, as well as the derivation of the normal equations will be explained below.