Math and science::INF ML AI
Ax=b, in the steepest descent method
When searching for an \( x \) such that \( Ax = b \), a fundamental equation used in optimization is:
\[
\frac{\partial}{\partial x} \left( \frac{1}{2} x^T A x \right) = A x
\]
Sometimes this appears by defining a function \( f \):
\[
f(x) = \frac{1}{2} x^T A x - b^T x
\]
and stating that the goal is to minimize:
\[
\nabla f(x) = \frac{\partial f}{\partial x}.
\]
When \( \nabla f(x) = 0 \), we must have \( Ax = b \).
Geometric interpretation
\( f(x) \) defines a paraboloid in the same space as \( x \). The minimum of this paraboloid is the solution to \( Ax = b \). The gradient of \( f(x) \) is the direction of steepest ascent. When the gradient is zero, we are at the minimum.