\( \newcommand{\matr}[1] {\mathbf{#1}} \newcommand{\vertbar} {\rule[-1ex]{0.5pt}{2.5ex}} \newcommand{\horzbar} {\rule[.5ex]{2.5ex}{0.5pt}} \newcommand{\E} {\mathrm{E}} \)
abstract banner
Show Answer
\( \newcommand{\cat}[1] {\mathrm{#1}} \newcommand{\catobj}[1] {\operatorname{Obj}(\mathrm{#1})} \newcommand{\cathom}[1] {\operatorname{Hom}_{\cat{#1}}} \newcommand{\multiBetaReduction}[0] {\twoheadrightarrow_{\beta}} \newcommand{\betaReduction}[0] {\rightarrow_{\beta}} \newcommand{\betaEq}[0] {=_{\beta}} \newcommand{\string}[1] {\texttt{"}\mathtt{#1}\texttt{"}} \newcommand{\symbolq}[1] {\texttt{`}\mathtt{#1}\texttt{'}} \newcommand{\groupMul}[1] { \cdot_{\small{#1}}} \newcommand{\groupAdd}[1] { +_{\small{#1}}} \newcommand{\inv}[1] {#1^{-1} } \newcommand{\bm}[1] { \boldsymbol{#1} } \newcommand{\qed} { {\scriptstyle \Box} } \require{physics} \require{ams} \require{mathtools} \)
Math and science::INF ML AI

One difference of squares from two. Posteriors with Gaussians.

\( (a-x)^2 + (x-b)^2 \)

This expression appears when dealing with posterior probabilities for the mean parameter of a Gaussian distribution. [Can you remember how?]

Minimum at \( x = \frac{a + b}{2} \)

Let \( a < b \) and define \( d=b - a \). If \( x \) is constrained to be between \( a \) and \( b \), then the expression is maximized when \( x=a \) or \(x=b \). The maximum value is \( d^2 \). If \( x \) is not constrained to \( [a, b] \), then the value of the expression can grow unbounded. The expression is minimized when \( x=\frac{d}{2} \). Let \( h=\frac{d}{2} \) be the half-width of the interval. When \( x \) is at the midpoint, \( x=a + h \), then the expression becomes:

\[ h^2 + h^2 = 2h^2 = \frac{d^2}{2} \]

If you increase \( x \) slightly, \( x \to x + \varepsilon \), then the expression would become:

\[ (h + \varepsilon)^2 + (h - \varepsilon)^2 = 2h^2 + 2\varepsilon^2. \]

The expression increases in value. It was indeed minimized at the midpoint when \( x = a + h \), taking the value \( 2h^2 = \frac{d^2}{2} \). Visualising the larger square \( (a-b)^2\) containing two quarter-sized \( h^2\) squares maybe help intuition. 

From two squares to one square

The expression:

\[ (a-x)^2 + (x-b)^2 \]

can be rewritten as one square involving \( x \):

\[ \begin{aligned} (a-x)^2 + (x-b)^2 &= a^2 + x^2 - 2ax + x^2 + b^2 - 2bx \\ &= 2x^2 - 2x(a + b) + a^2 + b^2 \\ &= 2((x - \frac{a + b}{2})^2 - \frac{(a + b)^2}{4}) + a^2 + b^2 \\ &= 2(x - \frac{a + b}{2})^2 - \frac{(a^2 + 2ab + b^2)}{2} + \frac{2a^2 + 2b^2}{2} \\ &= 2(x - \frac{a + b}{2})^2 + \frac{a^2 + b^2 - 2ab}{2} \\ &= 2(x - \frac{a + b}{2})^2 + \frac{(a - b)^2}{2} \end{aligned} \]

Here again we see that the expression is minimized when \( x = \frac{a + b}{2} \), and the minimum value is \( \frac{(a - b)^2}{2} \).