\(
\newcommand{\cat}[1] {\mathrm{#1}}
\newcommand{\catobj}[1] {\operatorname{Obj}(\mathrm{#1})}
\newcommand{\cathom}[1] {\operatorname{Hom}_{\cat{#1}}}
\newcommand{\multiBetaReduction}[0] {\twoheadrightarrow_{\beta}}
\newcommand{\betaReduction}[0] {\rightarrow_{\beta}}
\newcommand{\betaEq}[0] {=_{\beta}}
\newcommand{\string}[1] {\texttt{"}\mathtt{#1}\texttt{"}}
\newcommand{\symbolq}[1] {\texttt{`}\mathtt{#1}\texttt{'}}
\newcommand{\groupMul}[1] { \cdot_{\small{#1}}}
\newcommand{\groupAdd}[1] { +_{\small{#1}}}
\newcommand{\inv}[1] {#1^{-1} }
\newcommand{\bm}[1] { \boldsymbol{#1} }
\require{physics}
\require{ams}
\require{mathtools}
\)
Math and science::INF ML AI
Entropy of an ensemble
The entropy of an ensemble, \( X = (x, A_x, P_x) \), is defined to be the
average Shannon information content over all outcomes:
[\[ H(X) = \quad ? \] ]
Properties of entropy:
- [\( H(X) \geq \; ? \) ] with equality iff [...].
- Entropy is maximized if [something about the outcomes is true].
- The Entropy is less than [some opperation applied to] the number of outcomes.
The last two points can be expressed as:
[\[ H(X) \leq \; ? \text{, with equality iff ? } \]]
