Proof of Stein's example

Stein's example is an important result in decision theory which can be stated as

The ordinary decision rule for estimating the mean of a multivariate Gaussian distribution is inadmissible under mean squared error risk in dimension at least 3.

The following is an outline of its proof.[1] The reader is referred to the main article for more information.

Sketched proof

The risk function of the decision rule ${\displaystyle d(\mathbf {x} )=\mathbf {x} }$ is

${\displaystyle R(\theta ,d)=\operatorname {E} _{\theta }[|\mathbf {\theta -X} |^{2}]}$

${\displaystyle =\int (\mathbf {\theta -x} )^{T}(\mathbf {\theta -x} )\left({\frac {1}{2\pi }}\right)^{n/2}e^{(-1/2)(\mathbf {\theta -x} )^{T}(\mathbf {\theta -x} )}m(dx)}$

${\displaystyle =n.}$

Now consider the decision rule

${\displaystyle d'(\mathbf {x} )=\mathbf {x} -{\frac {\alpha }{|\mathbf {x} |^{2}}}\mathbf {x} }$

where ${\displaystyle \alpha =n-2}$. We will show that ${\displaystyle d'}$ is a better decision rule than ${\displaystyle d}$. The risk function is

${\displaystyle R(\theta ,d')=\operatorname {E} _{\theta }\left[\left|\mathbf {\theta -X} +{\frac {\alpha }{|\mathbf {X} |^{2}}}\mathbf {X} \right|^{2}\right]}$

${\displaystyle =\operatorname {E} _{\theta }\left[|\mathbf {\theta -X} |^{2}+2(\mathbf {\theta -X} )^{T}{\frac {\alpha }{|\mathbf {X} |^{2}}}\mathbf {X} +{\frac {\alpha ^{2}}{|\mathbf {X} |^{4}}}|\mathbf {X} |^{2}\right]}$

${\displaystyle =\operatorname {E} _{\theta }\left[|\mathbf {\theta -X} |^{2}\right]+2\alpha \operatorname {E} _{\theta }\left[{\frac {\mathbf {(\theta -X)^{T}X} }{|\mathbf {X} |^{2}}}\right]+\alpha ^{2}\operatorname {E} _{\theta }\left[{\frac {1}{|\mathbf {X} |^{2}}}\right]}$

— a quadratic in ${\displaystyle \alpha }$. We may simplify the middle term by considering a general "well-behaved" function ${\displaystyle h:\mathbf {x} \mapsto h(\mathbf {x} )\in \mathbb {R} }$ and using integration by parts. For ${\displaystyle 1\leq i\leq n}$, for any continuously differentiable ${\displaystyle h}$ growing sufficiently slowly for large ${\displaystyle x_{i}}$ we have:

${\displaystyle \operatorname {E} _{\theta }[(\theta _{i}-X_{i})h(\mathbf {X} )|X_{j}=x_{j}(j\neq i)]=\int (\theta _{i}-x_{i})h(\mathbf {x} )\left({\frac {1}{2\pi }}\right)^{n/2}e^{-(1/2)\mathbf {(x-\theta )} ^{T}\mathbf {(x-\theta )} }m(dx_{i})}$

${\displaystyle =\left[h(\mathbf {x} )\left({\frac {1}{2\pi }}\right)^{n/2}e^{-(1/2)\mathbf {(x-\theta )} ^{T}\mathbf {(x-\theta )} }\right]_{x_{i}=-\infty }^{\infty }-\int {\frac {\partial h}{\partial x_{i}}}(\mathbf {x} )\left({\frac {1}{2\pi }}\right)^{n/2}e^{-(1/2)\mathbf {(x-\theta )} ^{T}\mathbf {(x-\theta )} }m(dx_{i})}$

${\displaystyle =-\operatorname {E} _{\theta }\left[{\frac {\partial h}{\partial x_{i}}}(\mathbf {X} )|X_{j}=x_{j}(j\neq i)\right].}$

Therefore,

${\displaystyle \operatorname {E} _{\theta }[(\theta _{i}-X_{i})h(\mathbf {X} )]=-\operatorname {E} _{\theta }\left[{\frac {\partial h}{\partial x_{i}}}(\mathbf {X} )\right].}$

(This result is known as Stein's lemma.)

Now, we choose

${\displaystyle h(\mathbf {x} )={\frac {x_{i}}{|\mathbf {x} |^{2}}}.}$

If ${\displaystyle h}$ met the "well-behaved" condition (it doesn't, but this can be remedied—see below), we would have

${\displaystyle {\frac {\partial h}{\partial x_{i}}}={\frac {1}{|\mathbf {x} |^{2}}}-{\frac {2x_{i}^{2}}{|\mathbf {x} |^{4}}}}$

and so

${\displaystyle \operatorname {E} _{\theta }\left[{\frac {\mathbf {(\theta -X)^{T}X} }{|\mathbf {X} |^{2}}}\right]=\sum _{i=1}^{n}\operatorname {E} _{\theta }\left[(\theta _{i}-X_{i}){\frac {X_{i}}{|\mathbf {X} |^{2}}}\right]}$

${\displaystyle =-\sum _{i=1}^{n}\operatorname {E} _{\theta }\left[{\frac {1}{|\mathbf {X} |^{2}}}-{\frac {2X_{i}^{2}}{|\mathbf {X} |^{4}}}\right]}$

${\displaystyle =-(n-2)\operatorname {E} _{\theta }\left[{\frac {1}{|\mathbf {X} |^{2}}}\right].}$

Then returning to the risk function of ${\displaystyle d'}$:

${\displaystyle R(\theta ,d')=n-2\alpha (n-2)\operatorname {E} _{\theta }\left[{\frac {1}{|\mathbf {X} |^{2}}}\right]+\alpha ^{2}\operatorname {E} _{\theta }\left[{\frac {1}{|\mathbf {X} |^{2}}}\right].}$

This quadratic in ${\displaystyle \alpha }$ is minimized at

${\displaystyle \alpha =n-2,}$

giving

${\displaystyle R(\theta ,d')=R(\theta ,d)-(n-2)^{2}\operatorname {E} _{\theta }\left[{\frac {1}{|\mathbf {X} |^{2}}}\right]}$

which of course satisfies

${\displaystyle R(\theta ,d')<R(\theta ,d).}$

making ${\displaystyle d}$ an inadmissible decision rule.

It remains to justify the use of

${\displaystyle h(\mathbf {X} )={\frac {\mathbf {X} }{|\mathbf {X} |^{2}}}.}$

This function is not continuously differentiable, since it is singular at ${\displaystyle \mathbf {x} =0}$. However, the function

${\displaystyle h(\mathbf {X} )={\frac {\mathbf {X} }{\varepsilon +|\mathbf {X} |^{2}}}}$

is continuously differentiable, and after following the algebra through and letting ${\displaystyle \varepsilon \to 0}$, one obtains the same result.

References

1. Samworth, Richard (December 2012). "Stein's Paradox" (PDF). Eureka. 62: 38–41.