Cramér-Rao bound
| Calyampudi Radhakrishna Rao (2008), Scholarpedia, 3(8):6533. | doi:10.4249/scholarpedia.6533 | revision #137303 [link to/cite this article] |
Cramér-Rao bound stands for an inequality that is the basis of a method for determining a lower bound to the variance of an estimate of a deterministic parameter.
Contents |
Definition
Let \(f(x, \theta)\) be the probability density at observed data \(x\), where \( \theta = ( \theta_1,\cdots, \theta_p ) \) is an unknown p-vector parameter. The \(p \times p\) information matrix on \(\theta\) is defined by \(I(\theta) = (i_{rs})\) where \(i_{rs} = E\left[\left(\frac{\partial \log f}{\partial \theta_r}\right)\left(\frac{\partial \log f}{\partial \theta_s}\right)\right]\) and \(E\) stands for expectation.
Let \(T(x) = (T_1(x),\cdots, T_p(x))\) be an unbiased estimator of \(\theta = (\theta_1,\cdots,\theta_p)\), and \(C(T) = (j_{rs}), j_{rs} = E(T_r - \theta_r) (T_s - \theta_s)\), is the covariance matrix. The Cramér-Rao bound in a general form is
The <math>p \times p </math> matrix <math> C(T) - I^{-1}</math> is non-negative definite.
In case \(\theta\) is a one dimensional parameter, the bound becomes
- <math e1>
V(T) \ge 1/I </math> where \(I = E\left[\left(\frac{\partial \log f}{ \partial \theta}\right)^2\right]\). More generally, if \(E(T_i) = g_i(\theta)\) and \(G(\theta)\) is the matrix with the \((r,s)\) term as \(\frac{\partial g_r(\theta )} {\partial \theta_s}\) then (Bull. Cal. Math. Soc., 1945)
<math>C(T) - G I^{-1} G' </math> is nonnegative definite
History
The origin of Cramér-Rao Bound (CRB) as reported in the newspaper Times of India, dated December 31, 1988, is as follows: “At the young age of 24, Calyampudi Radhakrishna Rao was giving a course on estimation to Master’s students of Calcutta University, in the year 1943. He proved in his class a result first obtained by R.A. Fisher regarding the lower bound to the asymptotic (in large samples) variance of an estimator. When a student asked, “why don’t you prove it for finite samples?”, Rao went back home, worked all night and next day proved what is now known as Cramér-Rao inequality for finite samples.”
Applications
The CRB is useful in finding whether a given estimator has the minimum variance or how close is it to the best possible one. For instance, if the sample is from a Normal distribution, \(N( \mu, \sigma^2)\), then \[ I = \begin{pmatrix} \frac{n}{\sigma^2} & 0 \\ 0 & \frac{n}{2 \sigma^4} \end{pmatrix} \] and \[ I^{-1} = \begin{pmatrix} \frac{\sigma^2}{n} & 0 \\ 0 & \frac{2 \sigma^4}{n} \end{pmatrix} \]
If \( \overline{x} = \frac{ x_1+x_2+ \ldots +x_n }{ n } \) is an estimator of \(\mu\), then \[ V \left ( \overline{x} \right ) = \frac{\sigma^2}{n} = CRB \]
so that is the best possible estimator of the parameter \(\mu\) in terms of variance.
If \( s^2 =\frac{\left(x_1 - \overline{x}\right)^2 + \ldots + \left(x_n-\overline{x} \right)^2}{n-1} \), is an estimate of \(\sigma^2\), then \[ V \left( s^2 \right) = \frac{2 \sigma^4}{n-1} > \frac{2 \sigma^4}{n} (CRB). \] However, the ratio \(\frac{CRB}{V \left( s^2 \right)} =\frac{n}{n-1}\) which is close to 1 as \(n\) becomes large.
CRB, although originally introduced in estimation theory, has found many applications in statistical inference and other areas. It has been found useful in proving certain propositions in limit theorems, asymptotic inference, decision theory, signal processing and density estimation. Van Trees, in his book "Detection, Estimation and Modulation Theory" (Van Trees, 1968)presented a global CRB
- <math e2>
E(x-E(x|y))^2 \ge \frac{1}{E\left[\left(\frac{\partial}{\partial x}\right) \log p(x,y)\right]^2} </math>
where \(p(x,y)\) is the joint density of \((x,y)\). The above inequality and its generalization to the multivariate case have been used in bounding Bayes risk (Bobrovsky et al., 1987).
An interesting result due to A.J. Stam (Stam, 1959) is the derivation of Weyl-Heisenberg uncertainty principle in physics using a specific version of the CRB. Further applications in physics of CRB and Fisher information as a concept underlying the well known physical theories can be found in the book by B. Roy Frieden, "Physics from Fisher Information"(Frieden, 1998).
Recent developments are Quantum Cramér-Rao Bound in the estimation of manifolds in Quantum Physics, by Brody and Houghston(1998) and the concept of Cramér-Rao Functional based on Cramér-Rao Bound by Mayor Wolf (1990). CRB has been extended to estimation of "manifolds" as "complexified and intrinsic" CRB and used in signal processing.
References
- Bull. Cal. Math. Soc. 37, 81-91, 1945
- Van Trees. Detection, Estimation and Modulation Theory, Part 1. Wiley, 1968
- Bobrovsky, Wolf and Zakai, Ann. Statist. 15, 1421-1438, 1987
- A.J. Stam Information and Control 2, 101-112, 1959
- B. Roy Frieden. Physics from Fisher Information. Cambridge University Press, 1998
- D.C.Brody and L.P.Hughston, Poc. Roy. Soc. (1998), 454, 2445-2475.
- E. Mayor Wolf, Ann. Prob.(1990), 18, 840-850.
- S.T.Smith,IEEE Transactions on Signal Processing(2005), 1597-1609, 1610-1630.
Recommended reading
- T.Soderstrom and P. Stoica, System Identification, Prentice Hall,1988.
- C.R.Rao, Linear Statistical Inference and its Applications, Wiley, 1973.
- A. Bera, ET Interview with C.R Rao, Econometric Theory(2003),19, 329-398.
- M.H.Degroot, A conversation with C.R.Rao, Statistical Science(1987),53-67
External links
- Calyampudi R. Rao's web page
- The C.R. Rao Advanced Institute of Mathematics, Statistics and Computer Science
See also
Rao-Blackwell theorem, Fisher-Rao theorem, Second order efficiency
