Physics 6730 Jackknife Error Estimates
One of the central goals of data analysis is an estimate of the uncertainties in fit parameters. Sometimes standard methods for getting these errors are unavailable or inconvenient. In that case we may resort to a couple of useful statistical tools that have become popular since the advent of fast computers. One is called the ``jackknife'' (because one should always have this tool handy) and the other the ``bootstrap''. Here we describe the jackknife method, which was invented in 1956 by Quenouille and developed further by Tukey in 1957. The more recent bootstrap method, developed by Efron in the late 1970's, is discussed in Numerical Recipes. For a reference that discusses both methods, see M. C. K. Yang and David H. Robinson, Understanding and Learning Statistics by Computer, (World Scientific, Singapore, 1986).
First, by way of motivation, here is an example from theoretical
physics. Suppose we want to estimate the mass of an elementary
particle as predicted in a numerical simulation. The mass is obtained
by fitting an exponential to a simulation data set as follows:
We might think all we have to do is to take the raw data and construct means and standard errors at each time and
then do a standard least chi square fit. We would get the best values for the parameters and and we would get the errors from the error matrix. But we have a problem. The standard chi square fit assumes that the fluctuations in the data points are statistically independent. It turns out that with the numerical simulations (also often a problem with experimental data as well) the fluctuations in the data are correlated. That is, if fluctuates upwards, chances are better that also fluctuates upwards. So we can't use the standard formula for chi square. Now it is possible to modify the formula for chi square to take proper account of the correlations. But the analysis becomes much more involved, so one would like to develop more confidence in the resulting error in the mass parameter. Enter the jackknife. It provides an alternative and reasonably robust method for determining the propagation of error from the data to the parameters.
Starting from a sample of measurements, the jackknife begins by
throwing out the first measurement, leaving a jackknife data set of
``resampled'' values. The statistical analysis is done on the
reduced sample, giving a measured value of a parameter, say .
Then a new resampling is done, this time throwing out the second
measurement, and a new measured value of the parameter is obtained,
say . The process is repeated for each set in the sample,
resulting in a set of parameter values
The standard error is given by the formula
The jackknife method is also capable of giving an estimate of sampling
bias. We may have a situation in which a parameter estimate tends to
come out on the high side (or low side) of its true value if a data
sample is too small. Thus the estimate derived from a fit to
data points may be higher (or lower) than the true value. When this
happens, we might expect that removing a measurement, as we do in the
jackknife, would enhance the bias. We measure this effect by
comparing the mean of the jackknife values , call it
with the result of fitting the full data set. If there is a
difference, we can correct for the bias using
To see how the jackknife works, let us consider the much simpler problem of computing the mean and standard deviation of the mean of a random sample . The conventional approach gives
So if we get two error estimates and they don't agree, which should we
believe? A conservative approach would take the larger of the two.
And we would hope that enlarging the data sample would bring better