Physics 6730 Jackknife Error Estimates

One of the central goals of data analysis is an estimate of the uncertainties in fit parameters. Sometimes standard methods for getting these errors are unavailable or inconvenient. In that case we may resort to a couple of useful statistical tools that have become popular since the advent of fast computers. One is called the ``jackknife'' (because one should always have this tool handy) and the other the ``bootstrap''. Here we describe the jackknife method, which was invented in 1956 by Quenouille and developed further by Tukey in 1957. The more recent bootstrap method, developed by Efron in the late 1970's, is discussed in Numerical Recipes. For a reference that discusses both methods, see M. C. K. Yang and David H. Robinson, Understanding and Learning Statistics by Computer, (World Scientific, Singapore, 1986).

First, by way of motivation, here is an example from theoretical physics. Suppose we want to estimate the mass of an elementary particle as predicted in a numerical simulation. The mass is obtained by fitting an exponential to a simulation data set as follows:

$$y(t) = a \exp(-m t)$$

where the data are given as a table of $y$ values for integer values of $t$, as

$$\{y(0),y(1),y(2),\ldots{},y(L)\}.$$

Actually the simulation spits out a list of such values in one single measurement, runs for a while, and spits out another list, and so on. So our data set looks like

$$\{y_i(0),y_i(1),y_i(2),\ldots{},y_i(L)\},$$

where $i = 1,\ldots{},N$ labels a list of measurements.

We might think all we have to do is to take the raw data and construct means $\bar y(t)$ and standard errors $\sigma(t)$ at each time $t$ and

then do a standard least chi square fit. We would get the best values for the parameters $a$ and $m$ and we would get the errors from the error matrix. But we have a problem. The standard chi square fit assumes that the fluctuations in the data points are statistically independent. It turns out that with the numerical simulations (also often a problem with experimental data as well) the fluctuations in the data are correlated. That is, if $y(0)$ fluctuates upwards, chances are better that $y(1)$ also fluctuates upwards. So we can't use the standard formula for chi square. Now it is possible to modify the formula for chi square to take proper account of the correlations. But the analysis becomes much more involved, so one would like to develop more confidence in the resulting error in the mass parameter. Enter the jackknife. It provides an alternative and reasonably robust method for determining the propagation of error from the data to the parameters.

Starting from a sample of $N$ measurements, the jackknife begins by throwing out the first measurement, leaving a jackknife data set of $N-1$ ``resampled'' values. The statistical analysis is done on the reduced sample, giving a measured value of a parameter, say $m_{J1}$. Then a new resampling is done, this time throwing out the second measurement, and a new measured value of the parameter is obtained, say $m_{J2}$. The process is repeated for each set $i$ in the sample, resulting in a set of parameter values $\{m_{Ji},i=1,\ldots{},N\}$. The standard error is given by the formula

$$\sigma^2_{\rm Jmean} = (N-1)\sum_{i=1}^N (m_{Ji} - m)^2/N$$ (1)

where $m$ is the result of fitting the full sample.

The jackknife method is also capable of giving an estimate of sampling bias. We may have a situation in which a parameter estimate tends to come out on the high side (or low side) of its true value if a data sample is too small. Thus the estimate $m$ derived from a fit to $N$ data points may be higher (or lower) than the true value. When this happens, we might expect that removing a measurement, as we do in the jackknife, would enhance the bias. We measure this effect by comparing the mean of the jackknife values $m_{Ji}$, call it $m_{J.}$ with the result $m$ of fitting the full data set. If there is a difference, we can correct for the bias using

$$\tilde m = m - (N-1)(m_{J.} - m)$$

To see how the jackknife works, let us consider the much simpler problem of computing the mean and standard deviation of the mean of a random sample $\{x_i\}$. The conventional approach gives

$$\begin{eqnarray*} \bar x &=& \sum_{j=1}^N x_i/N \\ \sigma^2_{\rm mean} &=& \sum_{j=1}^N (x_j - \bar x)^2/[N(N-1)] \end{eqnarray*}$$

The jackknife approach computes the jackknife sample means

$$x_{Ji} = \sum_{j \ne i} x_i/(N-1)$$

for $i = 1,\ldots{},N$. Then we compute the jackknife error in the mean, which is given by

$$\begin{eqnarray*} \sigma^2_{\rm Jmean} &=& (N-1)\sum_{i=1}^N (x_{Ji} - \bar x)^2/N \end{eqnarray*}$$

Compare the placement of the factors of $N$ and $N-1$ here with the expression for $\sigma_{\rm mean}$. The reason for the difference is that the jackknife sample means are distributed $N-1$ times closer to the mean than the original values $x_i$, so we need a correction factor of $(N-1)^2$. In fact for this simple example, it is easy to show that

$$x_{Ji} - \bar x = (\bar x - x_i)/(N-1).$$

Consequently we can show trivially that

$$\sigma_{\rm Jmean} = \sigma_{\rm mean}$$

so the jackknife procedure hasn't gained us anything in this simple case. But our example of determining the mass of an elementary particle is not so simple. The error estimate is found from Eq (). This error estimate is not likely to be the same as the error obtained from a full correlated chi square analysis. However, we expect that in the limit of an infinitely large sample, both estimates should agree.

So if we get two error estimates and they don't agree, which should we believe? A conservative approach would take the larger of the two. And we would hope that enlarging the data sample would bring better agreement.