Bayes Theorem and Maximum Likelihood

So far we have been thinking of the probability for getting a result $k$ if we know that the mean value should be $\bar k$. Now suppose we make a measurement and get $k$ counts, but we don't know anything about $\bar k$, except that it must be nonnegative, of course. We may turn the question around and ask what is the most likely value for $\bar k$, given the result of our measurement. To make this turned-around idea more concrete, we use the concept of conditional probability. We say that the Poisson distribution $P(k,\bar k)$ tells us the probability that we get $k$, on the condition that the mean value is $\bar k$. The notation $P(A \vert C)$ denotes the probability for getting $A$, given that $C$ occurs or $C$ is true. Thus we could write

$$P(k \vert \bar k) = P(k, \bar k) = \frac{\bar k^{k} e^{-\bar k}}{k!} .$$ (19)

Now the reverse question is, ``What is the probability that the mean value is $\bar k$, given that we just made a measurement and got $k$?''. This probability would be denoted $P(\bar k \vert k)$. Now a trivial but important theorem due to Bayes states that

$$P(A \vert C)P(C) = P(C \vert A)P(A)$$ (20)

where $P(C)$ is the a priori probability for $C$ to occur, regardless of whether the event $A$ occurs, and $P(A)$ is the a priori probability for $A$ to occur, regardless of whether the event $C$ occurs. From this theorem we conclude that

$$P(\bar k \vert k) = P(k \vert \bar k)P(\bar k)/P(k)$$ (21)

So we need to know $P(\bar k)$ and $P(k)$ to make progress. The first is the a priori probability for getting a particular value for $\bar k$. If we don't know anything about $\bar k$, except that it is nonnegative, then we must say that any nonnegative value whatsoever is equally probable. Thus without benefit of knowing the outcome of the measurement, we say $P(\bar k)$ is constant, independent of $\bar k$ for nonnegative $\bar k$, and it is zero for negative $\bar k$. So the rhs of this equation reduces simply to

$$P(\bar k \vert k) = N \frac{\bar k^{k} e^{-\bar k}}{k!}$$ (22)

where the normalization factor $N = P(\bar k)/P(k)$ can be determined by requiring that the total probability for having any $\bar k$ is 1. In fact it turns out that $N = 1$, so

$$P(\bar k \vert k) = \frac{\bar k^{k} e^{-\bar k}}{k!} .$$ (23)

This distribution is called the likelihood function for the parameter $\bar k$. Notice that we are now thinking of the rhs as a continuous function of $\bar k$ with fixed $k$. This result is very remarkable, since a single measurement is giving us the whole probability distribution! Recall that if we were to measure the length of a table top, even if we started by assuming we were going to get a Gaussian distribution, a single measurement would allow us only to guess $\bar x$ and would tell us nothing about $\sigma$. To get $\sigma$ takes at least two measurements, and even then we would be putting ourselves at the mercy of the gods of statistics for taking a chance with only two measurements. If we weren't so rash as to assume a Gaussian, we would have to make many measurements of the length of the table top to get the probability distribution in the measured length.

We now ask, what is the most probable value of $\bar k$, given that we just found $k$? This is the value with maximum likelihood. If we examine the probability distribution, we see that it peaks at $k$, just as we might have expected. We may then ask, what is the error in the determination of this value. This is a tricky question, because the Poisson distribution is not shaped like a Gaussian distribution. However, for large $k$ it looks more and more like a Gaussian. Expanding the log of the Poisson distribution for large $k$ and fixed $\bar k - k$ gives

$$P(\bar k \vert k) \approx \exp[-(k - \bar k)^{2}/(2 \bar k)]/\sqrt{2 \pi \bar k}$$ (24)

so for large $k$ the error is

$$\sigma_{\bar k} = \sqrt{k}.$$ (25)

To summarize, a single measurement yields the entire probability distribution. For large enough $k$ we can say that

$$\bar k = k \pm \sqrt{k}.$$ (26)

To see how Bayesian statistics works, suppose we repeated the experiment and got a new value $k^{\prime}$. What is the probability distribution for $\bar k$ in light of the new result? Now things have changed, since the a priori probability for $\bar k$ is no longer constant because we already made one measurement and got $k$. Instead we have

$$P(\bar k) = \frac{\bar k^{k} e^{-\bar k}}{k!}$$ (27)

so

$$P(\bar k \vert k^{\prime}) = N \frac{\bar k^{k^{\prime}} e^{-\bar k}}{k^{\prime}!} \frac{ \bar k^{k} e^{-\bar k}}{k!}.$$ (28)

Notice that the likelihood function is now the product of the individual likelihood functions. A more systematic notation would write this function as $P(\bar k \vert k,k^{\prime})$, i.e. the probability for $\bar k$ having a particular value, given that we made two measurements and found $k$ and $k^{\prime}$. The normalization factor $N$ is obtained by requiring the total probability to be 1. For large $\bar k$, $k$, and $k^\prime$, the most likely value of $\bar k$ is easily shown to be just the average

$$\bar k = (k + k^\prime)/2,$$ (29)

as we should have expected.

The Bayesian approach insists that we fold together all of our knowledge about a parameter in constructing its likelihood function. Thus a generalization of these results would state that the likelihood function for the parameter set $C$, given the independently measured results $A_{1}$, $A_{2}$, $A_{3}$, etc. is just

$$P(C \vert A_{1},A_{2},A_{3},\ldots{}) = N P(A_{1} \vert C) P(A_{2} \vert C)P(A_{3} \vert C)\ldots{},$$ (30)

where $N$ is a normalization factor. Again, this is just the product of the separate likelihood functions. The result is completely general and applies to any probability distribution, not just a Poisson distribution. We will use this result in discussing $\chi^{2}$ fits to data as a maximum likelihood search.