Bayes theorem

Formula of total probability

First of all, it's important to remember the definition of mutually exclusive (disjoint) events. Two events are mutually exclusive if they can't both occur at the same time. For example, event "randomly chosen number is greater than $1$" and event "randomly chosen number is less than $1$" are disjoint.

Next, we need one more definition. When a sample space $\Omega$ is distributed down into some mutually exclusive events $H_1, H_2, \dots$ (finite or infinite set) such that their union forms the sample space itself $H_1 \cup H_2 \cup {}_ \cdots = \Omega$, then events $H_1, H_2, \dots$ are called exhaustive events. Also events $H_1, H_2, \dots$ are called hypotheses.

And the last in this paragraph is the law of total probability. Let's suppose, that $H_1, H_2, \dots$ are exhaustive events and we need to calculate the probability of some event $A$ the following way ($n$ can be $+\infty$):

$$\mathbb{P}(A) = \sum \limits_{i =1}^{n}\mathbb{P}(A|H_i)\mathbb{P}(H_i)$$

Why is this formula correct? Event $A$ can be represented as a union of non-intersecting events $A \cap H_i$. These events are mutually non-intersecting, because the events $H_1, H_2, \dots$ are mutually exclusive. Therefore:

$$\mathbb{P}(A) = \sum_{i=1}^{n}\mathbb{P}(A \cap H_i)$$

Next, we use the definition of the conditional probability:

$$\mathbb{P}(A \cap H_i) = \mathbb{P}(A|H_i)\mathbb{P}(H_i)$$

Finally, we have the already written formula:

$$\mathbb{P}(A) = \sum_{i =1}^{n}\mathbb{P}(A|H_i)\mathbb{P}(H_i)$$

Let's see a simple example of usage of the total probability formula. Three factories are producing the same pills. The first one is producing $20\%$ of all pills, the second one $-$ $30 \%$ and the third one $-$ $50 \%$. And the first factory has $5 \%$ defective pills, the second $-$ $3 \%$, the third $-$ $4 \%$. What is the probability of buying defective pills?

Let event $H_i$ mean that pills are produced by a factory number $i$, event $A$ is buying defective pills.

We have $\mathbb{P}(A | H_1) = 0.05$, $\mathbb{P}(A|H_2) = 0.03$ and $\mathbb{P}(A|H_3) = 0.04$.

Using the formula of total probability we get the answer:

$\mathbb{P}(A) = \mathbb{P}(A|H_1)\cdot\mathbb{P}(H_1) + \mathbb{P}(A|H_2)\cdot\mathbb{P}(H_2) + \mathbb{P}(A|H_3)\cdot\mathbb{P}(H_3) = 0.05 \cdot 0.2 + 0.03 \cdot 0.3 + 0.04 \cdot 0.5 = 0.039$

Bayes' theorem: simple form

In this paragraph we introduce the simple form of Bayes' formula.

Suppose, there are events $A$, $B$ and $\mathbb{P}(A) > 0, \mathbb{P}(B) > 0$.

First, let's look at the definition of conditional probability:

$$\mathbb{P}(A|B) = \frac{\mathbb{P}(A \cap B)}{\mathbb{P}(B)}$$

But, on the other hand:

$$\mathbb{P}(B|A) = \frac{\mathbb{P}(B \cap A)}{\mathbb{P}(A)} = \frac{\mathbb{P}(A \cap B)}{\mathbb{P}(A)}$$

Notice that both fractions have the same numerators and we can equate them:
$$\mathbb{P}(A|B)\mathbb{P}(B) = \mathbb{P}(B|A)\mathbb{P}(A)$$Also it can be divided by $\mathbb{P}(B)$ and the result is a simple form of Bayes' theorem:

$$\mathbb{P}(A|B) = \frac{\mathbb{P}(B|A)\mathbb{P}(A)}{\mathbb{P}(B)}$$

With this formula we could calculate conditional probability without knowing the probability of $A \cap B$.

Extended Bayes' theorem

So, we have the formula $\mathbb{P}(A|B) = \frac{\mathbb{P}(B|A)\mathbb{P}(A)}{\mathbb{P}(B)}$. It's easy to remember and use that, but in real problems, the probability of event $B$ is very often unknown. We can try to use the formula of total probability for event $B$:

$$\mathbb{P}(A|B) = \frac{\mathbb{P}(B|A)\mathbb{P}(A)}{\sum \limits _{i=1}^{n}\mathbb{P}(B|H_i)\mathbb{P}(H_i)}$$where $H_1, \dots, H_n$ are hypotheses

Okay, now let's try Bayes' theorem where $B$ is some hypothesis $H_k$ and understand the trustability of this hypothesis.

Suppose we have a hypothesis $H_1, H_2, \dots,H_n$ and some event $A$.

Using the previous formula we know that

$$\mathbb{P}(A|H_k) = \frac{\mathbb{P}(H_k|A)\mathbb{P}(A)}{\mathbb{P}(H_k)}$$

We can use the formula of total probability for event $A$, as we have done before: $$\mathbb{P}(A|H_k) = \frac{\mathbb{P}(H_k|A)\mathbb{P}(A)}{\mathbb{P}(H_k)}= \frac{\mathbb{P}(H_k|A)\mathbb{P}(A)}{\sum \limits_{i=1}^{n}\mathbb{P}(H_k|H_i)\mathbb{P}(H_i)}$$

The last formula is called Bayes' theorem or Bayes' formula.

Actually, if some event $A$ happened and we know all probabilities $\mathbb{P}(A|H_i)$ and $\mathbb{P}(H_i)$ we can calculate the probability that some hypothesis $H_k$ really happened.

For example, let's look at the problem about the factories from the previous paragraph. Now we need to find the probability of having bought a pill produced by the first factory, if this pill is defective.

$H_i$ means that pills are produced by factory number $i$, as it meant before, event $A$ is buying defective pills. Formally, we need to find $\mathbb{P}(H_1|A)$. We calculated that $\mathbb{P}(A) = 0.039$.

Let's use the Bayes' theorem:

$$\mathbb{P}(H_1|A) = \frac{\mathbb{P}(A|H_1)\mathbb{P}(H_1)}{\mathbb{P}(A|H_1)\mathbb{P}(H_1) + \mathbb{P}(A|H_2)\mathbb{P}(H_2) + \mathbb{P}(A|H_3)\mathbb{P}(H_3)} = \frac{0.05 \cdot 0.2}{0.039} \approx 0.256$$

By this way we recalculated the probability that we have bought a pill from the first factory. If it was unknown that this pill is defective, the probability that we have bought a pill from the first factory would be $\mathbb{P}(H_1) = 0.2$.

Example

Let's consider an example: two shooters toss a coin and decide which of them will shoot. If it comes up heads, the first one will shoot, if it comes up tails$-$ the second one. The probability that the first shooter hits the target is $1$, the second one $-$ $10^{-4}$. Suppose that we know the result of this experiment: the target was hit. For each shooter what is the probability that he was shooting in this case?

We have $2$ hypotheses: $H_1$ is that the first shooter will shoot, $H_2$ is for the second. Also let the event $A$ mean that the target was hit. We know that $\mathbb{P}(A|H_1) = 1$ and $\mathbb{P}(A|H_2) = 10^{-4}$. So, we need to calculate $\mathbb{P}(H_1 | A)$ and $\mathbb{P}(H_2|A)$.

Notice, that if we didn't know that the target was hit, it would be clear that $\mathbb{P}(H_1) = \mathbb{P}(H_2) = \frac{1}{2}$.

Now we are ready to use Bayes' formula.

$$\mathbb{P}(H_i|A) = \frac{\mathbb{P}(H_i)\mathbb{P}(A|H_i)} {\mathbb{P}(H_1)\mathbb{P}(A|H_1) + \mathbb{P}(H_2)\mathbb{P}(A|H_2)}$$

$$\mathbb{P}(H_1|A) = \dfrac{\frac{1}{2}\cdot1}{\frac{1}{2} \cdot 1 + \frac{1}{2} \cdot {10^{-4}}} \approx 0.9999$$

$$\mathbb{P}(H_2|A) = \dfrac{\frac{1}{2}\cdot 10^{-4}}{\frac{1}{2} \cdot 1 + \frac{1}{2} \cdot {10^{-4}}} \approx 0.0001$$

So, if it's unknown that target was shot, then the probabilities of heads and tails are equal, but if the target was shot, then the probability of heads is $10^4$ times more than the probability of tails.

Conclusion

Now we have learned very important formulas – the formula of the total probability and Bayes' formula. The formula of the total probability helps us to calculate probability using exhaustive events. And Bayes' formula can recalculate the probability of some event if something is known about the result of this event. Both formulas are widely used in next topics.