Dependence and independence of events

The patterns we see in data can sometimes be misleading.

For example, if you flip a coin and get tails half of the time, but on one day you only seem to get tails, it can be tempting to believe that the next flip is more likely to be heads. This is an example of the gambler's fallacy — a belief that a certain event is more or less likely to occur due to the past events. In reality, assuming the coin in fair, flips are independent of each other and past results do not affect future outcomes.

To understand this concept better, we will learn about the dependence and independence of events.

Definitions of Independence and Dependence

Let's start with two events:

$A$ – "It will rain today"

$B$ – "Random milk chocolate egg has a toy I still didn't have"

We consider probabilities $\mathbb P(A)$ and $\mathbb P(B)$ non-zero as favorable outcomes. Two numbers $\mathbb P(A)$ and $\mathbb P(B)$ aren't going to be related: even if it rains today, this would not automatically mean I would get the toy I want. And vice versa: if I get the toy I need, I can't be sure if it rains today.

Suppose a box consists of $7$ white marbles and $3$ black marbles. We already know that the probability of picking a white marble on one try is $P(get\ white) = 0.7$ and a black one is $P(get\ black) = 0.3$. But what if two marbles are picked one after the other? In that case, there is a dependency between the probabilities of picking white and black. Imagine that the first draw resulted in a white marble. So now we can pick black marble with probability $\dfrac39=\dfrac13$ (the total number of marbles has decreased to $10-1=9$ after the first try). But if the first draw resulted in a black marble, the probability of getting a white one on the next draw is $\dfrac79$.

Two different examples prompt the following definition:

The concept of independence can be expressed using conditional probabilities.

Conditional probability is the probability of event $A$ occurring given that event $B$ has already occurred with non-zero probability. Referring to the marble problem, $A$ — "The second draw resulted in a white marble", $B$ — "The first draw resulted in a black marble", and $\mathbb P(A |B)$ — the probability to pick a white marble after picking a black one. It is calculated by dividing the probability of both events happening simultaneously by the probability of event $B$ happening: $$\mathbb{P}(A | B) = \frac{\mathbb{P}(A \cap B)}{\mathbb{P}(B)}$$Let's use this formula to formulate a mathematical expression of independent variables. If two events are independent, like in the rain & chocolate egg case, the probability of the event $A$ — "It will rain" occurring remains unchanged, regardless of whether the event $B$ — "Random chocolate egg has the toy we need" has occurred or not. In mathematical terms for $\mathbb P(A \cap B)$ — the probability of rain is conditional that we got the toy, this is expressed as follows: $$\mathbb{P}(A) = \frac{\mathbb{P}(A \cap B)}{\mathbb{P}(B)}$$

Both sides of this formula can be multiplied by $\mathbb P(B)$:

$$\mathbb{P}(A)\cdot \mathbb{P}(B) = \frac{\mathbb{P}(A \cap B)}{\mathbb{P}(B)}\cdot \mathbb{P}(B)$$ On the right side $\mathbb{P}(B)$ cancels out, leaving only $\mathbb{P}(A \cap B)$:

$$\mathbb{P}(A)\cdot \mathbb{P}(B) = \mathbb{P}(A \cap B)$$Using this result, we are now ready to define the more rigorous expression for independence.

Gambler's fallacy: two coins

Let's use our new definition to disprove gambler's fallacy. To demonstrate the independence of a coin flip, consider events $A$ (getting heads on the first flip) and $B$ (getting tails on the second flip). It is important to mention that the coin is fair, so the probabilities of getting a head and tail are equal, and it is $\dfrac{1}{2}$. To find the combined probability, there are four possible outcomes: heads followed by heads, heads followed by tails, tails followed by heads, and tails followed by tails. Only the second outcome (heads followed by tails) satisfies both events. Therefore, $\mathbb{P}(A \cap B) = \dfrac{1}{4}$. The picture below describes all reasoning. To define all probabilities, we mark all numerators as sets of desired outcomes and denominators as sets of possible outcomes, next counting them.

By plugging these values into the formula, $\mathbb{P}(A) \cdot \mathbb{P}(B) = \dfrac{1}{2} \cdot \dfrac{1}{2} = \dfrac{1}{4}$. But we have already got the same result above by counting possible outcomes. Now it can be seen that the events $A$ and $B$ are independent. This means that if heads are obtained on the first flip, the result of the second flip won't be influenced in any way. This calculation can also be performed the other way around with the same result.

Gambler's fallacy: three coins

Consider we flip three different coins. All possible outcomes for $H-$ head and $T-$ tail are:$HHH,\ HHT,\ HTH,\ HTT,\ THH,\ THT,\ TTH,\ TTT$. What is the probability that the first coin comes up heads? Well, for two-sided coins, the probabilities of getting head or tail are equal, and it's $\dfrac12$. We can claim this more formally — by counting outcomes when $H$ goes first: $HHH,\ HHT,\ HTH,\ HTT$, and $P(first\ coin\ came\ up\ H) = \dfrac48 = \dfrac12$.

But suppose now that someone told us that only two of three coins have already come up heads. Now what is the probability that the first coin went heads? The key point is that these questions are completely different. If only two of three coins came up heads, we get a new set of possible outcomes: $HTH,\ THH,\ HHT$. Only two of them require "First coin came up $H$". Therefore, if we know that two out of three coins were heads, then the probability $P(first\ coin\ came\ up\ H)$ is equal to $\dfrac23$. To differentiate the two situations, they write the second case as

$P(first\ coin\ came\ up\ H\ | \ two\ out\ of\ three\ coins\ came\ up\ H)=\dfrac23$

We also can get the same result using the formula of conditional probability:

$\mathbb{P}(A | B) = \dfrac{\mathbb{P}(A \cap B)}{\mathbb{P}(B)}=\dfrac{\mathbb P(first\ coin\ came\ up\ H)}{\mathbb P(two\ coins\ came\ up\ H)}=\dfrac{\mathbb P(\{HHT,\ HTH\})}{\mathbb P(\{HHT,\ HTH,\ THH\})}=\dfrac{2/8}{3/8}=\dfrac23$

Continuing for $n$ coins, remember that there are $2^n$ possible outcomes, and calculating the probability for $n$ coins may be faster without naming all outcomes.

Independence and complements

Proving independence can sometimes be easier using the complement of an event.

For example, $3$ out of $10$ light bulbs don't work. Let $A$ be an event "Random light bulb doesn't work", therefore the complement $A^C$ is "Random light bulb works". If $3$ out of $10$ don't work, then $10-3=7$ light bulbs work. Probabilities of $A$ and $A^C$ are $0.3$ and $0.7$, or if we already knew $\mathbb P(A)=0.3$, according to the formula above $\mathbb P(A^C)=1-0.3=0.7$.

As you can see, the probability of the event is equal to the full sample space minus the probability of its complement:

To visualize it for two independent events $A$ and $B$ use Euler's diagram below:

For two independent events, their complements have special properties as follows:

Technical proof of this fact can be overwhelmingly tedious. To understand this visually, consider the Euler diagrams of two independent events. They have a bizarre property: the ratio of the area of event $A$ to the full event space is equal to the ratio of the area of $A \cap B$ to the area of event $B$. This is a graphical representation of the independence formula using conditional probabilities: $$\frac{\mathbb{P}(A)}{1} = \frac{\mathbb{P}(A \cap B)}{\mathbb{P}(B)}$$

It may seem confusing at first glance, but this way of proof is quite straightforward. Cross multiplicational rule works for the equation of two fractions, so just imagine that we "multiply" the first fraction's numerator by the second fraction's denominator and compare it to the product of the first fraction's denominator and the second fraction's numerator. In our case, the result of "multiplication" is a common colored part of two diagrams, and the statement is proved if two products are equal.

This visual representation of independent events seems convincing. Taking the complement of one or both events will not affect the size of the circles. For instance, let's consider the complement of event $A$:

The picture remains unchanged, but the numerators in the equation have switched. If the original equation was correct, it still holds true. Now, let's consider the complement of event $B$:

From a visual perspective, it yet again seems alright. Now let's try complementing both events:

The equation holds even when both events are complemented, as the bigger denominator matches the bigger numerator.

Disjoint and independent events

Talking about independent events, there is another important event type we need to mention:

Suppose we roll two six-sided dice. The first dice has numbers from $1$ to $6$, the second dice has numbers from $7$ to $12$. For two rolls at the same time, the probability of coming up with two equal numbers is $0$. Events $A\ —$ "First dice's outcome" and $B\ —$"Second dice's outcome" are disjoint: their outcome sets don't have common elements.

On Euler's diagram, two disjoint events are represented as two circles without a common area:

Although often confused, disjoint and independent events are very different things. Disjoint events never occur at the same time. Independent events do, but they do not influence each other's probabilities. Regular events have no explicit connection between the product of their probabilities and the probability of their mutual occurrence. This animation illustrates the difference:

1. Disjoint events: P(A), P(B) are random numbers, P(A∩B) is always 0
2. Independent events example: Product of P(A) and P(B) is always equal to P(A∩B)
3. Regular events example: Probabilities P(A), P(B) and P(A∩B) are random and aren't in a relationship

There is one particular connection between dependent and disjoint events.

This connection is quite straightforward to prove. For two disjoint events $A$ and $B$, If $\mathbb{P}(A) \neq 0$ and $\mathbb{P}(B) \neq 0$, their product $\mathbb{P}(A) \cdot \mathbb{P}(B)$ is not equal to zero:

$$\mathbb{P}(A) \cdot \mathbb{P}(B) \neq 0$$By definition of disjoint events:

$$\mathbb{P}(A \cap B) = 0$$

$$\mathbb{P}(A) \cdot \mathbb{P}(B) \neq \mathbb{P}(A \cap B)$$Hence, the events are dependent.

Types of independence

All this time, we have been talking about pairs of events. But what if you wanted to check three or more events for independence? Turns out, events can be independent in two distinctly different ways: they can either be pairwise independent or mutually independent from each other. Let's see how that can be expressed mathematically:

Events $A_1,\ A_2,\ \dots,\ A_n$ are pairwise independent if and only if for any two events $A_i$ and $A_j$

$$\mathbb{P}(A_i) \cdot\mathbb{P}(A_j) = \mathbb{P}(A_i \cap A_j),\ i \neq j$$

In simple terms, events are pairwise independent if all the pairs of events are independent. For example, a set of events $A_1,\ A_2,\ A_3$ is going to be pairwise independent if and only if $3$ following conditions are true:

$\mathbb{P}(A_1) \cdot\mathbb{P}(A_2) = \mathbb{P}(A_1 \cap A_2)\\ \mathbb{P}(A_1) \cdot\mathbb{P}(A_3) = \mathbb{P}(A_1 \cap A_3)\\ \mathbb{P}(A_2) \cdot\mathbb{P}(A_3) = \mathbb{P}(A_2 \cap A_3)$

Finally, events are mutually independent if every event is independent of any intersection of the other events. To be more formal,

events $A_1,\ A_2,\ \dots,\ A_n$ are mutually independent if and only if for every number $1 \leqslant k \leqslant n$ and for every subset $B_1,\ B_2,\ \dots,\ B_k \subset A_1,\ A_2,\ \dots,\ A_n$

$$\prod_{i=1}^k\mathbb{P}(B_i) = \mathbb{P}(\bigcap_{i=1}^k B_i)$$

$\bigcap$ is a fancy sign for multiple element intersection, like $\sum$ for a sum or $\prod$ for a product.

A set of events $A_1,\ A_2,\ A_3$ above is going to be mutually independent if and only if $4$ following conditions are true:

$\mathbb{P}(A_1) \cdot\mathbb{P}(A_2) = \mathbb{P}(A_1 \cap A_2)\\ \mathbb{P}(A_1) \cdot\mathbb{P}(A_3) = \mathbb{P}(A_1 \cap A_3)\\ \mathbb{P}(A_2) \cdot\mathbb{P}(A_3) = \mathbb{P}(A_2 \cap A_3)\\ \mathbb P(A_1) \cdot \mathbb P(A_2) \cdot \mathbb P(A_3)=\mathbb P(A_1 \cap A_2 \cap A_3)$

Usually, when mathematicians talk about independence without specifying "pairwise independence", they mean mutual independence. A mutual independence is a stronger requirement: there are problems when any two out of $3$ events are independent, but all events are not independent, therefore they can not be mutually independent. Moreover, mutual independence is a more restricting requirement because even for a set of $4$ events it asks for the independency of each pair, triple, and quartet.

Conclusion

Dependent events influence each other's probabilities, independent events don't. Whether you remove excessive variables from your mathematical model or just flip a coin, differentiating between the two comes in handy. Here are some key points:

• Events $A$ and $B$ are called independent, if, and only if $\mathbb{P}(A) \cdot \mathbb{P}(B) = \mathbb{P}(A \cap B)$. They are called dependent otherwise.

• The relationship between the event and its complement is defined by $\mathbb{P}(A) = 1 - \mathbb{P}(A^C)$. If events $A$ and $B$ are independent, their complements $A^C$ and $B$, $A$ and $B^C$, $A^C$ and $B^C$ are independent.

• Events $A$ and $B$ are called disjoint, if and only if $\mathbb{P}(A \cap B) = 0$. Two disjoint events with non-zero probabilities are dependent.

• For more than two events, there are two kinds of independence: pairwise independence and mutual independence. Events are pairwise independent if all the pairs of events are independent. Events are mutually independent if every event is independent of any intersection of the other events.