Formal definition of probability

In our everyday life, we often deal with such statements as:

• It is likely to snow tonight;

• Most probably he won't pass the exam;

• Someone will win the lottery, but it is most unlikely to be one of us;

• There is almost no chance to meet a dinosaur in the street.

All of them refer to some uncertainty in the situation. How can we express this uncertainty in a rigorous mathematical way? Here the concept of probability arises.

Probability is a function

You already know that for every experiment you can define a sample space $\Omega$ — a set of all possible outcomes. An event is a subset of the sample space. Probability is just a function that maps events to real numbers in $[0, 1]$, satisfying some special (but quite simple, especially for typical application cases) properties.

You know that a function is defined by its domain (the set of all possible inputs to which we can apply that function), its codomain (the set where the output value lies) and a rule describing how to obtain the output given the input. As you can see, in probability theory that domain is a set of events, which we call an event space and denote by $\mathcal F$. The codomain is $[0, 1]$.

Mathematically speaking, an event space must be a sigma-algebra or sigma-field, which means it must satisfy a number of properties from the measure theory. However, for most applications, especially when dealing with countable sample spaces, it is safe to simply define an event space as a set of all subsets of a sample space. In doing so, if our sample space $\Omega$ contains $n$ elements, then $\mathcal F$ must contain $2^n$ elements. That is why a set of all subsets of a set is normally called a power set. All in all, we commonly choose the event space to be a power set of the sample space, which we write as $\mathcal F = 2^{\Omega}$ for short.

Properties of probability

So we have a function $P: \mathcal{F} \rightarrow [0, 1]$. What properties must it satisfy to be a probability?

1. The probability of a sure event is 1: $P(\Omega) = 1$

2. If we have disjoint events (disjoint subsets of $\Omega)$ , denoted by $A_{1}, A_{2}, ... A_{n}$, the probability of their union is a sum of their probabilities:$P(A_1 \cup A_2 \cup \ldots \cup A_n) = P(A_1) + P(A_2) + \ldots + P(A_n)$Moreover, if you have an infinite countable set of disjoint events, the probability of their union is also a sum of its probabilities. (If you are not used to infinite sums, don't worry, we'll discuss them later.)

These properties imply, for example, that a probability of impossible event is zero: $P(\emptyset) = 0$. Why? Well, for any event $A$, its union with an impossible event is $A$ again: $A = A \cup \emptyset$. So,$P(A) = P(A \cup \emptyset) = P(A) + P(\emptyset)$

and $P(\emptyset) = P(A) - P(A) = 0$. That matches our intuition quite well!

Another conclusion we can make is related to an experiment where all outcomes are equally likely. Let's say there are $n$ possible outcomes (or, speaking more formally, $|\Omega| = n$). Then the probability of any elementary event is equal to $1/n$, again agreeing with our common sense.

Conclusion

In this topic, we have defined the concept of probability. Its roots lie in our common sense, but nevertheless it plays a key role in many modern and advanced areas, such as data science and financial mathematics.