Linear combinations and linear independence

You are already familiar with vectors and the key role they play in linear algebra and other fields. A vector is something that has magnitude and direction. It’s useful for describing various things that may need to be dealt with in programming situations, like the velocity of a vehicle in a video game.

However, vectors are rarely studied individually. We usually want to know how they are related and how they affect each other. For example, we may want to model whether the velocity of the wind affects that of a car. This is where the concept of linear dependence comes in handy, as it allows us to understand and describe the relationship between vectors, and this topic is a fundamental concept in linear algebra and other areas of math and science.

What is linear dependence?

Let's try to understand the concept of linear dependence through an analogy. Suppose that you love painting and are considering which color paints to buy. You don't have a lot of money, so you want to be strategic when picking the colors.

Does it make sense to buy red, yellow, and orange? Well, we can make orange by mixing red and yellow paints, so buying orange paint seems redundant. What if you buy red, yellow, and blue? In this case, none of the colors can be combined to produce the other, so we can say these colors are independent of each other.

A set of colored paints are dependent if some of them can be combined to produce another. Similarly, a vector is dependent on other vectors if it can be expressed as a linear combination of others. Such a vector is redundant; it can be obtained by combining other vectors, so it does not add more information. Let's go deeper into the concept of linear combinations and linear dependence in the next section.

Formal definition

Now we are ready to define our new concepts formally. Consider some vectors $v_1, v_2, \ldots, v_n$ and scalars $c_1, c_2, \ldots, c_n$. The vector:$$w = c_1v_1 + c_2 v_2 + \ldots + c_nv_n$$

is called a linear combination of $v_1, v_2, \ldots, v_n$.

If all scalars are zero, the linear combination is called trivial. If at least one scalar is not zero, it is called non-trivial. For example, the following combination is trivial.

And the following combination is non-trivial.

$$\textcolor{red}{1}\cdot \begin{pmatrix} 2 \\ 3 \end{pmatrix}-\textcolor{red}{3}\cdot \begin{pmatrix} 1 \\ 1.5 \end{pmatrix} = \textcolor{blue}{\begin{pmatrix} -1 \\ -1.5 \end{pmatrix}}$$

Now let's use our understanding of linear combinations and triviality to define linear dependence. A set of vectors is linearly dependent if one of the vectors in the set can be represented by some non-trivial linear combination of the other vectors in the set. More formally, we can say that $n$ vectors $x_1, x_2, \ldots, x_n$ are linearly dependent if there exist scalars $c_1, c_2, \ldots, c_n$, at least one of which is not zero, such that:$$c_1x_1 + c_2x_2 + \ldots+c_nx_n=0.$$For example, the vectors $x_1 = \begin{pmatrix} 2 \\ 3 \end{pmatrix} , \; x_2 = \begin{pmatrix} 1 \\ 1.5 \end{pmatrix}, \; x_3 = \begin{pmatrix} 3 \\ 2 \end{pmatrix}$ are linearly dependent. Indeed, we found the coefficients $c_1 = 1,\; c_2 = -2,\; c_3 = 0$, such that their linear combination is zero:

$$1\begin{pmatrix} 2 \\ 3 \end{pmatrix}-2 \begin{pmatrix} 1 \\ 1.5 \end{pmatrix} + 0\begin{pmatrix} 3 \\ 2 \end{pmatrix} = \begin{pmatrix} 0 \\ 0 \end{pmatrix}$$

We know about linear dependence, but how can we define linear independence? Pretty simple: if a set of vectors is not linearly dependent, then it is linearly independent. Not good enough? Well, formally we can say that $n$ vectors $x_1, x_2, \ldots, x_n$ are linearly independent if the only scalars $c_1, c_2, \ldots, c_n$ satisfying $c_1x_1 + c_2x_2 + \ldots+c_nx_n=0$, are$$c_1 = c_2 = \ldots =c_n= 0.$$

That is, a set of vectors is linearly independent if only the trivial linear combination and no other linear combination of the vectors equals the zero vector.

For example, the following vectors are linearly independent, because only the trivial combination of the vectors gives the zero vector (Try it yourself!):

$$0 \begin{pmatrix} 2 \\ 3 \end{pmatrix}+0 \begin{pmatrix} 3 \\ 10 \end{pmatrix} = \begin{pmatrix} 0 \\ 0 \end{pmatrix}$$

Geometric intuition

There is a pretty intuitive geometric interpretation of linear dependence in terms of geometry. In geometric representation, two non-zero vectors are linearly dependent if and only if they are collinear, that is, they lie on a single straight line. Algebraically, two vectors $\overrightarrow{p}$ and $\overrightarrow{q}$ are collinear if for some scalar $n$,

$$\overrightarrow{p} = n \cdot \overrightarrow{q}$$ Here is an example:

As shown in the graph above, $\overrightarrow{AB} = \begin{pmatrix} 1 \\ 2 \end{pmatrix}$, $\overrightarrow{AC} = \begin{pmatrix} 2 \\ 4 \end{pmatrix}$, and $\overrightarrow{AD} = \begin{pmatrix} 2 \\ 2 \end{pmatrix}$.

What do we know about these vectors? One thing is that $\overrightarrow{AB}$ and $\overrightarrow{AC}$ lie on the same line, so they are collinear. $\overrightarrow{AB}$ can be expressed as $\frac{1}{2} \overrightarrow{AC}$, and $\overrightarrow{AC}$ as $2 \overrightarrow{AB}$. On the other hand, the vector $\overrightarrow{AD}$ is not collinear with the other two and is not linearly dependent on them. So, no matter which number we pick, we won't be able to multiply $\overrightarrow{AB}$ or $\overrightarrow{AC}$ by it to get $\overrightarrow{AD}$.

Now let's get back to linear dependence. Any 3 vectors in 2D space would be linearly dependent. Let there be three 2D vectors:

$x = \begin{pmatrix} x_1 \\ x_2 \end{pmatrix}, \; y = \begin{pmatrix} y_1 \\ y_2 \end{pmatrix}, \; z = \begin{pmatrix} z_1 \\ z_2 \end{pmatrix}.$

Then it is always possible to solve the following system of equations with respect to $c_1, c_2, c_3$:

$\begin{cases} c_1 x_1 + c_2 y_1 +c_3 z_1 = 0 \\ c_1 x_2 + c_2 y_2 +c_3 z_2 = 0 \end{cases}$

Listing the solutions in all possible cases ($z_1 = 0$, $z_1 \neq 0$ and so on) would be tedious, but you can try and see it for yourself.

For example, in the case of $\overrightarrow{AB}$, $\overrightarrow{AC}$, and $\overrightarrow{AD}$:

$$\begin{pmatrix} 1 \\ 2 \end{pmatrix} + 0\begin{pmatrix} 2 \\ 2 \end{pmatrix} - 0.5\begin{pmatrix} 2 \\ 4 \end{pmatrix} = \begin{pmatrix} 0 \\ 0 \end{pmatrix}$$Geometrically, out of three vectors on a plane (i.e. directions of movement), one always can be expressed as the sum of two others. Hence any plane is a two-dimensional space.

We looked at 2D space, but what about 3D space? Considering everything just said, it is easy to geometrically interpret linear dependency in 3D space. Three vectors are linearly dependent if and only if they are coplanar, i.e. lie on the same plane.

Conclusion

Now let's quickly sum up the main points of what you have seen and learned about linear combinations and linear dependence:

• Vectors are linearly dependent if you can express one of them as a sum of other vectors multiplied by some scalars, such that at least one of these scalars is different from $0$.
• With linear combinations, the procedure is the same as with linearly dependent vectors. We express one vector in terms of the sum of other vectors multiplied by some scalars. If this can be satisfied only when all scalars are equal to $0$, then the linear combination is trivial. Otherwise, if one or more of the scalars are different from $0$, then the linear combination is non-trivial.
• If you interpret linear dependence geometrically, you will see that vectors are linearly dependent if they are colinear in 2D, coplanar in 3D, and $n$ vectors are linearly dependent if they are in $n-1$ dimensional space
• You can also see that if elements of two vectors are proportional, then they are linearly dependent.