Vector dot product

You already know how to add vectors and multiply them by numbers in $\mathbb{R}^{n}$. However, we haven't discussed yet whether it is possible to multiply vectors by other vectors in $\mathbb{R}^{n}$. Here you're going to learn one of the ways to construct such a new algebraic operation. It is going to provide you with new advanced geometric tools and shed new light on the concepts of angle and orthogonality. Some may argue that this sounds overly abstract and disconnected from real-life applications, but you will see that the opposite is true.

Motivation

Let’s look at a plane, which is a classic representation of a two-dimensional vector space $\mathbb{R}^{2}$. A basis $\{\vec{e}_{1},\vec{e}_{2}\}$ of $\mathbb{R}{^2}$ is typically chosen in such a way that $\vec{e}_{1}$ and $\vec{e}_{2}$ are perpendicular to each other, and the length of each of them is 1 (see the picture below). Here, a vector $\vec{x} = \begin{pmatrix} x_{1} & x_{2}\end{pmatrix}^{\mathsf{T}}$ is identified with $x_{1}\vec{e}_{1}+x_{2}\vec{e}_{2}$.

Now let’s look at another pair of perpendicular vectors $\vec{a}_{1} = \begin{pmatrix} 2 & 4 \end{pmatrix}^{\mathsf{T}}$ and $\vec{a}_{2} = \begin{pmatrix} - 6 & 3 \end{pmatrix}^{\mathsf{T}}$. We will perform a non-obvious procedure, namely, multiply the first coordinates of both vectors and add them with the product of the second coordinates, like this: $2\cdot (- 6) + 4\cdot 3 = -12 + 12 = 0$. Notice that the sum is zero. If you repeat it with another pair of perpendicular vectors, for example $\vec{b}_{1} = \begin{pmatrix} -3 & -2 \end{pmatrix}^{\mathsf{T}}$ and $\vec{b}_{2} = \begin{pmatrix} 2 & -3 \end{pmatrix}^{\mathsf{T}}$, you again obtain $(−3)⋅2+(−2)⋅(−3)=0$. On the contrary, if you take non-perpendicular vectors $\vec{a}_{1}$ and $\vec{b}_{1}$, you will get $2\cdot(-3) + 4\cdot(-2) = −14 \ne 0$. Seems like we stumbled upon a criterion of perpendicularity of vectors! But how are these weird calculations related to the mutual arrangement of vectors? Let’s figure it out!

Definition and properties

Let's start with the formal definition. Define a dot product of two arbitrary columns (vectors) $\vec{x} = \begin{pmatrix}x_{1} & x_{2} & \dots & x_{n}\end{pmatrix}^{\mathsf{T}}$ and $\vec{y} = \begin{pmatrix}y_{1} & y_{2} & \dots & y_{n}\end{pmatrix}^{\mathsf{T}}$ from $\mathbb{R}^{n}$as $$\vec{x}\cdot\vec{y}=\sum_{k = 1}^{n}x_{k}y_{k}=x_{1}y_{1} + x_{2}y_{2} + \dots +x_{k}y_{k}+ \dots + x_{n}y_{n}$$Thereby, given a pair of columns, the dot product returns a real number (scalar), which is the sum of their pairwise multiplied coordinates. This is why it is sometimes referred to as a scalar product. Despite not producing an element of the same nature as the elements in the input, there are a lot of similarities with the multiplication of numbers. For example, the dot product is

• Commutative: $$\vec{x}\cdot\vec{y} = \vec{y}\cdot\vec{x}$$ (This is obvious since $x_{k}y_{k} = y_{k}x_{k}$.)
• Distributive $$\vec{x}\cdot(\vec{y}+\vec{z})=\vec{x}\cdot\vec{y} + \vec{x}\cdot\vec{z}$$(As $x_{k}(y_{k}+z_{k})=x_{k}y_{k}+x_{k}z_{k}$).

However, there are, of course, some differences. For example, the notion of associative property doesn't apply to the dot product. The expression $(\vec{v}_{1}\cdot\vec{v}_{2})\cdot\vec{v}_{3}$ is not a dot product of three vectors, it is a product of a scalar $\vec{v}_{1}\cdot\vec{v}_{2}$ and a vector $\vec{v}_{3}$, which is a completely different operation.

The key idea of a vector space is that vectors interact with vectors through addition and with scalars by multiplication. The bilinear property describes the most general way of combining the dot product with these basic $\mathbb{R}^{n}$ operations

• $$\left(\alpha_{1}\vec{x}_{1} + \alpha_{2}\vec{x}_{2}\right)\cdot \left(\beta_{1}\vec{y}_{1} + \beta_{2}\vec{y}_{2}\right) = \alpha_{1}\beta_{1}\vec{x}_{1}\cdot\vec{y}_{1} + \alpha_{1}\beta_{2}\vec{x}_{1}\cdot\vec{y}_{2} + \alpha_{2}\beta_{1}\vec{x}_{2}\cdot\vec{y}_{1} + \alpha_{2}\beta_{2}\vec{x}_{2}\cdot\vec{y}_{2}$$

In particular, $(\alpha\vec{x})⋅(\beta\vec{y}) = \alpha\beta\vec{x}\cdot\vec{y}$ (here Greek letters denote scalars). The bilinear property follows directly from the definition of the dot product.

Bilinearity is called this way because both multiples of the dot product behave linearly with respect to addition and multiplication by numbers. It makes calculations with vectors almost identical to analogous with numbers. Moreover, sometimes it allows proceeding calculations without any information about columns themselves. For example, if $\vec{x}\cdot\vec{x} = 4$ and $\vec{x}\cdot\vec{y} = - 1$, then $$3\vec{x}\cdot\left(\frac{1}{4}\vec{x} -2\vec{y}\right)=\frac{3}{4}\vec{x}\cdot\vec{x} - 6\vec{x}\cdot\vec{y}=\frac{3}{4}\cdot 4 - 6\cdot(−1)=3+6=9$$

Geometric interpretation (plane)

We described a lot of algebraic properties of the dot product, showing that one can multiply vectors very similar to numbers or standard expressions with variables. When we motivated the introduction of the dot product at the beginning of this topic, we concentrated on its geometric meaning. Let’s find the connection of dot product with geometry. This match of algebraic and geometric properties will allow us to combine these two dominant areas of mathematics, giving us the ability to translate geometric concepts to the algebraic language and back around.

First, let's go back to the plane example.

Consider two arbitrary vectors $\vec{a} = \begin{pmatrix} a_{1} & a_{2} \end{pmatrix}^{\mathsf{T}}$ and $\vec{b} = \begin{pmatrix} b_{1} & b_{2}\end{pmatrix}^{\mathsf{T}}$ (see the picture). First of all, note that according to the Pythagorean theorem, the lengths of $\vec{a}$ and $\vec{b}$ (as a matter of fact of any vector on a plane), can be calculated using the dot product:$$\|\vec{a}\| = \sqrt{a_{1}^{2} + a_{2}^{2}} = \sqrt{\vec{a}\cdot\vec{a}}, \quad \|\vec{b}\| = \sqrt{b_{1}^{2} + b_{2}^{2}} = \sqrt{\vec{b}\cdot\vec{b}}$$Now, let $\varphi$ be an angle between $\vec{a}$ and $\vec{b}$. Thus if $\alpha$ is an angle between $\vec{a}$ and $\vec{e}_{1}$ and $\beta$ an angle between $\vec{b}$ and $\vec{e}_{1}$, then $\varphi = \beta - \alpha$. By definition $$\cos(\alpha) = \frac{a_{1}}{\|\vec{a}\|}, \quad \sin(\alpha) = \frac{a_{2}}{\|\vec{a}\|},\quad \cos(\beta) = \frac{b_{1}}{\|\vec{b}\|},\quad \sin(\beta) = \frac{b_{2}}{\|\vec{b}\|}$$As$$\cos(\beta - \alpha) = \cos(\beta)\cos(\alpha) + \sin(\beta)\sin(\alpha)$$we obtain$$\cos(\varphi) = \cos(\beta - \alpha) = \frac{b_{1}}{\|\vec{b}\|}\cdot\frac{a_{1}}{\|\vec{a}\|} + \frac{b_{2}}{\|\vec{b}\|}\cdot\frac{a_{2}}{\|\vec{a}\|} = \frac{a_{1}b_{1} + a_{2}b_{2}}{\|\vec{a}\|\cdot\|\vec{b}\|}$$By definition $a_{1}b_{1} + a_{2}b_{2} = \vec{a}\cdot\vec{b}$, which implies

$$\cos(\varphi) = \frac{\vec{a}\cdot\vec{b}}{\|\vec{a}\|\cdot\|\vec{b}\|} \Longrightarrow \boxed{\vec{a}\cdot\vec{b} = \|\vec{a}\|\cdot\|\vec{b}\|\cdot\cos(\varphi)}$$This is the most important junction between different points of view on the dot product. Here you saw that the plain calculation of the sum of pairwise products of coordinates of vectors gives the same result as calculating the product of vector lengths together with the cosine of the angle between them. From here we obtain our initial claim.

That follows from the fact that the cosine of a right angle is zero.

Multidimensional generalization

The idea of the dot product being connected with lengths of vectors and angles between them can be used to generalize the $\mathbb{R}^{2}$ case. For example, in $\mathbb{R}^{3}$ it gives the well-known formulas for Pythagorean theorem generalization. In fact, in the general case, this dot product intuition is the simplest way to define length and angles in $\mathbb{R}^{n}$ for any natural $n$. For vectors $\vec{x} = \begin{pmatrix}x_{1}& x_{2} &\dots &x_{n}\end{pmatrix}^{\mathsf{T}}$ and $\vec{y} = \begin{pmatrix}y_{1}& y_{2} &\dots &y_{n}\end{pmatrix}^{\mathsf{T}}$ we define

• the length of $\vec{x}$ as
• $$\|\vec{x}\| = \sqrt{\vec{x}\cdot\vec{x}} = \sqrt{x_{1}^{2} + x_{2}^{2} + \dots + x_{n}^{2}}$$
• the cosine of the angle between $\vec{x}$ and $\vec{y}$ as
• $$\frac{\vec{x}\cdot\vec{y}}{\|\vec{x}\|\cdot\|\vec{y}\|} = \frac{x_{1}y_{1} + x_{2}y_{2}+\dots+x_{n}y_{n}}{\sqrt{x_{1}^{2}+x_{2}^{2} + \dots + x_{n}^{2}}\cdot\sqrt{y_{1}^{2} + y_{2}^{2} + \dots+y_{n}^{2}}}$$There is an ambiguity when it comes to calculating the angle itself, as the angle could be defined by cosine up the sight and some number of $360^{\circ}$ rotations, however, usually, the angle is defined just as
• $$\arccos\left(\frac{\vec{x}\cdot\vec{y}}{\|\vec{x}\|\cdot\|\vec{y}\|}\right)$$to resolve this issue.

This way to define lengths and angles is very universal. Let us look at an example. $\mathbb{R}^{5}$ is a five-dimensional vector space. Unfortunately, our perception is settled in such a way that we can’t imagine it, however now you can calculate an angle between any two five-dimensional vectors, without having an actual picture. The length of vector $\vec{x} = \begin{pmatrix}1&1&-1&0&1\end{pmatrix}^{\mathsf{T}}$ is

$$\|\vec{x}\| = \sqrt{1 + 1+ 1 + 0+1} = 2$$

The length of $\vec{y} = \begin{pmatrix}1,0,0,0,1\end{pmatrix}$ is

$$\|\vec{y}\| = \sqrt{1 + 0 + 0 +0 + 1} = \sqrt{2}$$The angle between $\vec{x}$ and $\vec{y}$ is

$$\arccos\left(\frac{\vec{x}\cdot\vec{y}}{\|\vec{x}\|\cdot\|\vec{y}\|}\right) = \arccos\left(\frac{1 + 0 + 0 + 0 + 1}{2\cdot\sqrt{2}}\right) = \arccos\left(\frac{1}{\sqrt{2}}\right) = 45^{\circ}$$Note, that to find this purely geometric characteristic of vectors disposition not only didn’t we draw any pictures, we just did some simple algebraic calculations.

Lastly, multidimensionality works in another direction also. If we take $n = 1$ for $\mathbb{R}^{1} = \mathbb{R}$ dot product is just usual multiplication!

A few words on applications

Here you have the right to say that, of course, it’s wonderful that you can count some geometric things in a high-dimensional space, but isn’t it just a fantasy applicable only to the fiction of Nolan films or reflections at your leisure. And of course, one can use our developments for these purposes. However, dot product has many more mundane applications. It’s used ubiquitously in science, especially in modern physics, not only because according to some theories spacetime is 26- or 10-dimensional, but because usually physicists do not work with the physical space itself, but with the so-called phase space of equations arising in a problem, which can have any dimension, but in which it is vital to be able to do geometric calculations. It is also indispensable in game development to describe rotations, movements, and light intensities.

In machine learning and data analysis, as you can guess data is usually described using lists and arrays, thus equipping it naturally with a structure of vector space. The whole idea of data analysis is to describe geometric patterns in these spaces to make predictions and classify data, without being able to find lengths and directions of vectors it would be impossible, that is why you will be inevitably calculating dot products.

The philosophy that it is useful to take out from here is that the operation we have constructed, and its generalizations, allow us to endow some not necessarily geometric objects with a geometric structure and calculate distances and angles for them with the same immediacy with which we did it in a checkered notebook in the elementary school.

Conclusion

• The dot product is an operation that matches a real number to any pair of vectors of $\mathbb{R}^{n}$.
• The dot product of $\vec{x} = (x_{1},x_{2},\dots,x_{n})^{\mathsf{T}}$ and $\vec{y} = (y_{1},y_{2},\dots, y_{n})^{\mathsf{T}}$ is defined as
• $$\vec{x}\cdot\vec{y} = x_{1}y_{1} + x_{2}y_{2} + \dots + x_{n}y_{n}$$
• Dot product allows one to define the lengths of vectors and angles between them in a vector space. Namely $\left\|\vec{v}\right\| = \sqrt{(\vec{v}\cdot\vec{v})}$ and $$\arccos\left(\frac{\vec{v}\cdot\vec{w}}{\left\|\vec{v}\right\|\cdot\left\|\vec{w}\right\|}\right)$$ is a value of an angle between $\vec{v}$ and $\vec{w}$.
• For any two vectors $\vec{v}$ and $\vec{w}$, such that the angle between them is $\varphi$
• $$\vec{v}\cdot\vec{w} = \|\vec{v}\|\cdot\|\vec{w}\|\cdot\cos(\varphi)$$
• Two vectors $\vec{v}_{1}$ and $\vec{v}_{2}$, such that $\vec{v}_{1}\cdot\vec{v}_{2} = \vec{0}$, are called orthogonal or perpendicular.