Descriptive Statistics basics

Concept of averages

Averages are statistical measures used to summarize and describe a set of data. They provide a representative value that denotes the central tendency of the data. The most commonly used averages are the mean, median, and mode.

Mean

The mean, also known as the arithmetic mean, is calculated by summing up all the values in a dataset and dividing the sum by the number of values.

$\text{Mean} = \frac{\text{Sum of all values}}{\text{Number of values}}$

For instance, if you have a dataset of test scores: 85, 90, 78, 92, and 75, the mean score is calculated as follows:

Mean $\bar{x} = \frac{85 + 90 + 78 + 92 + 75}{5} = \frac{420}{5} = 84$ The mean test score is 84.

The mean is sensitive to extreme values and can be affected by outliers.

The mean is often used when the data follows a normal distribution.

Median

The median is the middle value in a dataset when the values are arranged in ascending or descending order. If the dataset has an odd number of values, the median is the middle value. If there's an even number of observations, the median is the average of the two middle numbers.

Imagine you have the following set of ages: 29, 22, 43, 38, 38, 40, 31. To find the median,

• Sort the dataset in ascending order: 22, 29, 31, 38, 38, 40, 43.

• Determine the number of data points in the dataset. In this case, we have 7 data points.

• Find the middle value(s) in the sorted dataset. Since we have an odd number of data points (7), there is only one middle value. In this case, the middle value is the 4th data point, which is 38.

• If the dataset has an even number of data points, calculate the average of the two middle values. However, since we have an odd number of data points in this example, we don't need to perform this step.

• The median of the dataset is 38.

The median is less affected by outliers and is a better measure of central tendency in skewed distributions.

Mode

The mode is the value that occurs most frequently in a dataset. A dataset may have one mode (unimodal), two or more (bimodal or multimodal, respectively), or none at all if all values are unique. Identifying the mode is as simple as counting the frequency of each value.

Consider a set of shoe sizes: 5, 7, 7, 7, 9, 10, 10. The size that appears most often is 7, making it the mode of this dataset.

The mode is useful for categorical or nominal data.

Understanding the spread in data

The spread of data, also known as dispersion, refers to how much variation there is in a dataset. It's a critical concept because it gives us insights into the consistency and reliability of the data. Let's delve into the common measures used to understand data spread.

1. Range

The range is the difference between the highest and lowest values in a dataset.

$Range = Maximum Value - Minimum Value$

Practical Example
Imagine a dataset of weekly sales figures for a small store: 110, 200, 220, 250, 350, 275, and 300.
Maximum value = 350, Minimum value = 110
The Range is $350-110 = 240$

While it gives a basic idea of the spread, it can be sensitive to outliers.

2. Interquartile range (IQR)

The interquartile range represents the range between the first quartile (25th percentile) and the third quartile (75th percentile). It is less sensitive to extreme values and provides a measure of spread within the central 50% of the data.

$IQR = Q3 - Q1$, where Q1 is the first quartile (25th percentile) and Q3 is the third quartile (75th percentile) of the dataset.

Practical Example
Consider the following ordered dataset of test scores: 55, 60, 65, 70, 75, 80, 85, 90, 95.
The first quartile (Q1) is 65, and the third quartile (Q3) is 85.
Thus, IQR is $85 - 65 = 20$.

3. Variance

Variance is a measure that tells us how much the values differ from the mean. It's an average of the squared differences between each value and the mean.

For a sample: $s^2 = \frac{1}{n-1} \sum_{i=1}^{n} (x_i - \bar{x})^2$

For a population: $\sigma^2 = \frac{1}{N} \sum_{i=1}^{N} (x_i - \mu)^2$

where ($\bar{x}$) is the sample mean,
($\mu$) is the population mean,
($x_i$) is each value,
($n$) is the sample size, and ($N$) is the population.

Practical Example
Imagine you are a data analyst working for an e-commerce company. Your task is to analyze the daily sales data for a particular product over a month. You have collected the following sales figures:

To calculate the variance, we need to find the mean first. The mean represents the average sales value.

Mean $(x̄) = (100 + 150 + 120 + 130 + 110) / 5 = 122$

Now, let's calculate the variance for this sample:

Variance $(s^2) = (1/5-1) * [(100 - 122)^2 + (150 - 122)^2 + (120 - 122)^2 + (130 - 122)^2 + (110 - 122)^2]$

$= (1/4) * [484 + 784 + 4 + 64 + 144]$

$(s^2) = (1/4) * 1480 = 370$

The variance tells us how much the sales figures deviate from the mean. In this case, the variance is 370, which indicates that the sales figures vary quite a bit from the average daily sales.

4. Standard deviation

The standard deviation is the square root of the variance and provides a measure of the average distance from the mean.

For a sample: $s = \sqrt{\frac{1}{n-1} \sum_{i=1}^{n} (x_i - \bar{x})^2}$

For a population: $\sigma = \sqrt{\frac{1}{N} \sum_{i=1}^{N} (x_i - \mu)^2}$

Practical Example
Using the same dataset of daily sales data from the variance example, to find the standard deviation, we take the square root of the variance.

Standard Deviation $(s) = √370 ≈ 19.24$

The standard deviation provides a measure of the average distance between each sales figure and the mean. In this example, the standard deviation is approximately 19.24, indicating that the daily sales fluctuate by an average of 19.24 around the mean.

5. Mean absolute deviation (MAD)

The mean absolute deviation is the average of the absolute differences between each data point and the mean.

$Mean Absolute Deviation = \frac{1}{n} \sum_{i=1}^{n} |x_i - \bar{x}|$

Practical Example
Using the same dataset of daily sales data from the variance example, to calculate the mean absolute deviation, we need to find the absolute difference between each sales figure and the mean, and then calculate the average of these absolute differences.

Mean $(x̄) = (100 + 150 + 120 + 130 + 110) / 5 = 122$

Absolute differences: $|100 - 122| = 22$ $|150 - 122| = 28$ $|120 - 122| = 2$ $|130 - 122| = 8$ $|110 - 122| = 12$

Mean absolute deviation $(MAD) = (1/5) * (22 + 28 + 2 + 8 + 12) = 14.4$

The mean absolute deviation for this example is approximately 14.4. It represents the average absolute difference between each sales figure and the mean, indicating the average amount of deviation from the mean sales value.

Conclusion

By mastering the calculation and application of averages such as the mean, median, and mode, you can identify central tendencies within your data. Measures of spread, like range, interquartile range, variance, standard deviation, and mean absolute deviation, provide insight into the variability of your data, which is crucial for interpreting results and making informed decisions. These foundational concepts form the bedrock of data analysis, enabling you to communicate findings effectively and build a robust analytical skill set that is essential for a successful career in data analytics.