Lucrative Majors and Gender Distribution

To find which majors are more lucrative, set the Major column name as the index of the DataFrame. Then, we will get a subset of the Series popular with men and women using .loc. After that, we will get only the Median column from the DataFrame. It creates a Series with Median income values, keeping the original index. We can visualize the results with a plot.

# set the subplots to be side-by-side
fig, axs = plt.subplots(nrows=1, ncols=2, figsize=(20, 4), sharey=True)

# get the most popular majors with men and women
index_popular_women = df.set_index("Major").ShareWomen.nlargest(10).index
index_popular_men = (1 - df.set_index("Major").ShareWomen).nlargest(10).index

# get the median income for the majors popular with women
(
    df.set_index("Major")
    .loc[index_popular_women]
    .Median
    .sort_values()
    .plot(kind="bar", ax=axs[0], alpha=0.9, color="red", title="The median income of majors with more Women", xlabel="")
)

# get the median income for the majors popular with men
(
    df.set_index("Major")
    .loc[index_popular_men]
    .Median
    .sort_values()
    .plot(kind="bar", ax=axs[1], alpha=0.7, color="orange", title="The median income of majors with more Men", xlabel="")
)

plt.show()

We can analyze which majors have the highest and lowest unemployment rates or the most part-time jobs.

Salary bands

Imagine this messy dataset contains the most recent graduate job and salary information. You are in the process of picking your major and want to choose one with higher prospects of securing a well-paying job.

To perform this analysis, you will first divide the median salary for the major categories into bands. The Major_category column is like a school that contains several departments or majors. You will see the typical median starting salaries for people in these schools (pay attention to the square brackets):

(
    df
    .assign(salary_range=pd.cut(df.Median, bins=[0, 30_000, 50_000, 70_000, 100_000], labels=['<30k', '30-50k', '50-70k', '70k+']))
    .groupby(['salary_range', "Major_category"])["Median"].mean().unstack()
    .style.background_gradient(cmap='PuBu')
)

The assign method creates a new single column in addition to the original ones in df without altering it. In other words, the assign method helps us create a new DataFrame object. The pd.cut function divides the median salaries into brackets. The groupby method creates a multi-index object that groups the rows in the DataFrame into their salary_range and Major_category values. We get a multi-index Series to find the mean of the Median salaries. We use the unstack method to convert this multi-index series into a DataFrame.

We can see from the results the median salaries of major categories such as Engineering, Computers and Mathematics, and Law and Public Policy are above the 30k starting salary band.

We can see the variations in median salaries for selected major categories with the code block:

fig, ax = plt.subplots(nrows=1, ncols=5, figsize=(20, 6), sharey=True)
ax = ax.ravel()

for i, category in enumerate(["Engineering", "Computers & Mathematics", "Physical Sciences", "Law & Public Policy", "Business"]):
    for major_cat, group_code in df.groupby("Major_category")["Median"]:
        if category == major_cat:
            ax[i].violinplot(group_code, showmeans=True, showextrema=True)
            ax[i].set_title(major_cat)

plt.tight_layout()
plt.show()

High wages and low unemployment

Knowing that it pays well is not enough if you're considering a major. You also want to make sure that job opportunities are abundant and long-lasting. Therefore, it's crucial to identify the majors that offer high salaries and low unemployment rates.

(
    df
    .assign(
        earning_quantile=pd.qcut(df.groupby("Major")["Median"].transform(func=np.mean), q=[0.0, 0.25, 0.50, 0.90, 1.0], labels=[25, 50, 90, 100]),
        unemployment_quantile=pd.qcut(df.groupby("Major")["Unemployment_rate"].transform(func=np.mean), q=[0.0, 0.1, 0.50, 0.75, 1.0], labels=[10, 50, 75, 100])
    )
    .where(lambda x: (x.earning_quantile == 90) & (x.unemployment_quantile == 10))
    [["Major", "Median", "Unemployment_rate"]]
    .dropna()
    .assign(
        Unemployment_rate=lambda x: x.Unemployment_rate * 100
    )
    .set_index("Major")
    .style.background_gradient(cmap='PuBu')
)

In the above code, we created new single columns for earning_quantile and unemployment_quantile. These columns contain the median income and Unemployment rate numerical values in quantiles. The where method can filter the resulting DataFrame for rows with the median income in the 90th quantile and the unemployment rate in the 10th quantile.

We can see that majors like Mathematics & Computer Science and Botany have zero unemployment rates. We can also notice Petroleum Engineering, the major with the highest median salary, does not appear here. This is because this major has an unemployment rate above the 10th quantile. Why demand a high wage for a job that is uncertain or unstable?

The .str accessor and Correlation Plot

You can use the .str accessor with pandas to perform operations on strings. You can access all the methods and functions you use with Python strings. You can answer the following questions with the .str accessor:

# Which major contains the word "Science" with the code:
df.Major[df['Major'].str.contains('Science', case=False)]

# Which major has the longest name:
df.loc[df['Major'].str.len().idxmax(), 'Major']

# Which major ends with "studies"
df[df['Major'].str.endswith('STUDIES')].Major

You can also use pandas package to perform descriptive statistics like skewness, mean, median, and correlation. In the following code, we performed a correlation plot for a subset of the DataFrame:

# get the Pearson correlation matrix
corr_matrix = df.iloc[:, 6:-7].corr()

# get the upper triangle of the correlation matrix
upper_triangle = np.triu(corr_matrix)

plt.figure(figsize=(10, 8))
sns.heatmap(corr_matrix, mask=upper_triangle, annot=True, cmap='YlGnBu', fmt=".2f")
plt.title('Upper Triangular Correlation Heatmap')
plt.show()

Conclusion

This article delved into the capabilities of the pandas library for Python, demonstrating its effectiveness in handling and analyzing data. Through practical examples using the recent-grads csv file, we explored various aspects of data manipulation operations, including filtering and aggregation. Visualization libraries enhanced our understanding of the data, providing insights into salary trends, unemployment rates, and the popularity of college majors.

The article emphasized the importance of considering earnings potential and job security when choosing a major. The provided code snippets enable the techniques to extract meaningful insight from data.

For further practice with the pandas package for data preprocessing, data manipulation methods, data analysis, and data visualization, consider exploring the topics and projects available in our Python tracks: Pandas for Data Analysis and Introduction to Data Science learning tracks.

‍

Related Hyperskill topics

• Introduction to pandas
• Series
• DataFrame
• Combining Data in Pandas
• Accessing data in a DataFrame
• .loc & .iloc
• Reading data in pandas