ProjectBeta
HackerNews Insights
Challenging
7 completions
~ 27 hours
3.7 This content is new. Please help us improve it by reporting bugs if you encounter them.
Learn automated data collection using APIs and Airflow, integrate financial and open-source data, and apply NLP libraries for analysis.
Provided by
JetBrains Academy
About
In this project, you will develop an automated pipeline to collect, enrich, and analyze Hacker News data, exploring correlations between stock prices, GitHub metrics, MAANG company mentions, and technology trends using Airflow, APIs, PostgreSQL, and natural language processing.
Graduate project
This project covers the core topics of the Data Engineering and ETL course, making it sufficiently challenging to be a proud addition to your portfolio.
At least one graduate project is required to complete the course.
What you'll learn
Once you choose a project, we'll provide you with a study plan that includes all the necessary topics from your course to get it built. Here’s what awaits you:
Build data workflows using Airflow and store results in PostgreSQL database
Connect and fetch data from external APIs
Perform natural language processing tasks on collected data
Reviews
9 months ago
It was a very fun project for learning the basics of Airflow and mixing it with data processing and NLP. For me, this was the longest project on Hyperskill, with a lot of pros and cons. I learned a lot about containerization, unfortunately due to problems with dependencies and similar issues.
12 months ago
I learned to use Airflow to create reusable concurrent functions to perform various NLP tasks within configured tasks.
1 year ago
Some of the elements were enjoyable using the different packages Pandas, csv, requests, spaCy, NLTK and different tech github API, docker etc. However it seems like different people wrote the stages. This project stages were not coherent. Stage 1, it's shown to use Airflow 2.8.0, Stage 3 they show ...
3.7
Learners who completed this project within the Data Engineering and ETL course rated it as follows: