Project
Web Scraper
Medium
1492 completions
~ 15 hours
4.3Master web scraping and data extraction by building a robust Python application that interacts with APIs and parses dynamic web content!
Provided by
JetBrains Academy
About
You will create a function that takes a website address and a number of webpages as input arguments and then goes all over the website saving every news article on the page to a separate .txt file on your computer.
Training project
This project allows you to practice and strengthen your coding skills, helping you get ready for more advanced tasks ahead.
What you'll learn
Once you choose a project, we'll provide you with a study plan that includes all the necessary topics from your course to get it built. Here’s what awaits you:
Send an HTTP request, process the results, and learn about response status codes.
Get familiar with the BeautifulSoup library. Learn how to parse simple data from a webpage.
Learn to work with files.
Create a function that scrapes every article on a page with a for loop.
Improve the function by adding ‘webpage number’ and 'article type' parameters.
Reviews
2 months ago
Interesting project, learned about HTML web structure and python lib's requests.
Ludoviko Azuaje
3 months ago
This has been an interesting project on scrapping web pages, it provides good insights for working with http requests and saving concrete pages locally depending on html meta data. The stages are well thought and based one on another, highly recommended project.
Daniel Lopez
3 months ago
This was a deceptively simple program, a few short lines of code that could scrape much of the internet.
4.3
Learners who completed this project within the Python Developer course rated it as follows: