What is Data Scraping and How to Do It Right

Using Puppeteer library solution for scraping

To do it in the right way, we can use npm package scripts, which look almost the same as the front-end one. For this purpose, we will use Puppeteer, which is a Node.js library which provides a high-level API to control Google Chrome/Chromium over the DevTools Protocol. Puppeteer allows us to execute JavaScript on page level, which makes it very useful for scraper tasks. You can watch a demonstration of the whole approach in this TikTok video.

To have Puppeteer installed and to use it from npm first we need to have a npm project which we start with the command npm init. This command will ask you several questions and create a package.json file. Then, when you have a package.json file, you can install Puppeteer with npm i puppeteer. This will create a record in package.json file like this:

When the project needs to be started on another PC, typing npm install or npm I (the short form) will be enough to provide all the necessary libraries. This is how our NodeJS-based version looks (see node-scrapper.js file).

import puppeteer from 'puppeteer';
import { writeFile } from 'fs/promises';


(async () => {
    const resultFileName = 'result-data.json';
    // Launch the browser and open a new blank page
    const browser = await puppeteer.launch({
        headless: 'new'
    });


    const page = await browser.newPage();


    // Navigate the page to a URL
    await page.goto('https://gonaumov.github.io/what_is_data_scraping/');


    // Wait for cards
    const cardsSelector = 'div.card.mb-3';


    await page.waitForSelector(cardsSelector);


    const dataItems = await page.evaluate((selector) => {
        const data = Array.from(document.querySelectorAll(selector)).map((item) => {
            const name = item.querySelector("div.card-body div.ms-3 > h5").textContent;
            const description = item.querySelector("div.card-body div.ms-3 > p").textContent;
            const qty = item.querySelector("div.card-body h5.fw-normal.mb-0").textContent;
            const price = item.querySelector("div.card-body h5:not(.fw-normal).mb-0").textContent;
            return {
                name,
                description,
                qty,
                price
            }
        });


        return JSON.stringify(data, null, ' ');
    }, cardsSelector);


    try {
        await writeFile(resultFileName, dataItems);
        console.log(`File has been written successfully. Please check ${resultFileName}`);
    } catch (err) {
        console.error('There was an error writing the file.', err);
    }


    await page.close();
    await browser.close();
})();

To be able to run it, you have to specify such a section in the package.json file.

  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1",
    "scrape": "node node-scrapper.js"
  },

In such a way, you will be able to run it with this command: npm run scrape and to ensure the command will be started from installed inside node modules binaries.

With this snippet, we import needed libraries:

import puppeteer from 'puppeteer';
import { writeFile } from 'fs/promises';

This part is

(async () => {
    // our code goes here ... 
})();

An IIFE (Immediately Invoked Function Expression) is a JavaScript function that runs as soon as it is defined. By using it, we avoid polluting the global namespace. With using async, we say that in the function we will use await promises. Promises are a very intriguing approach which JavaScript uses to handle asynchronous logic. You can think about Promises as something similar to eggs. An egg can hatch a chicken, can break and nothing happens and to not be still hatched — waiting state. After waiting, we will have chicken or an exception. So we will start the browser:

 const page = await browser.newPage();

And wait it until launch.

Then we will go to the website:

await page.goto('https://gonaumov.github.io/what_is_data_scraping/');

And wait until the goods appear:

    // Wait for cards
    const cardsSelector = 'div.card.mb-3';


    await page.waitForSelector(cardsSelector);

The interesting part is that Puppeteer has possibility to execute JavaScript on page level and receive the result:

    const dataItems = await page.evaluate((selector) => {
        const data = Array.from(document.querySelectorAll(selector)).map((item) => {
            const name = item.querySelector("div.card-body div.ms-3 > h5").textContent;
            const description = item.querySelector("div.card-body div.ms-3 > p").textContent;
            const qty = item.querySelector("div.card-body h5.fw-normal.mb-0").textContent;
            const price = item.querySelector("div.card-body h5:not(.fw-normal).mb-0").textContent;
            return {
                name,
                description,
                qty,
                price
            }
        });


        return JSON.stringify(data, null, ' ');
    }, cardsSelector);

Inside dataItems constant, we will have the needed data structure in the same way as from the pure front-end part. Then we just write the data in the result-data.json, close the page and close the browser. This is how the script output looks:

Conclusion

 Data scraping tools are powerful resources that allow users to extract valuable information from various websites and sources. The article provided a practical example using Puppeteer, which is a popular tool for web scraping.

The code snippet demonstrated how to scrape data from a website, specifically retrieving product details such as name, description, quantity, and price. It highlighted the use of JavaScript functions like querySelector and evaluate to navigate the page and extract relevant information.

Overall, data scraping tools offer a wide range of possibilities for collecting and analyzing data from various sources. They can be used for tasks such as email address extraction, screen scraping, content aggregation, and competitor analysis. These tools leverage technologies like optical character recognition, pattern recognition engines, and application programming interfaces to gather data efficiently.

However, it is crucial to consider ethical concerns and adhere to privacy policies while using data scraping tools. It is important to respect the terms of service of the websites being scraped and ensure that the process does not infringe upon user privacy or violate any legal regulations.

If you need any clarification, please let me know. 😊

‍

Related Hyperskill topics

• WebScrapper project
• JSON
• Taking user's input
• DOM