Getting data from net

One of the common tasks for any computer user is to get some data from the web. Maybe you want to download an image, video, or even some software. Whilst you can do this from your web browser, it is also entirely possible to download files of all kinds directly from the command line. In some cases, that might even be the preferred way that is faster than actually firing up a web browser. In this topic, we'll practice with the two popular utilities for this purpose: wget and curl.

Wget

One of the most commonly used tools to get files from the internet is wget. It is a tiny tool that comes pre-installed on most Linux distributions. To get a file from the net you just need to run a command with the URL you need, and it will download the file to your current directory:

wget https://www.example.com/

The wget comes with many additional features to customize your request. For example, if you suspect that a first attempt may not be successful, you can state how many times the utility should try to access the resource:

wget --tries=5 https://www.example.com/

Another useful feature is to choose the path where you want to save a file. You can do this with the -O or --output-document flags:

wget -O /tmp/index.html https://www.example.com/

If you need a more detailed instruction on wget, we recommend you to check out its man page by typing man wget. If you don't remember a particular command or if you've just got started with the utility, you can use the integrated help function to get some important information. To call it, try the following:

$ wget --help

GNU Wget 1.21.1, a non-interactive network retriever.
Usage: wget [OPTION]... [URL]...
...

The above list does not only show the current version of wget and the usage, but it also shows all of the commands and options that can be used with wget.

Curl

Another tool you can use to work with the files and web pages is curl. One of the noticeable differences between curl and wget is that with curl you can also upload files.

You can check if curl is installed on your system using the following console command:

curl --version

If curl is installed, you will get a version number and additional information, if not, you will get an error. In this case, you have to download curl with a package manager.

To get pages with curl just use the command and the source URL.

curl https://www.example.com/

If you want to save the page to a file, state the path to store it after the -o flag:

curl -o /tmp/index.html https://www.example.com/

Sometimes, in case of errors, you may want to retry the request several times, before it finally fails:

$ curl --retry 3 https://www.example.what

(6) Could not resolve host: www.example.what
Warning: Problem : timeout. Will retry in 1 seconds. 3 retries left.
...

Of course, there are many more flags and options that you can use. To get a list of the most common curl command options you can use the built-in help:

curl --help

Most flags and options work for both uploads and downloads. If you want to get more detailed information on curl and how it works, you can use the curl man page, which is installed automatically with the curl package.

Pros and cons of wget and curl

One question that might come up is, why would you want to use the command line to download files or packages? There are several pros and cons when it comes to it. The most notable advantages are:

• You can get access to some quite advanced features, not available otherwise
• With wget and curl you can check the file integrity and trustworthiness by verifying the checksum
• You can also automatically set the amount of time you are ready to wait or how many times you want to retry the request

And the downsides of using tools like curl and wget:

• They require you to already know the URL from where you want to download the file
• It takes time to practice knowing them well

There are many options with which you can run these tools, but which way you choose and whether it is worth it is only up to you.

Conclusion

That's all there is to know about the basics of curl and wget, the most commonly used tools to get files from the internet. While in most cases you can achieve the same goals with a browser, there might be some advantages in using the command line for these tasks.

In this topic we've only briefly introduced you to curl and wget. Now you should know:

• How to download a file from the command line
• How to customize your request with options and flags
• How to get help for proper usage of these tools.