The Shutil module

We already know how to interact with the operating system. In this topic, we will learn some of the high-level operations: copying and removing files and directories with the help of the shutil module. You can easily automate file operations without diving into low-level semantics, such as creating file objects or closing files.

To use the shutil module, you will need to import it first:

import shutil

Copying

Every day we copy, delete, rename, or move various files and folders. If we want to copy our file in the current directory, we can use the shutil.copyfile() function:

import shutil
import os

file_name = "news.txt"
copy_file_name = "news_copy.txt"

print("Before operation:", os.listdir("."))  # '.' is the current directory
shutil.copyfile(file_name, copy_file_name)
print("After operation:", os.listdir("."))

# Before the operation: ['news.txt']
# After the operation: ['news.txt', 'news_copy.txt']

This function copies the contents of the source file to the destination file. It takes two parameters: a source file and a destination one. In the code above, we use the os.listdir() function to show the current directory content.

Another useful function is shutil.copy() that allows copying files into another directory:

import shutil
import os

file_path = "/home/user/existence.pdf"
dst_folder = "/home/user/Articles"  # imagine that the folder contains 'innovative_ideas.pdf'

print(f"Before operation, Articles:", os.listdir(dst_folder))
shutil.copy(file_path, dst_folder)
print(f"After operation, Articles:", os.listdir(dst_folder))

# Before operation, Articles: ['innovative_ideas.pdf']
# After operation, Articles: ['existence.pdf', 'innovative_ideas.pdf']

The shutil.copy() function takes two arguments and copies the file with the same permissions. If the file is read-only, it will have the same permission in the target folder. However, the metadata of files, such as the time of creation, accessing, and modification, will not be preserved.

Copying file with metadata

When you use functions described in the section above, they create a new file with the same contents. Sometimes, however, you may need to clone a file with all metadata. In this case, you can use shutil.copy2().

import shutil
import os
import time

def show_metadata(file_path):
    stat = os.stat(file_path)  # returns the status of the file
    print("Mode    :", oct(stat.st_mode)) 
    print("Created :", time.ctime(stat.st_ctime))
    print("Accessed:", time.ctime(stat.st_atime))
    print("Modified:", time.ctime(stat.st_mtime))

file_path = "/home/user/existence.pdf"
dst_folder = "/home/user/Articles"

print("Source File:")
show_metadata(file_path)
dst_path = shutil.copy2(file_path, dst_folder)
print("\nDestination File:")
show_metadata(dst_path)

# Source File:
# Mode    : 0o100777
# Created : Sun Sep 12 13:57:22 2021
# Accessed: Sun Sep 12 13:57:22 2021
# Modified: Sun Sep 12 13:57:22 2021

# Destination File:
# Mode    : 0o100777
# Created : Mon Sep 13 00:51:21 2021
# Accessed: Sun Sep 12 13:57:22 2021
# Modified: Sun Sep 12 13:57:22 2021

As we can see in the example, the shutil.copy2() function takes the same arguments as shutil.copy() and copies the metadata from the source destination. In the output, the Mode, Accessed and Modified lines remain the same, but the Created line differs from the source file. This variation may occur as a result of different operating systems. For example, on Windows, the file owners and ACLs (permission list of file or directory) are not copied, as they may be private.

Copying and removing a directory

The shutil.copytree() function allows copying a complete directory to the destination directory recursively:

import shutil
import os

src_path = "/home/user/Articles"

src_dir = "/home/user/Articles/python_codes"
dst_dir = "/home/user/Articles/codes"

print("Before Copying:", os.listdir(src_path))

destination = shutil.copytree(src_dir, dst_dir)
print("After Copying:", os.listdir(src_path))
print("Destination path:", destination)

# Before Copying: ['python_codes']
# After Copying: ['codes', 'python_codes']
# Destination path: /home/user/Articles/codes

This function, again, takes two arguments — the source and destination directory. The destination directory should not exist; otherwise, the FileExistsError exception will be raised. We can also specify the copy_function argument. By default, it is the shutil.copy2() function.

If you want to delete an entire directory, you can easily select the shutil.rmtree() function.

shutil.rmtree("/home/user/Articles")

It takes a path as an argument. If the path does not exist, FileNotFoundError will be raised.

Moving and finding files

It is easy to move the file or directory from one place to another with shutil.move(). This method takes the source and destination paths as the parameters and returns the absolute path of the new location.

If the source is a file, and the destination is a directory, it moves the file and retains its name:

import shutil

src_path = "/home/user/Desktop/payment.txt"
dest_path = "/home/user/Desktop/payments"

dest = shutil.move(src_path, dest_path)

print(dest)
# /home/user/Desktop/payments/payment.txt

You can move the file to a different directory and change the filename:

import shutil

src_path = "/home/user/Desktop/number.txt"
dst_path = "/home/user/Desktop/math/new_number.txt"

dest = shutil.move(src_path, dst_path)
print(dest)
# /home/user/Desktop/math/new_number.txt

Additionally, it is easy to move an entire directory from one place to another:

import shutil

src_dir = "/home/user/Desktop/cars"
dst_dir = "/home/user/Desktop/vehicles"

dest = shutil.move(src_dir, dst_dir)

print(dest)
# /home/user/Desktop/vehicles/cars

If the destination path already exists, shutil.Error will be raised.

Another helpful function is shutil.which(). It finds the path to the provided executable application like Python3 or Java:

import shutil

path_python = shutil.which("python3")
path_java = shutil.which("java")

print(path_python, "and", path_java)
# /usr/bin/python3 and /usr/bin/java

The shutil.which() takes the executable file as a parameter. If it can't find the location, it will return None.

Making an archive

In this section, we will take a look at how to archive files in a specific format. First, we need to know the available formats. For this, we will use shutil.get_archive_formats():

import shutil

formats = shutil.get_archive_formats()
for format in formats:
    print(format)

# ('bztar', "bzip2'ed tar-file")
# ('gztar', "gzip'ed tar-file")
# ('tar', 'uncompressed tar file')
# ('xztar', "xz'ed tar-file")
# ('zip', 'ZIP file')

As we can see, the available archiving methods include bztar, gztar, tar, xztar, and zip. There is a way to register a new archive format with the shutil.register_archive_format(). The procedure is described in the Official Documentation in detail.

Now, we can perform the archiving with shutil.make_archive():

import shutil

directory = "/home/user/Desktop/Algorithms"
arc_name = shutil.make_archive("compressed_algo", "bztar", directory)

print(arc_name)
# /home/user/Desktop/compressed_algo.tar.bz2

It takes the name of the file, archive format, and directory as arguments. With this simple action, we have compressed a file in the compressed_algo file.

Conclusion

In this topic, we have learned about high-level file and directory operations with the shutil module. When using this, we don't need to perform operations such as opening, reading, and closing files. Let's take a brief look at the operations we have discussed:

• shutil.copyfile allows us to copy files in the current directory;

• shutil.copy copies the contents of the file to the destination file or directory;

• shutil.copy2 is similar to shutil.copy. It preserves the metadata;

• shutil.copytree copies an entire directory to the destination directory;

• shutil.rmtree deletes a directory;

• shutil.move easily moves a file or directory to the destination;

• shutil.which finds the path to the executable file;

• shutil.make_archive allows us to compress the files in the given format.

For additional information, you can check the Official Shutil Documentation.