The concept of streams in Node.js is mentioned quite often among developers. Stream is a fundamental and important concept that many core Node.js modules implement. It is difficult to imagine the processing and transferring of large files or images without streams. They are a reliable tool for working with such large amounts of data. Let's explore streams as a part of Node.js!
What are streams?
Streams are objects that allow you to read and write data in small sets of data or chunks. Since data has to be loaded into memory for processing, if you read all the data at once and then try to process it, the computer memory may not be able to store all the data. Working with small chunks of data helps you avoid situations where the memory becomes clogged while processing large files. Also, using streams saves time while processing or transmitting data. Indeed, if you first copied all the data into memory and only then began to write and convert it, then you will spend quite a lot of time. If you started reading the file in chunks (small portions of data) and almost simultaneously writing them to another file, then you will spend much less time.
Types of streams
Streams do not have to read, write, and transform data all at the same time. Different types of streams have varying functions. Let's look at them:
- Readable stream – used for reading data. For example,
fs.createReadStream()helps you read the contents of a file. - Writable stream – used for writing data. For example,
fs.createWriteStream()lets you write data to a file. - Duplex stream – used for both read and write operations. This type of stream allows you to read and write data at the same time, which is more convenient than using the two previous types of stream separately.
- Transform stream – a type of duplex stream that allows you to transform data. If you want to not only read and write files but also process them, then use the transform stream.
All streams are instances of the EventEmitter module. Moreover, streams generate events like data, end, error, pause, and close events.
Application
As mentioned, you may encounter streams in many Node.js modules. Take a look at the following code snippet:
const fs = require("node:fs");
const readStream = fs.createReadStream("./data.txt", {
highWaterMark: 1000
});
readStream.on('data', (chunk) => {
console.log("New chunk!");
console.log(chunk);
});In this example, there is a readable stream called readStream that reads the data.txt file. The highWaterMark parameter sets the size of each chunk that will be read in KiB. The default value of highWaterMark is 64 KiB.
Readable streams emit data event. You can handle this event and use it for various purposes. For example, displaying data to the console. In the code snippet, there is an event handler. The first argument of the event handles is the name of the generated event. The second argument is a callback that receives a chunk and prints it to the console.
Executing the code snippet gives the following output:
By default, the data is represented as a buffer instead of a string. To change the output format to string, apply the toString() method to each chunk.
As you can see, streams allow you to read the file in chunks. These chunks can be processed before the entire file is read which maintains a steady stream of data being read, processed, and written/outputted.
Conclusion
Node.js actively uses the concept of streams, which is why you might encounter them while using different core modules. Streams allow you to work with large files while only processing small parts of the file at a time. Streams also allow you to establish smooth data flow, reducing the time spent on processing them and increasing the convenience of reading, writing, or processing data.