Running several goroutines (WaitGroup, ErrGroup)

Go (or Golang) is a modern, concurrent programming language known for its simplicity and efficiency when it comes to handling concurrency. One of Go's key features is goroutines, which are lightweight threads used to execute functions concurrently.

While creating goroutines is straightforward, managing them and ensuring that they complete their tasks can be more challenging. In this article, we'll explore how to run several goroutines effectively using two important synchronization tools: WaitGroup and ErrGroup.

Using WaitGroup for Synchronization

A sync.WaitGroup is a handy tool provided in Go standard library's sync package for synchronization. This tool is used for waiting for a collection of goroutines to complete their execution before letting the program proceed. In situations where you need to collect outputs from multiple goroutines or perform cleanup tasks after their execution, WaitGroup comes in handy.

WaitGroup provides three main methods for managing goroutines.

Add(int) — This method increases the WaitGroup's counter by the value you specify. You generally call Add before launching a goroutine to let the WaitGroup know that you expect that number of goroutines to complete.

Done() — When a goroutine completes its task, it calls Done to reduce the WaitGroup's counter.

Wait() — The Wait method blocks until the counter in the WaitGroup decreases to zero. It effectively waits for all the goroutines to invoke Done and signals they've completed.

Here's a detailed, brief example of using WaitGroup:

import (
    "fmt"
    "sync" // we need "sync" package to use WaitGroup
)

func main() {
    var wg sync.WaitGroup
    numGoroutines := 3

    for i := 0; i < numGoroutines; i++ {
        wg.Add(1) // Increment the counter for each goroutine

        go func(id int) {
            defer wg.Done() // Decrement the counter when the goroutine completes

            fmt.Printf("Goroutine %d is working\n", id)
            // Your goroutine logic here
        }(i)
    }

    // Wait for all goroutines to finish
    wg.Wait()
    fmt.Println("All goroutines have completed.")
}

In this example, we use a WaitGroup to ensure all three goroutines complete their tasks before moving forward. Here's the breakdown of how it works:

1. We create a WaitGroup variable wg.

2. In a loop, we add each goroutine to the WaitGroup with wg.Add(1).

3. Each goroutine performs its task and then invokes wg.Done() to signal it's finished.

4. The main function waits for all goroutines to finish by invoking wg.Wait(), which will block until the counter decreases to zero.

Common Mistakes with WaitGroup

While sync.WaitGroup is a useful synchronization tool in Go, there are some typical pitfalls and potential issues that you might run into when using it. It's crucial to be aware of these challenges to write reliable concurrent Go programs. Let's discuss a few:

• Unbalanced Add and Done: One of the most frequent errors is forgetting to call Add or Done in your goroutines. If you call Add more times than Done, your program can freeze due to a deadlock or it can exit prematurely.

• Shared Data Access: When multiple goroutines access shared data, you need to employ proper synchronization mechanisms, such as mutexes or channels, to avoid data races. WaitGroups only provide synchronization; they don't grant safe shared data access.

• Calling Add in Goroutines: You should call Add before starting a goroutine to guarantee proper synchronization. Here's a basic example to illustrate:

import (
    "fmt"
    "sync"
)

func main() {
    var wg sync.WaitGroup
    numGoroutines := 3

    for i := 0; i < numGoroutines; i++ {
        // Incorrect: Calling Add after starting goroutine
        go func() {
            wg.Add(1)
            fmt.Printf("Goroutine %d is working\n", i)
            wg.Done() // This is called inside the goroutine
        }()
    }

    // May not wait for all goroutines to finish
    // WaitGroup's internal counter might be already 0
    wg.Wait() 
    fmt.Println("All goroutines have completed.") }

The problem here is that the order of running goroutines is not deterministic, hence, by the time a child goroutine calls Add, the parent goroutine with Wait might have already been unlocked because its counter would be at 0.

• Passing WaitGroup as a value parameter to a function: Another common mistake is to pass WaitGroup by value (for example - func Foo(wg sync.WaitGroup)) and not by pointer (func Foo(wg *sync.WaitGroup)) in a goroutine function. As WaitGroup holds a counter internally, passing it by value doesn't make sense because each function will contain a copy.

Handling Errors with ErrGroup

There are cases when merely waiting for the group to finish isn't enough: you might need to know if there was an error in any of the goroutines. ErrGroup is an extension of the sync.WaitGroup specifically designed for managing goroutines that can potentially return errors. It is part of the "golang.org/x/sync" module, which includes additional synchronization tools for advanced concurrency patterns. Even though it's not part of the standard library, it's widely used and considered a useful tool for creating robust concurrent Go code.

ErrGroup simplifies error handling in concurrent programs by providing a way to gather and handle errors returned by goroutines all the while still letting them run concurrently. This makes it especially useful when you have a collection of goroutines carrying out tasks, and you want to track any errors they encounter.

Here's an example of how to use ErrGroup:

package main

import (
	"fmt"

	"golang.org/x/sync/errgroup" // import the external package to use ErrGroup
)

func main() {
	var eg errgroup.Group

	for i := 0; i < 3; i++ {
		i := i // Create a local copy of i for the goroutine
		eg.Go(func() error {
			// Your goroutine logic here
			if i == 1 {
				return fmt.Errorf("error in goroutine %d", i)
			}
			return nil
		})
	}

	if err := eg.Wait(); err != nil {
		// Handle the first error encountered
		fmt.Printf("An error occurred: %v\n", err)
	}
}

Let's uncover some key features of ErrGroup.

Structured Concurrency: It promotes structured concurrency by making it easy to group and manage goroutines together, ensuring they start and stop as one unit, just like WaitGroup.

Error Propagation: ErrGroup can propagate errors from goroutines back to the calling code. When one or more goroutines return an error, ErrGroup gathers and returns the first error encountered. This allows for graceful error handling.

Cancellation (with Contexts): ErrGroup provides a way for graceful cancellation of all goroutines as soon as an error is detected. This is useful when you want to stop concurrent tasks the moment a problem arises. To introduce cancellation, you should create an ErrGroup with the function WithContext. When a goroutine from the group returns with an error, the context will be canceled.

Limiting the Number of Active Goroutines: Sometimes controlling the number of resources your program is using becomes important. If too many goroutines are started, you might run out of memory or other resources. To limit the number of active goroutines, you need to call the SetLimit(n int) method. This ensures that any future calls to the Go() method will block until an active goroutine finishes its task.

ErrGroup vs WaitGroup

The choice between ErrGroup and WaitGroup will depend on the specific needs and complexity of your concurrent program. In many cases, you may use both, with WaitGroup used for basic synchronization and ErrGroup for detailed error handling and task cancellation. This allows you to balance simplicity and sturdiness in your Go concurrent applications.

Let's compare these synchronization tools:

In the end, the choice between ErrGroup and WaitGroup depends on the specific needs and complexity of your concurrent program.

Conclusion

Go's goroutines allow you to carry out tasks concurrently, and with the help of WaitGroup and ErrGroup, you can manage and synchronize them effectively. WaitGroup ensures all goroutines complete their tasks before moving forward, while ErrGroup allows for graceful error handling. These tools are key in writing sturdy, concurrent Go programs.

Now that you understand how to run numerous goroutines using WaitGroup and ErrGroup, you can fully utilize Go's concurrency capabilities to create efficient and highly concurrent applications.