Serializing structured YAML

YAML is a human-readable data serialization format. It is used for writing configuration files and data exchange between languages with different data structures.

The gopkg.in/yaml.v3 package is a YAML parsing and encoding library for the Go programming language. It provides functionality for working with YAML data, such as encoding Go data structures into YAML object and decoding YAML objects into the corresponding Go data structures.

In this topic, you will learn how to encode a struct to YAML format and how to decode a YAML object to a struct in Go using the package gopkg.in/yaml.v3.

How to Install the YAML package

To integrate YAML functionality into your Go project, you will need to initialize Go modules and install the gopkg.in/yaml package.

Let’s create a directory named example-yaml move into it, and then initialize Go modules:

mkdir example-yaml && cd example-yaml
go mod init example-yaml

Then, use the go get command to install the external package gopkg.in/yaml.v3:

go get gopkg.in/yaml.v3

Finally, create a main.go file, open it on your favorite code editor, and you’re ready to start!

Mapping between Go Types and Yaml Types

YAML supports various data types, which includes strings, numbers, booleans, arrays, and objects. When using the gopkg.in/yaml.v3 package in Go, it is important to understand the mapping between Go data types and YAML data types for encoding and decoding operations. The following table summarizes these mappings, showing how different Go types transform into YAML data types during encoding and vice versa during the decoding process:

Serializing a struct to YAML

Now that you have learned about the mapping between Go and YAML data types, let's move forward with a hands-on example. Suppose you want to serialize a simple struct Movie; to achieve this, you simply initialize its fields and then use the yaml.Marshal() function as follows:

package main

import (
    "fmt"
    "log"
    "time"

    "gopkg.in/yaml.v3"
)

type Movie struct {
    Title         string
    Genres        []string
    Year          int
    movieLength   int32 // in minutes
    Rating        float32
    CreatedAt time.Time
}

func main() {
    movie := Movie{
        Title:       "Titanic",
        Genres:      []string{"drama", "romance"},
        Year:        1997,
        movieLength: 197,
        Rating:      7.9,
        CreatedAt:   time.Now(),
    }
    // Serializing the `movie` struct into a YAML object
    // `yaml.Marshal()` returns a slice of bytes and an error:
    movieYAML, err := yaml.Marshal(movie)
    if err != nil {
        log.Fatal(err)
    }
    // Remember to "cast" the returned slice of bytes as a 'string' to properly print it:
    fmt.Println(string(movieYAML))
}

After running the above program, you'll get the following output:

title: Titanic
genres:
    - drama
    - romance
year: 1997
rating: 7.9
createdat: 2024-02-03T15:33:34.50642821Z # RFC3339 string format for time

After the serialization process, we notice that Rating(type=float32) and Year(type=int) fields are transformed into YAML numbers, while Title(type=string) becomes YAML string and Genres(type=[]string) transforms into a YAML array of strings.

It is important to note that YAML lacks inherent support for representing time. However, the gopkg.in/yaml.v3 package addresses this by converting time.Time values into string representations formatted according to the RFC3339 standard during the encoding process. As a result CreatedAt(type time.Time) field is transformed into a string in YAML, adhering to the RFC3339 format.

Custom serialization

In the previous section, after serializing the Movie struct all of the keys of the resulting YAML object were in lower case format. While this is valid YAML, it deviates from the snake_case convention. If snake_case is the preferred naming style, manual adjustments would be necessary during the struct-to-YAML encoding process to ensure keys follow the snake_case pattern; you can achieve this by simply using the struct tags to customize the struct-to-YAML encoding.

package main

import (
	"fmt"
	"log"
	"time"

	"gopkg.in/yaml.v3"
)

// Observe the inclusion of yaml:"<key-name>" tags next to each field in the following struct. These tags are utilized to instruct the YAML package to customize the keys in the resulting YAML object.
type Movie struct {
	Title     string    `yaml:"title"`
	Genres    []string  `yaml:"genres"`
	Year      int       `yaml:"year"`
	Runtime   int32     `yaml:"movie_length_in_minutes"`
	Rating    float32   `yaml:"rating"`
	CreatedAt time.Time `yaml:"created_at"`
}

func main() {
	movie := Movie{
		Title:     "Titanic",
		Genres:    []string{"drama", "romance"},
		Year:      1997,
		Runtime:   197,
		Rating:    7.9,
		CreatedAt: time.Now(),
	}

	movieYAML, err := yaml.Marshal(movie)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(string(movieYAML))
}

Now, let's try running the program again and check the YAML output:

title: Titanic
genres:
    - drama
    - romance
year: 1997
movie_length_in_minutes: 197
rating: 7.9
created_at: 2024-02-03T15:33:34.50642821Z

As you can see, the Runtime field now gets serialized as movie_length_in_minutes, and the CreatedAt field gets serialized as created_at after applying the struct tags. Customizing or modifying key names is often necessary when sharing data across multiple services. It is Important to adopt a unified naming convention like snake_case in order to maintain consistency across various programming languages used in different services.

Optional struct tag directives

Let's explore optional tags that offer additional flexibility when encoding structs into YAML objects or decoding YAML into structs.

At times, it becomes necessary to conceal specific fields within a struct, either because they are irrelevant to users or contain sensitive information, such as a hashed password stored in the database instead of the plaintext version provided by the user. While making the field private achieves a similar outcome by restricting access, there are situations where fields need to be exported or made public for internal packages that may require access. In such cases, the - tag provides this flexibility:

package main

import (
    "fmt"
    "log"
    "time"

    "gopkg.in/yaml.v3"
)

type User struct {
    ID        int64     `yaml:"id"`
    CreatedAt time.Time `yaml:"created_at"`
    Name      string    `yaml:"name"`
    Email     string    `yaml:"email"`
    Password  string    `yaml:"-"` // Note the usage of the `-` tag
}

func main() {
    user := User{
        ID:        6001,
        Name:      "John Doe",
        Email:     "[email protected]",
        Password:  "very secret",
        CreatedAt: time.Now(),
    }

    userYAML, err := yaml.Marshal(user)
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println(string(userYAML))
}

After running the above program, the output shows that the Password field has been excluded during the encoding into YAML. Consequently, the resulting YAML object also lacks the password field. Below is the output we get from running the above program:

After running the above program, you'll get the following output:

id: 6001
name: John Doe
email: [email protected]
created_at: 2024-02-06T11:07:00.000402159Z

As expected, the Password field marked with a - tag has been omitted from the YAML serialization process; this deliberate exclusion ensures that sensitive information, such as passwords, does not appear in the serialized YAML object.

The inline directive

Let's now move to the next optional directive, called inline. It is used for embedding nested struct fields directly into the parent struct. This directive is useful when you want to flatten the structure and avoid unnecessary nesting in the serialized output. It also comes in handy when the embedded struct is small or does not add semantic value to the structure to make it more concise and readable. Let's see an example:

package main

import (
    "fmt"
    "log"

    "gopkg.in/yaml.v3"
)

type Address struct {
    City    string `yaml:"city"`
    Country string `yaml:"country"`
}

// Observe how we use the 'inline' directive for the 'Address' field
type Person1 struct {
    Name    string `yaml:"name"`
    Age     int    `yaml:"age"`
    Address `yaml:",inline"`
}

// Observe that for the following struct we do not use the 'inline' directive 
type Person2 struct {
    Name string `yaml:"name"`
    Age  int    `yaml:"age"`
    Address
}

func main() {
    addr := Address{
        City:    "New York",
        Country: "USA",
    }

    person1 := Person1{
        Name:    "John Doe",
        Age:     39,
        Address: addr,
    }

    person2 := Person2{
        Name:    "John Doe",
        Age:     39,
        Address: addr,
    }

    // encoding `person1`:
    person1YAML, err := yaml.Marshal(person1)
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println(string(person1YAML))

    // encoding `person2`:
    person2YAML, err := yaml.Marshal(person2)
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println(string(person2YAML))
}

After running the above program, you'll see different serialization outputs for the person1YAML and person2YAML due to the inline directive:

For person1YAML, with the inline directive applied to the Address struct, the output merges City and Country directly under Person1, eliminating the nested structure. This flattens the hierarchy, making the output more compact and readable.

In contrast, person2YAML lacks the inline directive, resulting in City and Country being serialized within a nested Address structure, maintaining the hierarchical structure.

The omitempty directive

Finally, we have the omitempty struct tag directive; this tag is used to omit a struct field when it contains a zero or a default value. It makes it easier to handle schema changes. If new fields are added to a struct but they are not applicable to existing data, using omitempty ensures that these fields are not present in the serialized output by default. Below is an example of how to use it:

package main

import (
    "fmt"
    "log"
    "time"

    "gopkg.in/yaml.v3"
)

type Movie1 struct {
    Title     string    `yaml:"title"`
    Genres    []string  `yaml:"genres"`
    Year      int       `yaml:"year"`
    Runtime   int32     `yaml:"movie_length_in_minutes"`
    Rating    float32   `yaml:"rating"`
    CreatedAt time.Time `yaml:"created_at,omitempty"` // Note the `omitempty` tag
}

type Movie2 struct {
    Title     string    `yaml:"title"`
    Genres    []string  `yaml:"genres"`
    Year      int       `yaml:"year"`
    Runtime   int32     `yaml:"movie_length_in_minutes"`
    Rating    float32   `yaml:"rating"`
    CreatedAt time.Time `yaml:"created_at"` // Note that no `omitempty` tag is used
}

func main() {
    movie1 := Movie1{
        Title:   "Titanic",
        Genres:  []string{"drama", "romance"},
        Year:    1997,
        Runtime: 197,
        Rating:  7.9,
    }

    movie2 := Movie2{
        Title:   "Titanic",
        Genres:  []string{"drama", "romance"},
        Year:    1997,
        Runtime: 197,
        Rating:  7.9,
    }

    // serializing the `movie1` struct into a YAML object:
    movie1YAML, err := yaml.Marshal(movie1)
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println(string(movie1YAML))

    // serializing the `movie2` struct into a YAML object:
    movie2YAML, err := yaml.Marshal(movie2)
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println(string(movie2YAML))
}

After running the above program, you'll get the following output:

Note that when the omitempty tag is not used, the encoder includes the field with its current value, even if that value is the zero value for the respective data type. It does not create a default value for the field before encoding.

Deserializing YAML to a struct

YAML objects can be decoded into either Go structs or map[string]interface{}. When you use map[string]interface{} in Go for decoding YAML or handling dynamic data structures, you often need to use reflection to inspect and work with the actual types of the values stored in the map.

The empty interface{} type is a way to represent values of any type, and reflection provides a mechanism to inspect and manipulate these values at runtime. Using reflection comes with some trade-offs, such as decreased type safety and potentially slower performance. Decoding YAML objects into map[string]interface{} is used only in case we are uncertain of the structure or the type of values, a certain YAML object may contain. When a YAML object's structure is known beforehand, it is better to decode it into a well-defined struct rather than using map[string]interface{} in Go.

Suppose we aim to deserialize a YAML object into a struct, and we are aware of the YAML object's structure beforehand. Achieving this is straightforward by using the yaml.Unmarshal() function.

Deserializing A YAML object to a struct in Go involves the following two steps:

1. Create an appropriate struct type that aligns with the structure of the YAML object you want to decode; this struct should be designed to hold the values in a way that matches the structure of the data.

2. Use the yaml.Unmarshal() function, which takes the YAML object as a slice of bytes and provides a reference to the variable (of the struct type) where you want to decode the YAML object.

Let's see an example of deserializing a moderately complex YAML object:

package main

import (
	"fmt"
	"log"

	"gopkg.in/yaml.v3"
)

type EmailServer struct {
	Host string `yaml:"host"`
	Port int    `yaml:"port"`
}

type EmailClient struct {
	Email    string `yaml:"email"`
	Password string `yaml:"password"`
	SendTo   string `yaml:"send_to"`
}

type Config struct {
	Server EmailServer `yaml:"server"`
	Client EmailClient `yaml:"client"`
}

func main() {
	yamlData := `
server:
  host: "127.0.0.1"
  port: 2500
client:
  email: "[email protected]"
  password: "123456"
  send_to: "[email protected]"
`
	var config Config
	err := yaml.Unmarshal([]byte(yamlData), &config)
	if err != nil {
		log.Fatalf("error: %v", err)
	}

	fmt.Printf("Server: %+v\n", config.Server)
	fmt.Printf("Client: %+v\n", config.Client)
}

// Output:
// Server: {Host:127.0.0.1 Port:2500}
// Client: {Email:[email protected] Password:123456 SendTo:[email protected]}

Conclusion

In this topic, you learned the process of encoding a Go struct into YAML format and decoding a YAML object back into a struct using the gopkg.in/yaml.v3 package.

You also explored some commonly used struct tag directives: omitempty, -, and inline. Throughout this exploration, you gained an understanding of the functionalities these directives provide and their respective use cases.

To sum up, decoding a YAML object into a Go struct is a straightforward process. It involves defining a struct type that mirrors the YAML object's structure and then using the yaml.Unmarshal() function for deserialization.

This was a lengthy topic, but there is still more to do! Let's test your newly acquired knowledge about working with YAML in Go with some comprehension and coding tasks!