Computer scienceSystem administration and DevOpsKubernetesRunning applications on Kubernetes

Running replicated workloads on Kubernetes

17 minutes read

So far, we have been running individual Pods. However, Pods are ephemeral, meaning they can fail or get evicted at any time for various reasons. By default, Kubernetes does not automatically restart failed Pods. The cluster administrator or controllers must maintain a desired state in response to failures. In this topic, we look at how these controllers can help you run highly available workloads in Kubernetes.

ReplicaSets

A ReplicaSet is a Kubernetes object designed to ensure that a specified number of Pod instances run at any given time. Running multiple replicas of a Pod has several benefits:

  • If one Pod fails, others can quickly pick up the workload to ensure uninterrupted operation.

  • You can process more requests if you have more instances of your application running.

  • You can load-balance the overall workload among several Pods to avoid overloading any single Pod.

Without ReplicaSets, you'd need to create Pod manifests for every desired replica, which is both tedious and error-prone. Also, you'd have to monitor your cluster to recreate Pods if they fail continuously. With ReplicaSets, Kubernetes automatically maintains the desired number of replicas, replacing any that fail.

To create a ReplicaSet, you create its definition in a YAML file. Because a ReplicaSet belongs to the Apps API group, the apiVersion is apps/v1. The resource type is ReplicaSet. You also need to provide some metadata, like the name. Minimally, your spec section should define the number of desired replicas, a selector to identify Pods managed by the ReplicaSet (using label selectors), and a template defining a spec for creating Pods. Notice that the template section is identical to a Pod's manifest without the apiVersion, kind, and name fields:

spec: 
  replicas: 3
  
  selector: 
    matchLabels:
      app: web-server
      
  template: 
    metadata:
      labels:
        app: web-server
    spec:
      containers: 
        - image: nginx:latest
          name: dev-nginx
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8080
              protocol: TCP
      restartPolicy: Always
      dnsPolicy: ClusterFirst

The selector filters objects that match specific labels. Run the following commands to try it yourself:

$ kubectl get pods --show-labels -n hyper # this allows you to see labels for Pods
# NAME           READY   STATUS    RESTARTS      AGE    LABELS
# dev-rs-c6gl5   1/1     Running   0          24s   app=web-server
# dev-rs-f2hwj   1/1     Running   0          24s   app=web-server
# dev-rs-wsqcr   1/1     Running   0          24s   app=web-server

$ kubectl get pods --selector app=web-server # this is similar to how a ReplicaSet finds its Pods
# NAME           READY   STATUS    RESTARTS   AGE   LABELS
# dev-rs-c6gl5   1/1     Running   0          24s   app=web-server
# dev-rs-f2hwj   1/1     Running   0          24s   app=web-server
# dev-rs-wsqcr   1/1     Running   0          24s   app=web-server

In our ReplicaSet manifest, we used matchLabels, which is a map of key-value pairs. You can also use matchExpressions, which uses a more flexible, operator-based syntax. You can use operators like In, NotIn, Exists, and DoesNotExist, instead of requiring an exact key-value match:

selector:
  matchExpressions:
    - { key: app, operator: In, values: [web-server] }

The ReplicaSet uses the template defined in the .spec.template section to create Pods. You can deploy the template with the kubectl apply command. Once deployed, you can then scale the ReplicaSet by editing the .spec.replicas section of the YAML manifest or with the imperative scale command:

$ kubectl get replicasets
# NAME                          DESIRED   CURRENT   READY   AGE
# dev-rs                        3         3         3       14s

$ kubectl scale rs dev-rs --replicas 7
# NAME                          DESIRED   CURRENT   READY   AGE
# dev-rs                        7         7         7       114s

To delete the ReplicaSet, simply use the kubectl delete command and supply the YAML file you used to create it.

DaemonSets

In the previous section, we used a ReplicaSet to run multiple copies of a Pod. This is useful when we don't care about the node our Pod lands on. In some cases, however, you want to ensure that a copy of a Pod runs on each node (or a subset of nodes). This is useful for tasks like log collection daemons and monitoring agents.

To create a DaemonSet, you need to define it in a YAML file similar to other Kubernetes objects. The apiVersion is apps/v1 and the kind is DaemonSet. You also need to define some metadata. The spec section mirrors that of a ReplicaSet, and you can configure additional fields:

spec:
  selector:
    matchLabels:
      app: node-exporter
      
  template:
    metadata:
      labels:
        app: node-exporter
        tier: monitoring
        
    spec:
      hostNetwork: true
      hostPID: true
      containers:
        - name: node-exporter
          image: prom/node-exporter:latest
          args:
            - --path.procfs=/host/proc
            - --path.sysfs=/host/sys
            - --path.rootfs=/host/root
            
          ports:
            - containerPort: 9100
              name: metrics
          volumeMounts:
            - name: proc
              mountPath: /host/proc
              readOnly: true
  
            - name: sys
              mountPath: /host/sys
              readOnly: true
              
            - name: root
              mountPath: /host/root
              readOnly: true
              
      volumes:
        - name: proc
          hostPath:
            path: /proc
            
        - name: sys
          hostPath:
            path: /sys
            
        - name: root
          hostPath:
            path: /

This configuration ensures that a copy of the node exporter agent deploys on each node:

$ kubectl get daemonsets

# NAME            DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
# node-exporter   3         3         3       3            3           <none>          102s

$ kubectl get pods -l app=node-exporter -o wide
# NAME                  READY   STATUS    RESTARTS   AGE     IP             NODE           NOMINATED NODE   READINESS GATES
# node-exporter-6tpzw   1/1     Running   0          7m36s   192.168.49.2   node-001       <none>           <none>
# node-exporter-j6j6w   1/1     Running   0          7m36s   192.168.49.3   node-002       <none>           <none>
# node-exporter-s6mc4   1/1     Running   0          7m36s   192.168.49.4   node-003       <none>           <none>

If we don't want to run a copy of the Pod on each node, we can use a node selector to limit eligible nodes to those with a particular label. But first, we need to add a label to a node:

$ kubectl label nodes <node_name> server=true
# output node/<node_name> labeled

Then, in our DaemonSet's .spec.template.spec, we add a node selector:

template:
  metadata:
    labels: 
      name: fluentd
  spec: 
    nodeSelector:
      server: "true"
    containers: 
      - name: fluentd
        image: fluent/fluentd:latest

Now, only nodes with this label will run a copy of the Pod:

$ kubectl get pods -l name=fluentd -o wide
                                                                                 
# NAME            READY   STATUS    RESTARTS   AGE   IP           NODE           NOMINATED NODE   READINESS GATES
# fluentd-bhpvs   1/1     Running   0          62s   10.244.2.2   node-003        <none>           <none>
# fluentd-v9sq7   1/1     Running   0          62s   10.244.1.2   node-002        <none>           <none>

To delete the DaemonSet, simply use the kubectl delete command and supply the YAML file you used to create it.

Deployments

We've seen how to run replicated sets of your application, including scenarios where a copy must run on all or a subset of nodes. In addition to the awesome features that these objects provide, Kubernetes helps you deploy applications in a more effective manner. The Deployment object makes this possible.

A Deployment manages the lifecycle of your applications in Kubernetes. It ensures that you can easily update your application while keeping it highly available. It does this by upgrading, or "rolling out", individual Pods in a controlled way. It also uses health checks to ensure the new version works as expected, avoiding any downtime. If anything goes wrong, the deployment will "rollback" to the previous version.

Like other Kubernetes objects, you need a YAML definition to create a Deployment. At a glance, it looks similar to a ReplicaSet, except that the kind is Deployment. However, just as a ReplicaSet manages Pods, a Deployment manages ReplicaSets.

Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hyperskill-clone
  namespace: hyper
  labels:
    app: hyper-app

  annotations:
    environment: "Production"

spec:
  replicas: 3

  selector:
    matchLabels:
      app: hyper-app

  template:
    metadata:
      labels:
        app: hyper-app
    spec:
      initContainers:
        - name: busybox
          image: busybox:latest
          imagePullPolicy: Always
          command: ["sh", "-c", "echo Initializing... && sleep 5"]
        - name: log-shipper
          image: fluent/fluent-bit:latest
          restartPolicy: Always

      containers:
        - name: frontend
          image: nginx:latest
          imagePullPolicy: Always
          ports:
            - containerPort: 80
              name: http
              protocol: TCP

          resources:
            requests:
              memory: "128Mi"
              cpu: "100m"
              ephemeral-storage: "1Gi"

        - name: hyperskill-backend
          image: initram/hyperskill-clone:latest
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8000
              protocol: TCP
              name: hyperskill
          env:
            - name: log-level
              value: "info"

          readinessProbe:
            httpGet:
              port: 8000
              path: /ready
            initialDelaySeconds: 5
            timeoutSeconds: 3
            failureThreshold: 2

          livenessProbe:
            httpGet:
              port: 8000
              path: /health
            failureThreshold: 3
            timeoutSeconds: 5
            periodSeconds: 10

          volumeMounts:
            - mountPath: /tmp/shared
              name: cache

          resources:
            requests:
              memory: "128Mi"
              cpu: "100m"

      volumes:
        - name: cache
          emptyDir:
            {}

      resources:
        requests:
          memory: "1Gi"
          cpu: "2.0"

        limits:
          memory: "1Gi"
          cpu: "3.0"

If we run the Deployment and check the ReplicaSet's details, we see that the Deployment manages it:

$ kubectl get deployments --show-labels
# NAME               READY   UP-TO-DATE   AVAILABLE   AGE     LABELS
# hyperskill-clone   3/3     3            3           2m57s   app=hyper-app

$ kubectl get replicasets
# NAME                          DESIRED   CURRENT   READY   AGE
# hyperskill-clone-668dfbdffc   3         3         3       3m12s 

$ kubectl get rs hyperskill-clone-668dfbdffc -o jsonpath='{.metadata.ownerReferences[0].kind}/{.metadata.ownerReferences[0].name}'
# Deployment/hyperskill-clone

To scale, you must scale the Deployment. If you scale the ReplicaSet that the Deployment manages, Kubernetes will create or delete new Pods immediately to maintain the desired state of the Deployment.

$ kubectl scale deployments hyperskill-clone --replicas 4

Both the Deployment and DaemonSet objects support a field that specifies how to release new versions. Let's take a closer look at them.

Update strategies

A Deployment supports two update strategies: Recreate and RollingUpdate. A DaemonSet does not support the Recreate strategy, but has OnDelete and RollingUpdate strategies. The Recreate strategy creates a new ReplicaSet that creates new Pods. Unfortunately, this means that there is a period of unavailability during the update as the system takes all Pods down first. Therefore, you should only use this strategy when you expect downtime or when specific application requirements require it.

spec: 
  strategy:
    type: Recreate
spec: 
  updateStrategy:
    type: OnDelete

With the OnDelete strategy, new DaemonSet Pods will only be created when you manually delete old ones.

RollingUpdate, on the other hand, smoothly transitions from one version to another without any downtime. It updates a subset of Pods at a time, moving incrementally, until all Pods are updated. For this strategy, you need to define additional fields, maxSurge and maxUnavailable. You can define these fields either as integers or percentages:

spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25% 
spec:
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1

maxSurge specifies the maximum number of additional Pods that you can create beyond the replica count. The maxUnavailable subfield specifies the maximum number of Pods that are unavailable (not running) at any given time during the update. In the example above, let's say replicas is equal to 5, then maxSurge = .25x5 = 1 and maxUnavailable = .25x5 = 1.

So, what triggers a rollout? This happens if the Pod template (.spec.template) of a Deployment or DaemonSet changes. For example, if you change the container image or labels. You can do this imperatively with the kubectl set command or edit the Deployment. To practice this, update the Deployment to include the update strategy and deploy it. Then, modify the Deployment to use v1 of the hyperskill-clone image:

$ kubectl set image deployment/hyperskill-clone hyperskill-backend=initram/hyperskill-clone:v1

Once you run this command or edit the Deployment, a rollout triggers immediately. To see the rollout status, use the following command:

$ kubectl rollout status deployment/hyperskill-clone
# deployment "hyperskill-clone" successfully rolled out

To see the rollout history, run:

$ kubectl rollout history deployment hyperskill-clone
# REVISION  CHANGE-CAUSE
# 1         <none>
# 2         <none>

To view more details about a particular revision, use the --revision flag:

$ kubectl rollout history deployment/hyperskill-clone --revision 2

Now that you know how to update to new versions, how do you revert to previous versions? For that, simply undo the rollout:

$ kubectl rollout undo deployment hyperskill-clone
# deployment.apps/hyperskill-clone rolled back

Now, when you check the rollout history, revision 1 is gone. This is because Kubernetes simply reuses the previous template and renumbers it to the latest version. Revision 1 is now revision 3. You can also roll back to a specific version with the --to-revision flag:

$ kubectl rollout undo deployment hyperskill-clone --to-revision 2 
# deployment.apps/hyperskill-clone rolled back

Conclusion

In this topic, you learned about the key controllers — ReplicaSets, DaemonSets, and Deployments — for running replicated workloads in Kubernetes. ReplicaSets offer reliability, scalability, and load balancing. Deployments offer these benefits and go further by providing the ability to update your application in a declarative and controlled manner. DaemonSets come in when you need to ensure a copy of a Pod runs on all or a subset of nodes.

Both DaemonSets and Deployments let you define an update strategy. The Recreate strategy terminates all existing Pods before deploying new ones. While simple, this can lead to downtime. On the other hand, RollingUpdate allows for a smoother transition, ensuring that a specific number of Pods are always available. For DaemonSet Pods, instead of Recreate, you use the OnDelete strategy to only replace Pods when old ones are removed.

How did you like the theory?
Report a typo