Kubernetes CKAD — Chhota shortcut — Volumes, Namespace , StatefulSet , and DaemonSet — Part 3

Abhishek Purohit
17 min readMay 25, 2024

--

Let’s break down these Kubernetes concepts of Volumes, Namespaces, StatefulSets, and DaemonSets in Kubernetes, particularly within Azure Kubernetes Service (AKS), and how they can be managed using Terraform. The explanations will include examples and mnemonics to help you understand these concepts better.

Volumes

Simple Explanation: Volumes in Kubernetes are like backpacks for Pods. Just as you use a backpack to carry your books and essentials regardless of where you go, a volume provides a way for containers within a pod to store and access data persistently, even as the containers start and stop.

Volumes are critical for any application that needs to maintain data beyond the lifecycle of a single container. In AKS, Terraform can be used to provision and manage these volumes. For example, you can attach Azure Disk or Azure Files as volumes to your containers, ensuring that data persistence is handled seamlessly.

resource "kubernetes_persistent_volume_claim" "example" {
metadata {
name = "example-vol-claim"
}
spec {
access_modes = ["ReadWriteOnce"]
resources {
requests = {
storage = "10Gi"
}
}
storage_class_name = "default"
}
}

Mnemonic: Very Organized Luggage Uses Memory Efficiently (VOLUME)

Namespaces

Simple Explanation: Think of Namespaces as different rooms in a big office building. Each room (namespace) can have its own rules, settings, and limited access, allowing teams (applications) to operate independently without interference.

Namespaces are used to partition cluster resources between multiple users and applications. They are particularly useful in Terraform to manage different environments (like development, testing, and production) within the same AKS cluster, applying specific resource quotas and access controls.

resource "kubernetes_namespace" "example" {
metadata {
name = "example-namespace"
}
}

Mnemonic: Not All Members Enter Some Places Alike (NAMESPACE)

StatefulSets

Simple Explanation: If Pods need a permanent home, StatefulSets are the solution. They ensure that each pod has a unique, stable identity and persistent storage, which is crucial for services like databases that must remember information between restarts.

StatefulSets are essential for running stateful applications in Kubernetes, such as databases or other services that require stable, unique network identifiers, predictable deployment and scaling, and persistent data storage. Terraform can be utilized to automate the deployment and management of these resources in AKS.

resource "kubernetes_stateful_set" "example" {
metadata {
name = "example-statefulset"
}
spec {
selector {
match_labels = {
app = "example-app"
}
}
service_name = "example-service"
replicas = 3

template {
metadata {
labels = {
app = "example-app"
}
}

spec {
container {
name = "example"
image = "example/image"
}
}
}
}
}

Stateful Tasks Always Take Extra Focused Understanding Logically (STATEFUL)

DaemonSets

Simple Explanation: DaemonSets are like having a security camera in every room of a building. They ensure that every node (room) in your Kubernetes cluster runs a copy of a specific pod, ideal for deploying system-wide services like monitoring agents or log collectors.

DaemonSets are used when you need a specific type of service to run on all or certain nodes within a cluster. This is useful for services that need to monitor or manage the nodes themselves, such as network proxies, log collectors, or storage daemons. Terraform can define and manage these DaemonSets within AKS.

resource "kubernetes_daemon_set" "example" {
metadata {
name = "example-daemonset"
}
spec {
selector {
match_labels = {
app = "daemon-app"
}
}

template {
metadata {
labels = {
app = "daemon-app"
}
}

spec {
container {
name = "daemon"
image = "daemon/image"
}
}
}
}
}

More Terraform Examples

1. Dynamic Provisioning of Persistent Volumes

resource "kubernetes_persistent_volume_claim" "example_pvc" {
metadata {
name = "example-pvc"
}
spec {
access_modes = ["ReadWriteOnce"]
resources {
requests = {
storage = "10Gi"
}
}
storage_class_name = "azure-disk"
}
}

Explanation: This example shows how to dynamically provision a Persistent Volume using an Azure Disk as the backend storage. ReadWriteOnce means the volume can be mounted as read-write by a single node.

2. Namespaces for Environment Separation

resource "kubernetes_namespace" "dev" {
metadata {
name = "development"
}
}

resource "kubernetes_namespace" "prod" {
metadata {
name = "production"
}
}

Explanation: These examples create two separate namespaces for development and production environments, helping to isolate resources and manage permissions more effectively.

3. StatefulSet with Volume Claim Templates

resource "kubernetes_stateful_set" "example_statefulset" {
metadata {
name = "example-statefulset"
}
spec {
replicas = 3
selector {
match_labels = {
app = "example-app"
}
}
template {
metadata {
labels = {
app = "example-app"
}
}
spec {
container {
name = "nginx"
image = "nginx"
volume_mounts {
mount_path = "/usr/share/nginx/html"
name = "html"
}
}
}
}
volume_claim_templates {
metadata {
name = "html"
}
spec {
access_modes = ["ReadWriteOnce"]
resources {
requests {
storage = "10Gi"
}
}
}
}
}
}

Explanation: This StatefulSet definition includes a volume claim template, which automatically provisions persistent storage for each replica of the StatefulSet, ideal for applications like databases that require persistence.

4. DaemonSet for Node Monitoring

resource "kubernetes_daemon_set" "monitoring" {
metadata {
name = "monitoring"
}
spec {
selector {
match_labels = {
app = "monitor"
}
}
template {
metadata {
labels = {
app = "monitor"
}
}
spec {
container {
name = "monitor"
image = "prometheus/node-exporter"
}
}
}
}
}

Explanation: This DaemonSet deploys a Prometheus Node Exporter on every node of the cluster, ensuring all nodes are monitored consistently.

5. Namespace with Resource Quota

resource "kubernetes_namespace" "test" {
metadata {
name = "test-namespace"
}
}

resource "kubernetes_resource_quota" "test_quota" {
metadata {
name = "test-quota"
namespace = kubernetes_namespace.test.metadata[0].name
}

spec {
hard = {
"cpu" = "20"
"memory" = "1Gi"
"pods" = "10"
}
}
}

Explanation: This example sets a resource quota in the test namespace, limiting the total CPU, memory, and pods that can be created, helping to prevent resource starvation in multi-tenant environments.

6. Advanced Volume Configuration with Reclaim Policy

resource "kubernetes_persistent_volume" "example_pv" {
metadata {
name = "example-pv"
}
spec {
capacity = {
storage = "10Gi"
}
access_modes = ["ReadWriteOnce"]
persistent_volume_reclaim_policy = "Retain"
storage_class_name = "manual"
persistent_volume_source {
azure_disk {
disk_name = "my-disk"
disk_uri = "https://example.blob.core.windows.net/vhds/mydisk.vhd"
kind = "Shared"
}
}
}
}

Explanation: This persistent volume example sets a manual storage class with an Azure Disk as the source and a Retain reclaim policy, which means that the volume will remain after the associated PVC is deleted, protecting data from accidental deletion.

let’s compare different types of Kubernetes volumes that are essential for managing data storage in Azure Kubernetes Service (AKS). Understanding these volume types will help anyone choose the right storage solutions based on specific needs, whether for performance, data persistence, or ease of management. Here’s a detailed table to break down the characteristics and uses of each volume type.

Detailed Explanation and Context

  • EmptyDir: Think of it as a scratchpad that disappears when you’re done. It’s fast and tied to the pod’s lifecycle, making it ideal for temporary data like caches.
  • HostPath: It’s akin to giving your pod direct access to the underlying server’s filesystem. This is powerful but should be used cautiously as it can expose critical host files.
  • PersistentVolume and PersistentVolumeClaim: These are like renting a storage unit that remains yours even if you move houses (pods). They’re essential for applications like databases where data must persist beyond the life of a pod.
  • Azure Disk and Azure Files: These are two flavors of managed storage from Azure. Azure Disk is like a dedicated drive that’s great for performance-sensitive scenarios, while Azure Files is like a shared network drive, accessible by multiple clients simultaneously.
  • ConfigMap and Secret: These are like safes within your Kubernetes environment. ConfigMaps hold non-sensitive configuration files, while Secrets are encrypted containers for sensitive information, keeping credentials secure.
  • NFS: This is the communal park of file storage — accessible by everyone who has access, suitable for applications that require a common storage platform across multiple pods.

1. Persistent Volume Claim (PVC) with Azure Disk

This example creates a PersistentVolumeClaim that automatically provisions an Azure Disk to store persistent data. This is ideal for stateful applications like databases that require stable, reliable storage.

resource "kubernetes_persistent_volume_claim" "azure_disk_pvc" {
metadata {
name = "azure-disk-pvc"
}
spec {
access_modes = ["ReadWriteOnce"]
resources {
requests {
storage = "100Gi"
}
}
storage_class_name = "default"
}
}

Explanation: This PVC requests a 100 GB Azure Disk with ReadWriteOnce access mode, meaning the disk can be mounted as read-write by a single node. This is standard for databases or other applications where data integrity and performance are critical.

2. Azure Files Volume

This example sets up a PersistentVolumeClaim that is backed by Azure Files, providing shared storage accessible by multiple pods across different nodes.

resource "kubernetes_persistent_volume_claim" "azure_files_pvc" {
metadata {
name = "azure-files-pvc"
}
spec {
access_modes = ["ReadWriteMany"]
resources {
requests {
storage = "50Gi"
}
}
storage_class_name = "azurefile"
}
}

Explanation: This PVC uses Azure Files with a ReadWriteMany access mode, allowing the volume to be mounted as read-write by multiple nodes simultaneously. This setup is suitable for applications that require a common file system accessible from multiple pods.

3. ConfigMap Volume

Here’s how to define a ConfigMap in Terraform and mount it as a volume inside a Kubernetes pod. ConfigMaps are useful for storing non-sensitive configuration details that need to be dynamically accessible within pods.

resource "kubernetes_config_map" "example_config_map" {
metadata {
name = "example-config"
}

data = {
"config.json" = jsonencode({
"property1" = "value1"
"property2" = "value2"
})
}
}

resource "kubernetes_pod" "example_pod" {
metadata {
name = "config-demo-pod"
}

spec {
container {
image = "busybox"
name = "example-container"

volume_mounts {
mount_path = "/etc/config"
name = "config-volume"
}
}

volume {
name = "config-volume"

config_map {
name = kubernetes_config_map.example_config_map.metadata[0].name
default_mode = "0400"
}
}
}
}

Explanation: This Terraform configuration first defines a ConfigMap with some JSON data and then mounts it into a pod. The pod’s container will have access to the contents of the ConfigMap at /etc/config. This approach is suitable for applications that need to read configuration data or other non-sensitive information at runtime.

“Namespace” is a built-in concept used to support multiple virtual clusters on the same physical cluster. Rather than different types of Namespaces, you typically manage different Namespaces each dedicated to a specific purpose or environment. This can help segregate resources, manage access, and simplify resource management in large systems.

Detailed Explanation and Context

  • Default Namespace: This is like the main lobby of a building where general public activities happen unless specified otherwise. It’s where resources land if you don’t specify a particular Namespace, serving general purposes.
  • kube-system Namespace: Think of this as the engine room of your Kubernetes cluster. It runs all the critical system processes that Kubernetes needs to operate. Because of its importance, only cluster administrators typically have access to modify resources in this Namespace.
  • kube-public Namespace: This can be likened to a public bulletin board within a company. It’s available to anyone, even anonymous users, which can be useful for resources that need to be universally accessible within the cluster.
  • Development Namespace: Consider this as the R&D department where developers experiment and test new applications. It’s typically configured to allow developers to freely create and manage resources without affecting production workloads.
  • Production Namespace: This is akin to the high-security area of a business where critical business operations occur. Access is tightly controlled, and resources are monitored and managed to ensure optimal performance and security.

Creating a Basic Namespace

This example shows how to create a simple Kubernetes namespace using Terraform. This can be used for general, non-specific workloads or for initial testing.

resource "kubernetes_namespace" "basic" {
metadata {
name = "basic-namespace"
}
}

Explanation: This defines a new Kubernetes namespace named basic-namespace. It’s a straightforward example of how to segregate resources in a Kubernetes cluster without additional configurations.

Namespace for Development

Here’s how to create a namespace specifically for development purposes, which might have less restrictive settings compared to a production environment.

resource "kubernetes_namespace" "development" {
metadata {
name = "development"
labels = {
environment = "development"
}
}
}

Explanation: This namespace is labeled with environment: development to indicate its usage. Labels help in managing attributes connected with different environments, making it easier to filter and apply policies.

Namespace with Resource Quota

Managing resources within namespaces is crucial, especially in environments like production where resource utilization must be carefully controlled. This example shows how to set a resource quota in a namespace to manage resource consumption.

resource "kubernetes_namespace" "production" {
metadata {
name = "production"
}
}

resource "kubernetes_resource_quota" "prod_quota" {
metadata {
name = "prod-quota"
namespace = kubernetes_namespace.production.metadata[0].name
}

spec {
hard = {
"cpu" = "20"
"memory" = "10Gi"
"pods" = "100"
}
}
}

Explanation: This setup creates a production namespace and establishes a resource quota named prod-quota. The quota limits the CPU usage to 20 cores, memory to 10 GiB, and a maximum of 100 pods, preventing resource starvation and ensuring that resources are available evenly.

Namespace for Staging Environment

A staging environment often mirrors the production settings but is used for testing. Here’s how to define it, potentially with annotations to manage policies or integrations specific to staging.

resource "kubernetes_namespace" "staging" {
metadata {
name = "staging"
annotations = {
description = "Staging environment for pre-production testing"
}
}
}

Explanation: This namespace includes an annotation providing a description of its purpose. Annotations can be used to store additional metadata about the namespace, which can be utilized by tools and applications for specific operational behaviors.

StatefulSets are not differentiated by types in the same way as services or volumes. Instead, StatefulSets are a single type of Kubernetes controller specifically designed for managing stateful applications. However, StatefulSets can be configured and used in various ways depending on the needs of your application, such as scaling policies, update strategies, and persistent storage configurations.

Detailed Explanation and Context

  • Basic StatefulSet: Think of this as the foundational way to deploy applications that need specific identities and stable storage, such as an application that maintains user sessions or transaction histories.
  • StatefulSet with Dynamic Volume Provisioning: This setup is like having an expandable filing cabinet where new drawers are added automatically as you need more space, ideal for growing data-centric applications.
  • StatefulSet with Custom Update Strategies: Adjusting the update strategies is akin to planning roadwork in a city — you ensure that every lane change or closure (update) happens in a manner that least disturbs the traffic (application availability).
  • StatefulSet with Pod Anti-Affinity: By spreading Pods across different nodes, it’s like ensuring that not all executives (Pods) are on the same flight (node), which spreads out the risk and increases the resilience of the overall architecture.

Basic StatefulSet Configuration

This example demonstrates a basic StatefulSet configuration for deploying a stateful application, such as a simple database.

resource "kubernetes_stateful_set" "basic" {
metadata {
name = "basic-statefulset"
namespace = "default"
}

spec {
replicas = 3

selector {
match_labels = {
app = "myapp"
}
}

service_name = "myapp-service"

template {
metadata {
labels = {
app = "myapp"
}
}

spec {
container {
image = "nginx"
name = "nginx"
}
}
}

volume_claim_template {
metadata {
name = "storage"
}

spec {
access_modes = ["ReadWriteOnce"]
resources {
requests {
storage = "10Gi"
}
}
}
}
}
}

Explanation: This Terraform configuration sets up a StatefulSet with 3 replicas. Each Pod in the StatefulSet will use a PersistentVolumeClaim to request 10 GB of storage. This setup is ideal for applications that require stable storage and identity but do not have complex update requirements.

StatefulSet with Custom Update Strategy

Here’s how to configure a StatefulSet with a custom RollingUpdate strategy to manage how updates to Pods are rolled out, minimizing downtime for critical applications.

resource "kubernetes_stateful_set" "update_strategy" {
metadata {
name = "update-strategy-statefulset"
namespace = "default"
}

spec {
replicas = 3
update_strategy {
type = "RollingUpdate"
rolling_update {
partition = 1
}
}

selector {
match_labels = {
app = "myapp"
}
}

service_name = "myapp-service"

template {
metadata {
labels = {
app = "myapp"
}
}

spec {
container {
image = "nginx"
name = "nginx"
}
}
}
}
}

Explanation: This configuration specifies a RollingUpdate update strategy with a partition. This means that updates will roll out from the newest to the oldest Pod, stopping at the specified partition, which allows for more controlled deployments and testing of new versions in a subset of Pods.

StatefulSet with Pod Anti-Affinity

This example sets up a StatefulSet with pod anti-affinity rules to ensure that the pods are not placed on the same node, enhancing the application’s availability and resilience.

resource "kubernetes_stateful_set" "high_availability" {
metadata {
name = "high-availability-statefulset"
namespace = "default"
}

spec {
replicas = 3

selector {
match_labels = {
app = "high-avail-app"
}
}

service_name = "high-avail-service"

template {
metadata {
labels = {
app = "high-avail-app"
}
}

spec {
affinity {
pod_anti_affinity {
required_during_scheduling_ignored_during_execution {
label_selector {
match_expressions {
key = "app"
operator = "In"
values = ["high-avail-app"]
}
}

topology_key = "kubernetes.io/hostname"
}
}
}

container {
image = "nginx"
name = "nginx"
}
}
}
}
}

Explanation: This Terraform script configures pod anti-affinity to ensure that no two pods of the StatefulSet are placed on the same physical node. This configuration is crucial for high-availability setups, such as those needed for large-scale databases or critical application components that must withstand node failures without service disruption.

DaemonSets in Kubernetes do not inherently come in different types, but they can be configured in various ways to serve different purposes. DaemonSets are used to ensure that each node in your Kubernetes cluster runs a copy of a specific pod, which is ideal for node-wide services like log collectors, monitoring agents, or security agents.

Detailed Explanation and Context

  • Basic DaemonSet: This is like assigning a security camera to every room in a building, ensuring complete coverage. It’s straightforward and covers all bases but might be overkill if some rooms don’t require surveillance.
  • DaemonSet with Limited Node Selection: This approach is like only placing cameras in rooms with valuables. By using node selectors, you can target specific nodes that have particular characteristics (like GPU nodes for compute-intensive tasks), optimizing resource use.
  • DaemonSet with Tolerations: Imagine a scenario where only certain rooms are allowed to handle hazardous materials. Tolerations in DaemonSets allow pods to be scheduled on nodes with specific taints, ensuring that only suitable nodes run these critical components.
  • DaemonSet with Host Networking: This is akin to giving security personnel an all-access key to every room’s network infrastructure, allowing them to monitor and manage network traffic directly. It’s powerful for network-related tasks but needs to be used cautiously to avoid security issues.

Basic DaemonSet Configuration

This example demonstrates a basic DaemonSet configuration that ensures a monitoring agent is deployed on every node in the cluster.

resource "kubernetes_daemon_set" "basic" {
metadata {
name = "basic-daemonset"
namespace = "default"
}

spec {
selector {
match_labels = {
app = "monitoring-agent"
}
}

template {
metadata {
labels = {
app = "monitoring-agent"
}
}

spec {
container {
image = "my-monitoring-agent:latest"
name = "monitoring-agent"
}
}
}
}
}

Explanation: This Terraform configuration sets up a DaemonSet to deploy a monitoring agent across all nodes. It’s a straightforward approach to ensure that every node in your cluster is monitored uniformly.

DaemonSet with Node Selector

Here’s how to configure a DaemonSet to deploy pods only on nodes with specific labels, such as nodes equipped with GPUs.

resource "kubernetes_daemon_set" "gpu_nodes" {
metadata {
name = "gpu-daemonset"
namespace = "default"
}

spec {
selector {
match_labels = {
app = "gpu-monitoring"
}
}

template {
metadata {
labels = {
app = "gpu-monitoring"
}
}

spec {
node_selector = {
"kubernetes.io/has-gpu" = "true"
}

container {
image = "gpu-monitoring-tool:latest"
name = "gpu-monitoring"
}
}
}
}
}

Explanation: This configuration uses a node selector to ensure the DaemonSet deploys the monitoring tool only on nodes labeled with kubernetes.io/has-gpu=true. This optimizes resource usage by targeting only relevant nodes.

DaemonSet with Host Networking

This example sets up a DaemonSet that uses the host network, which is ideal for network monitoring tools that need to observe or manage the host network.

resource "kubernetes_daemon_set" "network_monitor" {
metadata {
name = "network-monitor"
namespace = "default"
}

spec {
selector {
match_labels = {
app = "network-monitor"
}
}

template {
metadata {
labels = {
app = "network-monitor"
}
}

spec {
host_network = true

container {
image = "network-monitoring-tool:latest"
name = "network-monitor"
}
}
}
}
}

Explanation: By setting host_network to true, the DaemonSet allows pods to use the host’s network namespace. This is particularly useful for applications that need to monitor or interact directly with the host’s network, like security and performance tools.

DaemonSet with Tolerations

This example demonstrates how to use tolerations to allow a DaemonSet to deploy pods on tainted nodes, which might be reserved for specific services.

resource "kubernetes_daemon_set" "tainted_nodes" {
metadata {
name = "tainted-daemonset"
namespace = "default"
}

spec {
selector {
match_labels = {
app = "special-service"
}
}

template {
metadata {
labels = {
app = "special-service"
}
}

spec {
tolerations {
key = "node-type"
operator = "Equal"
value = "special"
effect = "NoSchedule"
}

container {
image = "special-service-tool:latest"
name = "special-service"
}
}
}
}
}

Explanation: This DaemonSet is configured with a toleration that matches a taint on certain nodes. This allows the DaemonSet to schedule pods specifically on nodes designated for special services, ensuring that critical, specialized applications have the required exclusivity and resources.

Let’s summarize the concepts of Volumes, Namespaces, StatefulSets, and DaemonSets in Kubernetes, with a twist inspired by the TV series “Justified,” weaving in some wit and analogies to make these concepts stick.

Volumes: The Raylan Givens’ Hat

Summary: In Kubernetes, a Volume is like Raylan Givens’ hat in “Justified” — it goes wherever Raylan goes, holding everything he needs, just like a Volume ensures data persists and is available wherever a container needs it. Whether Raylan is in a shootout or a quiet chat, his hat is a consistent presence, much like how Volumes provide consistent data access to Pods in Kubernetes, regardless of the underlying container lifecycle.

Namespaces: The Harlan County Lines

Summary: Namespaces in Kubernetes are like the different territories in Harlan County. Just as territories in “Justified” dictate who has control and where certain laws apply, Namespaces partition resources within a Kubernetes cluster, defining what can run where, and who has access to what. This helps in managing resources efficiently, ensuring that developments in one ‘territory’ don’t mess up the order in another.

StatefulSets: The Boyd Crowder’s Enterprises

Summary: If Pods in Kubernetes were outlaws, StatefulSets would be Boyd Crowder’s carefully managed enterprises. Just as Boyd’s operations need consistency and identity (each operation has its specific place and role in his grand plan), StatefulSets manage Pods that require stable, unique identities and persistent storage — like databases that must remember data across restarts and rebalancing, even if chaos ensues.

DaemonSets: The Marshals’ Surveillance

Summary: DaemonSets in Kubernetes are like having a U.S. Marshal stationed in every part of Harlan, ensuring law and order is maintained uniformly. No matter which part of the county trouble starts in, there’s someone there to handle it. Similarly, DaemonSets ensure that every node in your Kubernetes cluster runs a copy of a specific Pod, ideal for deploying node-wide services like log collectors or monitoring agents, maintaining order and oversight across the entire cluster.

“Justified” in Kubernetes

In the rough and tumble world of “Justified,” where you’ve got to keep your friends close and your enemies closer, managing a Kubernetes cluster isn’t much different. You need your Volumes like Raylan needs his hat — reliable and ever-present. Namespaces help you divide and conquer, much like defining territories in Harlan. StatefulSets are your plan for keeping the essential operations running smoothly, no matter the chaos. And DaemonSets? They’re your eyes and ears on the ground, making sure nothing slips past you.

--

--