Kubernetes DaemonSets: Complete Real-Time Guide for Running Background Tasks on Every Node

In Kubernetes, most applications are deployed using Deployments, ReplicaSets, or StatefulSets. These controllers are useful when we want to run a specific number of application Pods across the cluster.

But some workloads are different. They must run on every node in the Kubernetes cluster.

Examples include:

Log collection agents
Node monitoring agents
Security scanning agents
Network plugins
Storage agents
Backup agents
System-level observability tools

For these node-level workloads, Kubernetes provides DaemonSets.

A DaemonSet ensures that one copy of a Pod runs on every eligible node. When a new node joins the cluster, Kubernetes automatically schedules the DaemonSet Pod on that node. When a node is removed, the corresponding Pod is also removed.

What is a DaemonSet?

A DaemonSet is a Kubernetes controller used to run background or infrastructure Pods on every node in a cluster.

Simple definition:

A DaemonSet ensures that each selected Kubernetes node runs one copy of a specific Pod.

This makes DaemonSets ideal for cluster-wide services that must collect data, monitor health, manage networking, or support node-level operations.

Why DaemonSets Are Needed?

Imagine a Kubernetes cluster with five worker nodes.

[ Kubernetes Cluster ]

Node-1
Node-2
Node-3
Node-4
Node-5

If we want a log collector on every node, manually deploying five separate Pods is not practical.

Also, if a sixth node is added tomorrow, we would need to manually remember to deploy another log collector Pod.

DaemonSet solves this automatically.

[ DaemonSet Created ]
        |
        v
Node-1 ---> Log Collector Pod
Node-2 ---> Log Collector Pod
Node-3 ---> Log Collector Pod
Node-4 ---> Log Collector Pod
Node-5 ---> Log Collector Pod

New Node Added
        |
        v
Node-6 ---> Log Collector Pod Automatically Created

Real-Time Example: Logging in Production

In a production Kubernetes cluster, application logs are usually written on the node where the Pod is running.

If logs are not collected properly, debugging becomes very difficult.

A logging agent such as Fluentd, Fluent Bit, or Filebeat is commonly deployed as a DaemonSet.

Logging Architecture

[ Application Pod on Node-1 ]
          |
          v
Writes Logs
          |
          v
[ Log Collector DaemonSet Pod ]
          |
          v
Sends Logs to Loki / Elasticsearch / Cloud Logging

Because the log collector runs on every node, it can collect logs from all applications running across the cluster.

Real-Time Example: Monitoring Every Node

Monitoring tools need node-level metrics such as:

CPU usage
Memory usage
Disk usage
Network traffic
Filesystem health
Node availability

Prometheus Node Exporter is commonly deployed as a DaemonSet.

[ Node-1 ] ---> node-exporter
[ Node-2 ] ---> node-exporter
[ Node-3 ] ---> node-exporter
[ Node-4 ] ---> node-exporter

Prometheus then scrapes metrics from each node-exporter Pod.

[ Node Exporter DaemonSet ]
          |
          v
[ Prometheus ]
          |
          v
[ Grafana Dashboard ]

This helps DevOps teams identify overloaded nodes, memory pressure, disk problems, and performance bottlenecks.

Real-Time Banking Example

In a banking Kubernetes cluster, every node must be monitored carefully because services may process sensitive and high-value operations such as:

Money transfers
UPI payments
Loan processing
Account balance checks
Fraud detection

A monitoring DaemonSet can collect node metrics from all worker nodes.

A security DaemonSet can scan node activity and detect suspicious behavior.

A logging DaemonSet can forward application logs to a centralized audit system.

Banking DaemonSet Architecture

[ Banking Kubernetes Cluster ]

Node-1:
  Payment Pods
  Logging Agent
  Monitoring Agent
  Security Agent

Node-2:
  Auth Pods
  Logging Agent
  Monitoring Agent
  Security Agent

Node-3:
  Fraud Detection Pods
  Logging Agent
  Monitoring Agent
  Security Agent

This architecture improves visibility, security, and compliance.

Real-Time E-Commerce Example

During a festival sale, an e-commerce platform may experience sudden traffic spikes.

DaemonSets help platform teams monitor every node in real time.

Node exporter tracks CPU and memory usage
Fluent Bit collects application logs
Security agents detect abnormal activity
CNI plugins manage network connectivity

If one node becomes overloaded, monitoring alerts can notify the DevOps team before users experience failures.

DaemonSet vs Deployment

Feature	Deployment	DaemonSet
Main Purpose	Run application replicas	Run one Pod per node
Pod Count	Defined by replicas	Based on number of nodes
Best For	APIs, frontend apps, microservices	Logging, monitoring, networking agents
New Node Behavior	No automatic Pod unless scheduler chooses it	Pod automatically runs on new node
Typical Use	Business applications	Infrastructure services

Basic DaemonSet YAML Example

apiVersion: apps/v1
kind: DaemonSet

metadata:
  name: log-collector

spec:
  selector:
    matchLabels:
      app: log-collector

  template:
    metadata:
      labels:
        app: log-collector

    spec:
      containers:
      - name: log-collector
        image: fluentd:latest

        resources:
          requests:
            cpu: "100m"
            memory: "200Mi"

          limits:
            cpu: "300m"
            memory: "500Mi"

This DaemonSet deploys one Fluentd log collector Pod on every eligible node.

Understanding DaemonSet YAML

Field	Purpose
kind: DaemonSet	Defines DaemonSet resource
selector	Matches DaemonSet-managed Pods
template	Defines Pod specification
resources	Controls CPU and memory usage

DaemonSet Workflow

DaemonSet Created
        |
        v
Kubernetes Checks Eligible Nodes
        |
        v
Creates One Pod Per Node
        |
        v
New Node Added
        |
        v
DaemonSet Pod Automatically Scheduled
        |
        v
Node Removed
        |
        v
DaemonSet Pod Cleaned Up

Using Node Selectors with DaemonSets

Sometimes we do not want a DaemonSet to run on every node. We may want it to run only on selected nodes.

For example, GPU monitoring agents should run only on GPU nodes.

Node Selector Example

spec:
  template:
    spec:
      nodeSelector:
        node-type: gpu

      containers:
      - name: gpu-monitor
        image: gpu-monitor:latest

This DaemonSet runs only on nodes labeled:

node-type=gpu

Using Taints and Tolerations

Some nodes are tainted to prevent normal workloads from running on them.

DaemonSets may need tolerations to run on such nodes.

For example, monitoring or networking agents may need to run on control-plane nodes.

Toleration Example

tolerations:
- key: "node-role.kubernetes.io/control-plane"
  operator: "Exists"
  effect: "NoSchedule"

This allows the DaemonSet Pod to run on tainted control-plane nodes.

DaemonSet Update Strategy

DaemonSets support rolling updates.

When the DaemonSet image or configuration changes, Kubernetes updates Pods gradually across nodes.

Rolling Update Flow

Update DaemonSet Image
        |
        v
Update Pod on Node-1
        |
        v
Wait Until Ready
        |
        v
Update Pod on Node-2
        |
        v
Continue Until All Nodes Updated

This avoids updating all node agents at the same time.

DaemonSet for Kubernetes Networking

Many Kubernetes networking plugins run as DaemonSets.

Examples:

Calico
Flannel
Cilium
Weave Net

These agents must run on every node to provide networking between Pods.

[ Node-1 ] ---> CNI Agent
[ Node-2 ] ---> CNI Agent
[ Node-3 ] ---> CNI Agent

If the CNI DaemonSet fails, Pod networking may break.

DaemonSet for Security Agents

Security teams often deploy agents as DaemonSets to monitor every node.

These agents may detect:

Suspicious processes
Unauthorized file changes
Malware activity
Container escape attempts
Unexpected network traffic

This is common in banking, healthcare, fintech, and enterprise SaaS platforms.

Production Troubleshooting Commands

kubectl get daemonsets

kubectl get ds -A

kubectl describe daemonset log-collector

kubectl get pods -o wide

kubectl logs daemonset/log-collector

kubectl describe pod pod-name

Common DaemonSet Problems

1. Pod Not Running on Some Nodes

Possible reasons:

Node selector mismatch
Taints without tolerations
Insufficient resources
Node not ready

2. DaemonSet Consuming Too Many Resources

Always set requests and limits. Infrastructure agents should not overload business workloads.

3. Wrong Image Version

Using latest may create unexpected behavior. Use fixed image tags.

4. No RBAC Permissions

Monitoring and logging agents often need permissions to read node or Pod metadata.

Production Debugging Flow

DaemonSet Pod Missing
        |
        v
Check Nodes
        |
        v
Check Node Labels
        |
        v
Check Taints
        |
        v
Check DaemonSet Selector
        |
        v
Check Pod Events
        |
        v
Fix Scheduling Rules

Best Practices for DaemonSets

Use DaemonSets only for node-level workloads
Always configure CPU and memory requests
Use limits to avoid node resource pressure
Use node selectors for targeted scheduling
Use tolerations carefully for tainted nodes
Use fixed image versions instead of latest
Monitor DaemonSet health continuously
Use RBAC with least privilege

Interview Questions

Q1: What is a DaemonSet?

A DaemonSet ensures that one copy of a Pod runs on every eligible node in the Kubernetes cluster.

Q2: When do we use DaemonSet?

DaemonSets are used for node-level workloads such as logging agents, monitoring agents, networking plugins, and security agents.

Q3: Difference between Deployment and DaemonSet?

Deployment runs a specified number of replicas, while DaemonSet runs one Pod per eligible node.

Q4: What happens when a new node is added?

The DaemonSet automatically schedules its Pod on the new node.

Q5: Can DaemonSet run only on selected nodes?

Yes. Node selectors, affinity, taints, and tolerations can control where DaemonSet Pods run.

Interview Trap Questions

Does DaemonSet always run on every node?

Not always. It runs on every eligible node based on selectors, taints, tolerations, and scheduling rules.

Can DaemonSet be used for frontend applications?

Usually no. Frontend apps should use Deployments. DaemonSets are better for node-level agents.

Can DaemonSet Pods be updated?

Yes. DaemonSets support rolling updates.

Does DaemonSet replace monitoring tools?

No. DaemonSet is only a deployment method. Monitoring tools like Prometheus and Grafana still collect and visualize metrics.

Recommended Learning Path

Summary

DaemonSets are essential for running background infrastructure tasks on every Kubernetes node.

They are commonly used for:

Logging
Monitoring
Networking
Security
Storage agents

A well-designed Kubernetes cluster often depends on multiple DaemonSets to collect logs, monitor health, manage networking, and enforce security.

Understanding DaemonSets helps developers and DevOps engineers manage production Kubernetes clusters more effectively and confidently.