Infrastructure Monitoring with Node Exporter, Blackbox Exporter, and cAdvisor

Table of Contents

• What You Will Learn
• Introduction to Infrastructure Monitoring
• Why Exporters Are Required
• Node Exporter Overview
• Node Exporter Architecture
• Node Exporter Installation
• Important Node Exporter Metrics
• Blackbox Exporter Overview
• Blackbox Exporter Workflow
• Blackbox Exporter Configuration
• cAdvisor Overview
• cAdvisor Architecture
• Important cAdvisor Metrics
• Enterprise Monitoring Architecture
• Production Best Practices
• Troubleshooting
• Interview Questions
• FAQs
• Summary

What You Will Learn

• Purpose of Node Exporter
• Purpose of Blackbox Exporter
• Purpose of cAdvisor
• Prometheus scraping workflow
• Infrastructure monitoring architecture
• Container monitoring strategies
• Endpoint availability monitoring
• Production deployment patterns
• Capacity planning metrics
• Troubleshooting monitoring failures
• Enterprise observability best practices

Introduction to Infrastructure Monitoring

Infrastructure monitoring is the process of collecting, storing, analyzing, and visualizing metrics generated by servers, virtual machines, containers, databases, networks, and applications.

A production banking platform may run hundreds or thousands of servers. Without monitoring, teams cannot identify:

• CPU saturation
• Memory leaks
• Disk exhaustion
• Network bottlenecks
• Container resource issues
• Website outages
• Service downtime
• Slow response times

Monitoring provides visibility into system health before customers experience failures.

Users
  |
  V
Applications
  |
  V
Infrastructure
  |
  V
Metrics Collection
  |
  V
Prometheus
  |
  V
Grafana
  |
  V
Alerts + Dashboards

Why Exporters Are Required

Prometheus uses a pull-based model. It periodically scrapes HTTP endpoints exposing metrics in Prometheus format.

Most operating systems and applications do not expose Prometheus-compatible metrics natively.

Exporters bridge this gap.

Linux Server
     |
     V
Node Exporter
     |
     V
/metrics Endpoint
     |
     V
Prometheus Scrape

Different exporters are designed for different monitoring requirements.

Node Exporter Overview

Node Exporter is the standard Prometheus exporter for operating system metrics.

It collects:

• CPU utilization
• Memory consumption
• Disk usage
• Disk I/O
• Filesystem information
• Network traffic
• System load
• Kernel statistics
• Process statistics
• Hardware information

Node Exporter is installed on every server that needs monitoring.

+----------------------+
| Linux Server         |
|                      |
| Node Exporter        |
| Port 9100            |
+----------+-----------+
           |
           V
      Prometheus

Node Exporter Architecture

Node Exporter consists of multiple collectors.

Each collector gathers a specific category of metrics.

Node Exporter
      |
      +--- CPU Collector
      |
      +--- Memory Collector
      |
      +--- Filesystem Collector
      |
      +--- Disk Collector
      |
      +--- Network Collector
      |
      +--- Load Average Collector

Metrics are exposed through:

http://server-ip:9100/metrics

Node Exporter Installation

Download Binary

wget https://github.com/prometheus/node_exporter/releases/download/v1.8.1/node_exporter.tar.gz

tar -xvf node_exporter.tar.gz

cd node_exporter

Start Exporter

./node_exporter

Verify Metrics

curl http://localhost:9100/metrics

Prometheus Configuration

scrape_configs:

- job_name: "node-exporter"

  static_configs:
  - targets:
    - 10.0.1.10:9100
    - 10.0.1.11:9100
    - 10.0.1.12:9100

Important Node Exporter Metrics

CPU Utilization Query

100 -
(
avg by(instance)
(
rate(node_cpu_seconds_total{mode="idle"}[5m])
)
* 100
)

Memory Usage Query

(
1 -
(
node_memory_MemAvailable_bytes
/
node_memory_MemTotal_bytes
)
)
* 100

Blackbox Exporter Overview

Node Exporter tells us whether a server is healthy.

Blackbox Exporter tells us whether a service is reachable.

It performs active probing.

Supported protocols include:

• HTTP
• HTTPS
• TCP
• ICMP (Ping)
• DNS
• TLS

Prometheus
    |
    V
Blackbox Exporter
    |
    V
Target Endpoint

Blackbox Exporter Workflow

Prometheus
    |
    | Scrape
    V
Blackbox Exporter
    |
    | Probe
    V
https://banking-api.com
    |
    V
Response Metrics

Collected metrics include:

• Availability
• Status Code
• DNS Resolution Time
• TLS Expiry
• Response Time
• Redirect Count

Blackbox Exporter Configuration

blackbox.yml

modules:

  http_2xx:

    prober: http

    timeout: 5s

    http:

      valid_status_codes: []

      method: GET

      preferred_ip_protocol: ip4

Prometheus Configuration

- job_name: blackbox-http

  metrics_path: /probe

  params:
    module: [http_2xx]

  static_configs:
    - targets:
      - https://banking-api.company.com
      - https://payments.company.com

  relabel_configs:

    - source_labels: [__address__]
      target_label: __param_target

    - source_labels: [__param_target]
      target_label: instance

    - target_label: __address__
      replacement: blackbox-exporter:9115

cAdvisor Overview

Modern applications run inside containers.

Node Exporter provides host-level visibility but cannot provide detailed container metrics.

cAdvisor fills this gap.

cAdvisor stands for Container Advisor.

It collects:

• Container CPU Usage
• Container Memory Usage
• Network Usage
• Filesystem Usage
• Container Lifecycle Data
• Container Metadata

cAdvisor Architecture

Docker Host
     |
     +---- Container A
     |
     +---- Container B
     |
     +---- Container C
     |
     V
   cAdvisor
     |
     V
Prometheus

cAdvisor continuously monitors Docker runtime APIs and Linux cgroups.

Important cAdvisor Metrics

Docker Deployment

docker run \
--volume=/:/rootfs:ro \
--volume=/var/run:/var/run:rw \
--volume=/sys:/sys:ro \
--volume=/var/lib/docker/:/var/lib/docker:ro \
--publish=8080:8080 \
gcr.io/cadvisor/cadvisor:latest

Enterprise Monitoring Architecture

+----------------------------------------------------+
|                    Grafana                         |
+----------------------------------------------------+
                       |
                       V
+----------------------------------------------------+
|                  Prometheus                        |
+----------------------------------------------------+
        |                |                  |
        |                |                  |
        V                V                  V

 Node Exporter   Blackbox Exporter     cAdvisor

        |                |                  |

        V                V                  V

 Servers       APIs/Websites        Containers

Large organizations deploy all three exporters simultaneously.

For example:

• Node Exporter monitors Linux hosts.
• Blackbox Exporter monitors customer-facing APIs.
• cAdvisor monitors Docker containers.

Production Best Practices

Secure Exporters

• Restrict exporter access using firewalls
• Use private networks
• Enable TLS where supported
• Apply network policies in Kubernetes

Use Service Discovery

Avoid static target configuration in large environments.

Use:

• Kubernetes Service Discovery
• Consul Service Discovery
• AWS EC2 Discovery
• Azure Discovery

Label Standards

environment=production
application=payments
region=ap-south-1
team=platform

Alerting

Create alerts for:

• CPU greater than 90%
• Memory greater than 85%
• Disk greater than 80%
• Endpoint unavailable
• Container restart spikes

Troubleshooting

Target Down

Verify exporter process is running.

ps -ef | grep node_exporter

Connection Refused

netstat -tulpn | grep 9100

Metrics Missing

Check Prometheus targets page.

http://prometheus:9090/targets

Blackbox Probe Failure

Verify:

• DNS Resolution
• Firewall Rules
• TLS Certificates
• Network Connectivity

cAdvisor Issues

Validate Docker volume mounts and permissions.

Interview Questions and Answers

1. What is Node Exporter?

Node Exporter collects operating system and hardware metrics from Linux and Unix servers and exposes them for Prometheus scraping.

2. Why is Blackbox Exporter called black-box monitoring?

Because it monitors systems externally without requiring internal access to application code.

3. What is cAdvisor?

cAdvisor is a container monitoring agent that exposes container-level resource metrics.

4. Can Node Exporter monitor Docker containers?

Only at the host level. Detailed container metrics require cAdvisor.

5. Which exporter measures API availability?

Blackbox Exporter.

6. Which exporter provides CPU and memory metrics for Linux servers?

Node Exporter.

7. Which exporter provides container resource consumption metrics?

cAdvisor.

8. What is the default Node Exporter port?

9100.

9. What is the default Blackbox Exporter port?

9115.

10. What is the default cAdvisor port?

8080.

Frequently Asked Questions

What does Node Exporter monitor?

CPU, memory, disk, filesystem, network, load average, and operating system metrics.

What does Blackbox Exporter monitor?

Website availability, APIs, DNS endpoints, TCP services, and network connectivity.

What does cAdvisor monitor?

Container CPU, memory, network, and filesystem metrics.

Can Prometheus work without exporters?

Only if applications expose Prometheus-compatible metrics natively.

Can Kubernetes use these exporters?

Yes. All three exporters are commonly deployed in Kubernetes clusters.

Which exporter should be installed on every Linux server?

Node Exporter.

Why is endpoint monitoring important?

It detects customer-facing outages even when infrastructure appears healthy.

Why is container monitoring necessary?

Containers have independent resource consumption patterns that require dedicated visibility.

Summary

Node Exporter, Blackbox Exporter, and cAdvisor form the foundation of infrastructure observability in Prometheus ecosystems.

• Node Exporter provides host-level visibility.
• Blackbox Exporter provides external service availability monitoring.
• cAdvisor provides container-level observability.

Together they enable comprehensive monitoring across physical servers, virtual machines, containers, APIs, and customer-facing services. Enterprise organizations rely on these exporters to build scalable observability platforms, implement proactive alerting, reduce downtime, and maintain high availability systems.