Terraform Provisioners: Theory, Internals, and Production Reality

Terraform provisioners are special blocks used to execute imperative commands during the lifecycle of a Terraform-managed resource. They are usually used when infrastructure must be bootstrapped after creation, such as installing packages, running scripts, registering a server, triggering Ansible, or performing cleanup actions before destruction.

However, provisioners are different from normal Terraform resources. Terraform resources are declarative and state-aware. Provisioners are imperative and mostly state-blind. This difference is the main reason provisioners must be used carefully in production environments.

Why Provisioners Exist

Terraform was designed to create and manage infrastructure resources such as virtual machines, networks, databases, IAM roles, DNS records, Kubernetes objects, and cloud services. But sometimes infrastructure creation alone is not enough.

Example situations:

• A VM is created, but you need to install Nginx.
• An EC2 instance is created, but you need to run a bootstrap script.
• A server is created, but you need to register it in monitoring.
• A resource is destroyed, but you need to call an external cleanup API.
• Terraform creates infrastructure, then Ansible configures software.

Provisioners were introduced to handle these edge cases. They provide a bridge between infrastructure provisioning and operational scripting.

Terraform Resource Creation
        │
        ▼
Cloud Provider Creates Infrastructure
        │
        ▼
Provisioner Executes Script or Command
        │
        ▼
External Configuration / Bootstrap Action
    

The Main Types of Terraform Provisioners

1. local-exec

The local-exec provisioner runs commands on the machine where Terraform is executed. This could be a developer laptop, Jenkins agent, GitHub Actions runner, GitLab runner, or Terraform Cloud worker.

provisioner "local-exec" {
  command = "echo ${self.public_ip} >> inventory.txt"
}

Common uses include generating local inventory files, triggering Ansible, sending notifications, calling APIs, or updating CMDB systems.

2. remote-exec

The remote-exec provisioner connects to a remote machine and executes commands on that machine. It usually uses SSH for Linux and WinRM for Windows.

provisioner "remote-exec" {
  inline = [
    "sudo apt-get update -y",
    "sudo apt-get install -y nginx",
    "sudo systemctl start nginx"
  ]
}

This requires a connection block because Terraform needs to know how to access the remote server.

3. file Provisioner

The file provisioner copies files or directories from the Terraform machine to the remote resource.

provisioner "file" {
  source      = "app.conf"
  destination = "/tmp/app.conf"
}

Provisioner Lifecycle: When Do They Run?

Provisioners usually run after a resource is created. These are called creation-time provisioners.

terraform apply
        │
        ▼
Resource Created by Provider
        │
        ▼
Terraform Receives Resource ID
        │
        ▼
Provisioner Executes
        │
        ▼
Provisioner Success?
        │
        ├── Yes → State Updated Normally
        │
        └── No  → Resource Marked Tainted
    

Provisioners can also run during destruction by using when = destroy.

provisioner "local-exec" {
  when    = destroy
  command = "echo Resource is being destroyed"
}

What Happens If a Provisioner Fails?

This is one of the most important production concepts. If a creation-time provisioner fails, Terraform may mark the resource as tainted. A tainted resource is considered unreliable because Terraform created the resource, but the post-creation configuration did not complete successfully.

VM Created Successfully
        │
        ▼
Provisioner Fails
        │
        ▼
Terraform Marks Resource as Tainted
        │
        ▼
Next terraform apply
        │
        ▼
Terraform Plans Replacement
    

This can be dangerous in production. If a provisioner fails on a critical VM, Terraform may try to replace it in the next apply unless the issue is handled carefully.

Why Provisioners Are Considered a Last Resort

Provisioners are not bad, but they are risky because they operate outside Terraform's normal declarative model.

Terraform can track this:

resource "aws_instance" "web" {
  instance_type = "t3.micro"
}

But Terraform cannot fully track this:

provisioner "remote-exec" {
  inline = [
    "sudo apt install nginx -y",
    "sudo useradd appuser",
    "sudo chmod 755 /app"
  ]
}

Terraform does not understand the final OS-level state created by these commands. It does not know the exact package version, user state, file permissions, or service health unless you explicitly model those elsewhere.

Production-Level Problem: Provisioners Are Not Idempotent by Default

Idempotency means running the same operation multiple times produces the same result safely. Terraform resources are designed to be idempotent. Shell scripts are not automatically idempotent.

Bad Non-Idempotent Script

provisioner "remote-exec" {
  inline = [
    "echo 'server configured' >> /etc/app.conf"
  ]
}

If this runs multiple times, it appends duplicate lines.

Better Idempotent Script

provisioner "remote-exec" {
  inline = [
    "grep -q 'server configured' /etc/app.conf || echo 'server configured' >> /etc/app.conf"
  ]
}

Production scripts must check current state before making changes.

local-exec: Production Explanation

local-exec is useful when Terraform needs to interact with systems outside the created resource.

Production Use Cases

• Triggering an Ansible playbook after VM creation.
• Generating a dynamic inventory file.
• Sending deployment notifications to Slack or Teams.
• Updating a CMDB with resource metadata.
• Calling internal approval or audit APIs.
• Running post-deployment validation scripts.

Example: Generate Ansible Inventory

resource "aws_instance" "app" {
  ami           = var.ami_id
  instance_type = "t3.micro"

  provisioner "local-exec" {
    command = "echo '${self.public_ip} ansible_user=ubuntu' >> inventory.ini"
  }
}

Production Risk

This depends on the machine running Terraform. If Terraform runs on a laptop, the file is created on the laptop. If Terraform runs in GitHub Actions, the file is created inside the temporary runner. That means behavior changes based on execution environment.

remote-exec: Production Explanation

remote-exec is more fragile than local-exec because it requires remote connectivity.

For remote-exec to work, all of these must be ready:

• The VM must be created.
• The operating system must finish booting.
• SSH or WinRM must be running.
• Firewall rules must allow access.
• Security groups must allow access.
• Private key or password must be correct.
• The Terraform runner must be able to reach the VM network.

Production-Ready remote-exec Example

resource "aws_instance" "web_server" {
  ami                    = var.ami_id
  instance_type          = "t3.micro"
  key_name               = var.key_name
  vpc_security_group_ids = [aws_security_group.ssh_access.id]
  subnet_id              = aws_subnet.public.id

  connection {
    type        = "ssh"
    user        = "ubuntu"
    private_key = file(var.private_key_path)
    host        = self.public_ip
    timeout     = "5m"
  }

  provisioner "remote-exec" {
    inline = [
      "cloud-init status --wait",
      "sudo apt-get update -y",
      "sudo apt-get install -y nginx",
      "sudo systemctl enable nginx",
      "sudo systemctl start nginx",
      "sudo systemctl status nginx --no-pager"
    ]
  }
}

This is better because it waits for cloud-init before installing packages.

Better Production Alternative: user_data and cloud-init

For VM bootstrapping, prefer cloud-init or cloud provider metadata scripts over remote-exec.

resource "aws_instance" "web" {
  ami           = var.ami_id
  instance_type = "t3.micro"

  user_data = <<-EOF
    #!/bin/bash
    set -e
    apt-get update -y
    apt-get install -y nginx
    systemctl enable nginx
    systemctl start nginx
  EOF
}

Why this is better:

• No SSH connection required from Terraform.
• Runs naturally during instance boot.
• Works better with autoscaling.
• Reduces security exposure.
• Improves repeatability.

Best Production Alternative: Immutable Infrastructure

The strongest production pattern is immutable infrastructure.

Packer Builds Golden AMI
        │
        ▼
AMI Contains Required Packages
        │
        ▼
Terraform Deploys AMI
        │
        ▼
No Runtime Shell Provisioning
        │
        ▼
Application Deployed via CI/CD or Kubernetes
    

In this approach, Terraform does not install packages during deployment. It simply deploys prebuilt images.

Decision Matrix: Provisioners vs Alternatives

Internal Links for Better SEO and Learning Flow