Mastering ArgoCD ApplicationSets for Dynamic Deployments

In enterprise Kubernetes environments, managing configurations across dozens of clusters, hundreds of microservices, and thousands of environments presents a significant operational challenge. Traditional GitOps patterns, such as the "App of Apps" pattern, require substantial manual overhead to maintain, template, and scale. As your infrastructure footprint grows, editing individual Application manifests or maintaining complex Helm wrapper charts becomes a bottleneck.

ArgoCD ApplicationSets solve this scalability problem. An ApplicationSet is a custom resource managed by the ApplicationSet controller that automates the generation, modification, and deletion of ArgoCD Applications. By leveraging "generators," ApplicationSets scan your Git repositories, cluster registries, SCM providers, or custom APIs to dynamically orchestrate deployments.

Table of Contents

• Introduction
• What You Will Learn
• Prerequisites
• The Problem with Scale: App of Apps vs. ApplicationSets
• ApplicationSet Architecture & Internal Workflow
• Anatomy of an ApplicationSet CRD
• Deep Dive: The Core Generators
• Advanced Feature: Progressive Rollouts
• Production Scenario 1: Multi-Tenant Multi-Cluster via Matrix Generator
• Production Scenario 2: Dynamic Preview Environments via SCM/PR Generator
• Enterprise Best Practices & Security Hardening
• Monitoring, Observability, and Metrics
• Troubleshooting & Common Failure Modes
• Technical Interview Questions & Answers
• Frequently Asked Questions (FAQs)
• Summary & Next Steps

What You Will Learn

This guide provides deep, production-grade coverage of ArgoCD ApplicationSets. By the end of this lesson, you will be able to:

• Design and implement dynamic, multi-cluster GitOps deployment pipelines.
• Understand the internal architecture and reconciliation loop of the ApplicationSet controller.
• Configure and chain complex generators using the Matrix and Merge generators.
• Implement automated preview environments triggered by Git Pull Requests.
• Secure your ApplicationSet controller using RBAC, namespace isolation, and SCM token rotation.
• Configure Progressive Rollouts to orchestrate canary and blue-green deployments across clusters.
• Monitor and troubleshoot ApplicationSet performance using Prometheus metrics and debug logs.

Prerequisites

To fully benefit from this advanced guide, you should have:

• A solid understanding of core Kubernetes concepts (Namespaces, CRDs, RBAC, Secrets).
• Familiarity with foundational ArgoCD concepts (Applications, Sync Policies, Projects). If you are new to ArgoCD, we recommend starting with our ArgoCD Architecture and Declarative Setup guides.
• Access to a Kubernetes cluster (such as minikube, kind, or an enterprise EKS/GKE/AKS cluster) with ArgoCD installed.
• Basic knowledge of YAML syntax, Git workflows, and Helm or Kustomize templating formats.

The Problem with Scale: App of Apps vs. ApplicationSets

In the early stages of GitOps adoption, teams typically manage multiple applications using the App of Apps pattern. In this pattern, a root ArgoCD Application points to a directory containing other ArgoCD Application manifests.

While the App of Apps pattern is simple and effective for small-scale deployments, it introduces significant pain points as the system scales to hundreds of applications and target clusters:

The App of Apps pattern forces platform teams to write and maintain wrapper Helm charts or Kustomize overlays just to generate standard ArgoCD Application manifests. This introduces a "template of templates" anti-pattern that is difficult to debug and maintain. ApplicationSets eliminate this layer of abstraction by handling dynamic generation natively within the ArgoCD control plane.

ApplicationSet Architecture & Internal Workflow

To build reliable GitOps platforms, you must understand how the ApplicationSet controller processes resources and manages state. The ApplicationSet controller runs as a separate pod within the ArgoCD namespace (typically argocd-applicationset-controller).

+---------------------------------------------------------------------------------+
|                                ArgoCD Namespace                                 |
|                                                                                 |
|  +--------------------+                                                         |
|  |  ApplicationSet    |                                                         |
|  |     Resource       |                                                         |
|  +---------+----------+                                                         |
|            |                                                                    |
|            | Reconciles                                                         |
|            v                                                                    |
|  +--------------------+       Queries       +--------------------------------+  |
|  |  ApplicationSet    |====================>| External Sources:              |  |
|  |     Controller     |                     | - Git Repositories (YAML/JSON) |  |
|  +---------+----------+                     | - SCM APIs (GitHub/GitLab PRs) |  |
|            |                                | - Kubernetes Cluster Registry  |  |
|            | Generates / Updates            +--------------------------------+  |
|            v                                                                    |
|  +--------------------+                                                         |
|  | ArgoCD Application |                                                         |
|  |     Resources      |                                                         |
|  +---------+----------+                                                         |
|            |                                                                    |
|            | Reconciles & Deploys                                               |
|            v                                                                    |
+------------+--------------------------------------------------------------------+
             |
             +=============> Target Kubernetes Clusters
                             - Cluster Alpha (Dev)
                             - Cluster Beta (Staging)
                             - Cluster Gamma (Prod)
    

The internal execution loop of the ApplicationSet controller operates as follows:

1. Resource Discovery: The controller watches for creation, modification, or deletion of ApplicationSet CRDs in the ArgoCD namespace.
2. Generator Execution: For each active ApplicationSet, the controller executes the defined generators. This may involve querying the local Kubernetes API for cluster secrets, polling Git repositories for directory structures or config files, or calling external SCM APIs (e.g., GitHub, GitLab) to list pull requests.
3. Parameter Generation: The generators produce a list of key-value pairs (parameters). For example, a Cluster Generator might produce parameters like {{name}} (cluster name) and {{server}} (API server URL).
4. Template Rendering: The controller injects these parameters into the defined template block of the ApplicationSet. It renders a complete, valid ArgoCD Application manifest for each parameter set.
5. Reconciliation & Mutation: The controller compares the rendered Applications with the existing Applications in the cluster:
   • If an Application does not exist, it is created.
   • If an Application exists but differs from the rendered template, it is updated.
   • If an Application exists but its corresponding parameters are no longer returned by the generator, the controller handles it based on the defined reclaimPolicy (either deleting or preserving the orphaned Application).
6. ArgoCD Application Loop: Once the Application resources are generated, the core ArgoCD application controller takes over. It reconciles the state of the target cluster with the Git repository specified in each generated Application.

Anatomy of an ApplicationSet CRD

An ApplicationSet manifest consists of three primary blocks: metadata, generators, and the template. Understanding how these blocks interact is essential for writing clean manifests.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: guestbook-appset
  namespace: argocd
spec:
  # 1. Generators define the data sources and parameter inputs
  generators:
    - list:
        elements:
          - cluster: engineering-dev
            url: https://kubernetes.default.svc
            env: dev
          - cluster: engineering-prod
            url: https://10.0.0.10:6443
            env: prod
  # 2. Template defines the skeleton of the ArgoCD Application to be generated
  template:
    metadata:
      name: '{{cluster}}-guestbook'
    spec:
      project: default
      source:
        repoURL: https://github.com/argoproj/argocd-example-apps.git
        targetRevision: HEAD
        path: guestbook
        helm:
          parameters:
            - name: environment
              value: '{{env}}'
      destination:
        server: '{{url}}'
        namespace: guestbook-{{env}}
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true

In this basic example:

• The List Generator defines a static list of environments. Each element in the list contains three custom parameters: cluster, url, and env.
• The Template block defines a standard ArgoCD Application. Inside the template, double curly braces (e.g., {{cluster}}, {{env}}, {{url}}) reference the parameters generated by the generator.
• During reconciliation, the controller renders and creates two separate Applications: engineering-dev-guestbook (deploying to the local cluster in the guestbook-dev namespace) and engineering-prod-guestbook (deploying to the remote production cluster in the guestbook-prod namespace).

Deep Dive: The Core Generators

Generators are the engine of the ApplicationSet controller. They determine how deployment targets are discovered and how templates are parameterized. Let's explore the core generators used in enterprise environments.

1. List Generator

The List Generator is the simplest generator. It takes a hardcoded list of key-value maps. While it does not offer dynamic discovery, it is useful for migrating legacy systems, defining static environment lists, or onboarding a fixed set of clusters before dynamic discovery is configured.

spec:
  generators:
    - list:
        elements:
          - clusterName: cluster-us-east
            apiServer: https://api.us-east.k8s.local
            replicaCount: "3"
          - clusterName: cluster-us-west
            apiServer: https://api.us-west.k8s.local
            replicaCount: "5"

Parameters Generated: Any key-value pair defined in the elements array is exposed as a parameter (e.g., {{clusterName}}, {{apiServer}}, {{replicaCount}}). Note that values must be strings.

2. Cluster Generator

The Cluster Generator allows you to target Kubernetes clusters managed by ArgoCD. By default, ArgoCD stores information about managed clusters as Kubernetes Secrets in its installation namespace, labeled with argocd.argoproj.io/secret-type: cluster.

The Cluster Generator queries these secrets and generates parameters based on cluster metadata, labels, and annotations.

spec:
  generators:
    - clusters:
        # Selector filters clusters based on their Secret labels
        selector:
          matchLabels:
            environment: staging
            region: us-west

Standard Parameters Generated:

• {{name}}: The name of the cluster in ArgoCD.
• {{server}}: The target API server URL of the cluster.
• {{metadata.labels.labelName}}: Access to any custom labels defined on the cluster secret.
• {{metadata.annotations.annotationName}}: Access to annotations on the cluster secret.

3. Git Generator

The Git Generator scans a Git repository to discover configurations. It has two modes:

• Directory Mode: Scans the repository for directories matching a specific wildcard path. Useful for monorepos where each folder represents a service.
• File Mode: Scans the repository for specific JSON or YAML files containing configuration parameters. This allows developers to define applications in configuration files without modifying Kubernetes manifests.

Git Directory Generator Example:

spec:
  generators:
    - git:
        repoURL: https://github.com/enterprise/monorepo.git
        revision: main
        directories:
          - path: apps/*

If the repository contains apps/auth-service and apps/payment-service, the Git Directory Generator will generate parameters for both services.

Parameters Generated:

• {{path}}: The full path to the directory (e.g., apps/auth-service).
• {{path.basename}}: The name of the leaf directory (e.g., auth-service). This is often used as the Application name.

Git File Generator Example:

The File Generator reads configuration files from Git. Assume we have a file in our repository at config/apps.json with the following content:

[
  {
    "name": "catalog-api",
    "port": "8080",
    "resources": {
      "limits": { "cpu": "500m", "memory": "512Mi" }
    }
  },
  {
    "name": "cart-service",
    "port": "9090",
    "resources": {
      "limits": { "cpu": "1000m", "memory": "1Gi" }
    }
  }
]

We can ingest this file using the Git File Generator:

spec:
  generators:
    - git:
        repoURL: https://github.com/enterprise/app-configs.git
        revision: HEAD
        files:
          - path: config/apps.json

The controller parses the JSON array and generates parameters for each object. Nested objects can be accessed using dot notation (e.g., {{resources.limits.cpu}}).

4. Matrix Generator

The Matrix Generator is a powerful tool for enterprise multi-cluster deployments. It combines the outputs of two separate generators, producing a Cartesian product (multiplication) of their parameters.

For example, if you want to deploy every application discovered by a Git Generator to every cluster discovered by a Cluster Generator, you use the Matrix Generator.

spec:
  generators:
    - matrix:
        generators:
          # Generator A: Discover target clusters
          - clusters:
              selector:
                matchLabels:
                  tier: production
          # Generator B: Discover applications in Git
          - git:
              repoURL: https://github.com/enterprise/monorepo.git
              revision: HEAD
              directories:
                - path: microservices/*

If you have 3 production clusters and 5 microservices, the Matrix Generator will render 15 separate ArgoCD Applications (3 clusters × 5 microservices = 15 combinations), keeping them in sync automatically.

5. Merge Generator

The Merge Generator is similar to the Matrix Generator but combines parameters based on a shared "merge key" rather than generating a Cartesian product. This is useful for defining default configuration values for all applications while allowing specific clusters or applications to override those defaults.

spec:
  generators:
    - merge:
        mergeKeys:
          - appName
        generators:
          # Generator 1: Define global defaults for apps
          - list:
              elements:
                - appName: billing-service
                  cpuLimit: "500m"
                  memoryLimit: "512Mi"
                - appName: shipping-service
                  cpuLimit: "1000m"
                  memoryLimit: "1Gi"
          # Generator 2: Override configs for specific environments
          - list:
              elements:
                - appName: billing-service
                  cpuLimit: "2000m" # Production override

The Merge Generator merges these lists using appName as the key. The second generator overrides the cpuLimit for billing-service, while shipping-service retains its default values.

Advanced Feature: Progressive Rollouts

Deploying changes to hundreds of clusters simultaneously can lead to widespread outages if a misconfiguration is introduced. ArgoCD ApplicationSets address this risk with Progressive Rollouts.

Progressive Rollouts allow you to group your generated Applications into sequential stages. The controller updates Applications in later stages only after the Applications in earlier stages have successfully synced and reached a Healthy status.

To enable progressive rollouts, you must configure the strategy block within the ApplicationSet spec:

spec:
  # Progressive rollout strategy config
  strategy:
    type: RollingSync
    rollingSync:
      steps:
        # Step 1: Deploy to Canary/Dev clusters first
        - matchExpressions:
            - key: env
              operator: In
              values:
                - dev
                - canary
          # MaxUpdate controls how many apps in this step can be updated concurrently
          maxUpdate: 10%
        
        # Step 2: Deploy to Staging clusters after Step 1 is fully Healthy
        - matchExpressions:
            - key: env
              operator: In
              values:
                - staging
          maxUpdate: 1
          
        # Step 3: Final rollout to Production
        - matchExpressions:
            - key: env
              operator: In
              values:
                - production
          maxUpdate: 20%

When a change is committed to Git, the ApplicationSet controller evaluates the steps in order:

1. It identifies all generated Applications where the label/parameter env is dev or canary. It triggers updates for these Applications, limiting concurrency to 10% of the target group.
2. The controller pauses and monitors the status of the Step 1 Applications. It does not proceed until all Applications in Step 1 are both Synced and Healthy.
3. Once Step 1 is complete, it triggers updates for the Staging Applications in Step 2, updating them one at a time (maxUpdate: 1).
4. Once Staging is fully verified and healthy, the controller rolls out the changes to the Production Applications in Step 3.

If any Application fails to sync or becomes degraded during a step, the rollout halts immediately. This prevents the broken configuration from reaching downstream environments like staging or production.

Production Scenario 1: Multi-Tenant Multi-Cluster via Matrix Generator

Let's walk through a production-grade implementation of a multi-tenant, multi-cluster deployment pattern.

The Goal: We want to deploy a suite of microservices (discovered from Git directories) across multiple Kubernetes clusters (discovered via cluster labels). Additionally, we need to inject environment-specific configurations (such as database endpoints or replica counts) depending on whether the target cluster is labeled as development, staging, or production.

Step 1: Define Cluster Secrets with Labels

First, ensure your target clusters are registered in ArgoCD with appropriate labels. Here is an example manifest for a production cluster secret:

apiVersion: v1
kind: Secret
metadata:
  name: cluster-prod-us-east-secret
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: cluster
    environment: production
    region: us-east
    tenant: core-banking
type: Opaque
stringData:
  name: prod-us-east
  server: https://api.prod-useast.k8s.internal
  config: |
    {
      "bearerToken": "dummy-token-value",
      "tlsClientConfig": {
        "insecure": false,
        "caData": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCg=="
      }
    }

Step 2: Define the Matrix ApplicationSet Manifest

This ApplicationSet uses a Matrix Generator to combine a Cluster Generator (filtering for environment: production or environment: staging) with a Git Generator (scanning our microservices directory).

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: enterprise-core-banking-appset
  namespace: argocd
spec:
  # Sync policy for the ApplicationSet controller itself
  syncPolicy:
    preserveResourcesOnDeletion: false
  
  generators:
    - matrix:
        generators:
          # Generator A: Target clusters labeled with environment
          - clusters:
              selector:
                matchExpressions:
                  - key: environment
                    operator: In
                    values:
                      - staging
                      - production
          
          # Generator B: Scan Git repository for microservice directories
          - git:
              repoURL: https://github.com/enterprise-org/banking-services.git
              revision: main
              directories:
                - path: services/*
  
  template:
    metadata:
      # Dynamic naming pattern ensures unique application names across clusters
      name: '{{metadata.labels.environment}}-{{path.basename}}'
      namespace: argocd
      labels:
        environment: '{{metadata.labels.environment}}'
        region: '{{metadata.labels.region}}'
        tenant: '{{metadata.labels.tenant}}'
        service: '{{path.basename}}'
    spec:
      project: default
      source:
        repoURL: https://github.com/enterprise-org/banking-services.git
        targetRevision: main
        path: '{{path}}'
        # Pass variables dynamically into Helm values
        helm:
          valueFiles:
            - 'values.yaml'
            - 'values-{{metadata.labels.environment}}.yaml'
          parameters:
            - name: global.clusterName
              value: '{{name}}'
            - name: global.region
              value: '{{metadata.labels.region}}'
      destination:
        server: '{{server}}'
        # Deploy to a namespace scoped by tenant and environment
        namespace: '{{metadata.labels.tenant}}-{{metadata.labels.environment}}'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true
          - PrunePropagationPolicy=background
          - PruneLast=true
        retry:
          limit: 5
          backoff:
            duration: 5s
            factor: 2
            maxDuration: 3m0s

How this works in production:

• Dynamic Value Injection: The Helm configuration automatically loads a base values.yaml file, followed by an environment-specific values file (e.g., values-production.yaml) to apply targeted overrides.
• Self-Healing & Pruning: The syncPolicy is configured with prune: true and selfHeal: true. If an engineer manually modifies resources on the target cluster, ArgoCD automatically reverts those changes. If a service directory is deleted from the Git repository, the corresponding ArgoCD Application and its resources are automatically pruned.
• Namespace Isolation: Services are deployed into isolated namespaces matching the pattern tenant-environment (e.g., core-banking-production), enforcing logical boundary controls at the cluster level.

Production Scenario 2: Dynamic Preview Environments via SCM/PR Generator

An elegant use case for ApplicationSets is automated preview environments. When a developer opens a Pull Request (PR), the SCM/PR Generator detects the PR, provisions a temporary namespace, and deploys the feature branch. Once the PR is merged or closed, the environment is automatically torn down.

Step 1: Configure a GitHub Personal Access Token (PAT)

To prevent rate-limiting issues on SCM APIs, you must provide the ApplicationSet controller with an access token. Store this token as a Kubernetes secret in the argocd namespace:

apiVersion: v1
kind: Secret
metadata:
  name: github-token-secret
  namespace: argocd
type: Opaque
stringData:
  token: ghp_ExampleSecretTokenValueDoNotShare1234567

Step 2: Define the SCM Pull Request ApplicationSet

The following manifest scans a target repository for open pull requests. It uses a filter to only deploy PRs that contain a specific label (e.g., preview-env), protecting cluster resources from being exhausted by every draft PR.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: dynamic-preview-environments
  namespace: argocd
spec:
  generators:
    - pullRequest:
        # 1. Target GitHub Repository
        github:
          owner: enterprise-org
          repo: core-payment-service
          # Reference the SCM token for authentication
          tokenRef:
            secretName: github-token-secret
            key: token
        
        # 2. Filter criteria for Pull Requests
        filters:
          - branchMatch: ^.*$
            # Only trigger preview environments for PRs with this label
            labels:
              - preview-env
  
  template:
    metadata:
      # Use the PR number to ensure a unique Application name
      name: 'payment-pr-{{number}}'
      namespace: argocd
    spec:
      project: default
      source:
        repoURL: https://github.com/enterprise-org/core-payment-service.git
        # Target the specific git commit SHA from the Pull Request
        targetRevision: '{{head_sha}}'
        path: kustomize/overlays/preview
        # Dynamically inject the PR branch and SHA into the manifests
        kustomize:
          commonAnnotations:
            preview.enterprise.com/pr-number: '{{number}}'
            preview.enterprise.com/branch: '{{branch}}'
            preview.enterprise.com/commit: '{{head_sha}}'
      destination:
        # Deploy to the local non-production/management cluster
        server: https://kubernetes.default.svc
        # Dynamic namespace for isolation
        namespace: 'payment-pr-{{number}}'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true

The Lifecycle of a Preview Environment:

1. Developer Action: A developer creates a pull request in the core-payment-service repository and adds the label preview-env.
2. Discovery: The SCM controller polls the GitHub API, detects the labeled PR, and extracts metadata parameters, including {{number}}, {{branch}}, and {{head_sha}}.
3. Deployment: The controller renders the template and creates the ArgoCD Application payment-pr-142. ArgoCD provisions the payment-pr-142 namespace and deploys the application resources.
4. Updates: If the developer pushes new commits to the PR, the {{head_sha}} parameter updates. ArgoCD detects the change and triggers an automated sync to deploy the latest code.
5. Teardown: When the PR is merged or closed, the SCM API no longer returns this PR in the active list. The ApplicationSet controller detects that the parameters are gone, deletes the Application payment-pr-142, and prunes the temporary namespace and its resources.

Enterprise Best Practices & Security Hardening

Because ApplicationSets can dynamically generate and manage resources across multiple production clusters, securing and optimizing them is a critical operational concern.

1. Enforce Generator Restrictions and Namespace Isolation

By default, the ApplicationSet controller can create Applications in any namespace and target any cluster registered with ArgoCD. To prevent unauthorized actions, apply the following restrictions:

• Restrict Project Settings: Ensure that the project field in the ApplicationSet template points to an ArgoCD AppProject that restricts destination namespaces and cluster targets. Never use the default AppProject for production workloads.
• Namespace Isolation: Run the ApplicationSet controller with restricted RBAC permissions. Use the --namespace flag to limit its operation to the argocd namespace.

2. Use SCM Rate-Limit Protection

SCM Providers (GitHub, GitLab) enforce strict API rate limits. If you have dozens of SCM or PR generators polling every few minutes, you will quickly exhaust your rate limits, causing deployment failures.

• Always use API Tokens: Never use unauthenticated SCM generators. Unauthenticated requests are limited to 60 requests per hour on GitHub, whereas authenticated requests allow up to 5,000 requests per hour.
• Configure Webhooks: Instead of relying on aggressive polling intervals, configure webhooks from your SCM provider to push events to ArgoCD. This triggers an immediate reconciliation when a PR is opened or updated, allowing you to increase the polling interval to 10–15 minutes as a fallback.

3. Define a Safe Reclaim Policy

The reclaimPolicy determines what happens to generated Applications when their parameters are removed from the generator (for example, when a cluster secret is deleted or a folder is removed from Git).

spec:
  # Options: trash (default) or keep
  reclaimPolicy: keep

• trash: Deletes the generated Application resource. This is the default setting and is ideal for ephemeral workloads like preview environments.
• keep: Preserves the generated Application resource, even if the generator no longer lists it. This is recommended for critical production workloads to prevent accidental mass-deletion if a Git repository or cluster registry is misconfigured.

4. Optimize Template Rendering Performance

When dealing with complex applications, use the goTemplate: true option in your ApplicationSet specification. This enables the Go templating engine, which supports advanced functions (such as string manipulation, loops, and conditionals) and processes templates more efficiently than the default fast-template engine.

spec:
  goTemplate: true
  generators:
    - list:
        elements:
          - name: auth-service
            env: prod
  template:
    metadata:
      name: '{{ .name }}-{{ .env }}' # Uses Go template dot notation

Monitoring, Observability, and Metrics

Operating ApplicationSets at scale requires visibility into the health and performance of the reconciliation loop. The ApplicationSet controller exposes Prometheus metrics on port 8080 (or 8082 depending on configuration) under the /metrics path.

Critical Prometheus Metrics to Watch:

Configuring a ServiceMonitor for Prometheus Operator:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: argocd-applicationset-controller-monitor
  namespace: argocd
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: argocd-applicationset-controller
  endpoints:
    - port: metrics
      interval: 30s
      path: /metrics

Troubleshooting & Common Failure Modes

When working with dynamic generators, debugging issues can be challenging because errors can occur at multiple levels: during parameter generation, template rendering, or Application reconciliation.

Step-by-Step Debugging Workflow

1. Inspect the ApplicationSet Status and Events

Your first step should always be to inspect the ApplicationSet resource itself. The controller writes events and error details directly to the status block of the CRD.

kubectl describe appset enterprise-core-banking-appset -n argocd

Look at the Events section at the bottom of the output. Common errors like template syntax issues, missing secrets, or failed SCM authentication will be listed here:

Warning ReconcileError 12s applicationset-controller failed to parse template: template: :1: undfined variable "$env"

2. Examine Controller Logs

If the events do not provide enough context, inspect the controller logs. You can increase the log level by editing the deployment and setting the --loglevel flag to debug.

kubectl logs -f deployment/argocd-applicationset-controller -n argocd

Look for reconciliation errors or API rate-limiting warnings:

level=error msg="API rate limit exceeded for github user" generator="PullRequest" appset="dynamic-preview-environments"

3. Verify SCM Token Permissions

If your Pull Request or SCM generators are not producing any applications:

• Verify that the SCM token stored in the Kubernetes secret has not expired.
• Ensure the token has the necessary scopes. For GitHub, it requires repo (for private repositories) or public_repo.
• Test the token manually from your terminal using curl to ensure the SCM API is reachable and authorized.

4. Check for Orphaned Applications

If you deleted an ApplicationSet but its generated Applications are still running (or vice-versa), inspect the ownerReferences of the generated Application: