Secrets Management in GitOps with Sealed Secrets

Table of Contents

• 1. Introduction to GitOps Secrets Management
• 2. What You Will Learn
• 3. Prerequisites
• 4. The GitOps Secret Dilemma: Why Plain Secrets Fail
• 5. What are Sealed Secrets? (Featured Snippet)
• 6. Cryptographic Architecture & Deep-Dive Workflows
• 7. Enterprise Installation & Controller Setup
• 8. Step-by-Step Implementation Guide
• 9. Deep Dive: Sealed Secrets Scopes (Strict, Namespace, Cluster)
• 10. Sealing Key Lifecycle & Rotation Strategies
• 11. Seamless ArgoCD Integration Patterns
• 12. Architectural Comparison: Sealed Secrets vs. ESO vs. Vault vs. SOPS
• 13. Disaster Recovery & Key Backup Procedures
• 14. Production Monitoring, Alerting & Observability
• 15. Troubleshooting & Common Failure Modes
• 16. Technical Interview Questions & Detailed Answers
• 17. Frequently Asked Questions (FAQs)
• 18. Summary & Next Steps

1. Introduction to GitOps Secrets Management

In a modern cloud-native enterprise, the adoption of GitOps has revolutionized how infrastructure and application states are managed. By treating Git as the single source of truth, engineering teams achieve unparalleled auditability, rapid disaster recovery, and consistent continuous delivery. However, this paradigm introduces a critical, non-trivial security challenge: How do we manage sensitive data (API keys, database credentials, TLS certificates, and OAuth tokens) when our entire cluster state is declared in public or private Git repositories?

Storing raw, unencrypted Kubernetes Secret manifests in a Git repository is a catastrophic security vulnerability. Even if the repository is private, credentials are leaked to anyone with read access to the repository, history logs remain permanently exposed, and compliance standards such as PCI-DSS, SOC2, and HIPAA are immediately violated.

To bridge this gap, modern platform engineers leverage Bitnami Sealed Secrets. This lesson provides an exhaustive, production-grade guide to designing, implementing, and operating a secure secrets management pipeline using Sealed Secrets inside an ArgoCD-driven enterprise GitOps ecosystem.

2. What You Will Learn

By the end of this highly detailed, hands-on masterclass, you will have mastered:

• The core cryptographic mechanisms powering Bitnami Sealed Secrets, including asymmetric envelope encryption.
• How to install, configure, and secure the Sealed Secrets controller in a multi-tenant Kubernetes cluster.
• The execution of the developer workflow: converting native Kubernetes Secrets into safe-for-Git SealedSecret Custom Resources using the kubeseal CLI.
• How to design and enforce strict isolation boundaries using Sealed Secrets Scopes (strict, namespace-wide, and cluster-wide).
• Enterprise lifecycle management, including automated and manual key rotation, backup strategies, and disaster recovery.
• Integrating Sealed Secrets into automated CI/CD pipelines and ArgoCD Application manifests.
• Setting up production-grade observability, Prometheus metrics, and troubleshooting real-world decryption failures.

3. Prerequisites

To successfully follow and implement the configurations in this lesson, you should have:

• A solid understanding of basic GitOps principles. If you are new, consider reviewing our guide on GitOps Principles and Core Concepts.
• An active Kubernetes cluster (v1.22 or higher) with administrative access (cluster-admin privileges).
• A working installation of ArgoCD. Learn how to set this up in Installing ArgoCD in Production.
• The kubectl CLI tool installed and configured on your workstation.
• Basic familiarity with asymmetric cryptography (public and private keys).

4. The GitOps Secret Dilemma: Why Plain Secrets Fail

To understand why tools like Sealed Secrets are mandatory, we must first deconstruct the native Kubernetes Secret resource. Beginners often mistake Kubernetes Secrets for secure, encrypted storage mechanisms. In reality, native Kubernetes Secrets are merely Base64 encoded.

Base64 is a reversible encoding scheme designed to handle binary data transmission, not encryption. Anyone who intercepts or reads a Base64 string can instantly decode it back to plain text. For example, decoding a secret is as simple as running:

echo "bXktc3VwZXItc2VjcmV0LXBhc3N3b3Jk" | base64 --decode

If you commit a standard Kubernetes Secret manifest to a Git repository, you are effectively publishing your raw credentials in plain text. Even if you delete the file in a subsequent commit, the secret remains permanently accessible within the Git commit history (reflog).

This creates a fundamental conflict with GitOps:

The GitOps Conflict: GitOps demands that 100% of the desired cluster state be declared in Git. Security demands that 100% of secrets remain strictly out of Git.

To resolve this, we need a mechanism that allows us to commit encrypted files to Git that only our target Kubernetes cluster can decrypt. This is exactly where Bitnami Sealed Secrets excels.

5. What are Sealed Secrets?

Bitnami Sealed Secrets is an open-source, Kubernetes-native secrets decryption system designed specifically for GitOps. It solves the GitOps secret dilemma by utilizing asymmetric cryptography to encrypt sensitive Kubernetes Secrets into a custom resource called a SealedSecret, which is safe to commit to public or private Git repositories.

The system consists of two primary components:

1. The Cluster-Side Controller: A Kubernetes controller running inside your cluster that holds the private key. It watches for SealedSecret resources and decrypts them back into native, standard Kubernetes Secret resources in real-time.
2. The Client-Side CLI (kubeseal): A command-line utility used by developers to encrypt standard Kubernetes Secret manifests using the controller's public key.

Because only the controller running in the secure boundary of the Kubernetes cluster possesses the private key required for decryption, the encrypted SealedSecret manifest is completely useless to unauthorized users, making it perfectly safe to store in Git.

6. Cryptographic Architecture & Deep-Dive Workflows

Sealed Secrets relies on asymmetric cryptography (specifically, RSA-OAEP with SHA-256) combined with symmetric cryptography (AES-GCM) to implement a highly secure envelope encryption model. Let's break down the exact operational workflows of encryption and decryption.

The Encryption Workflow (Developer Workstation)

When a developer encrypts a secret, the following sequence of events occurs:

1. The developer creates a standard Kubernetes Secret manifest locally.
2. The developer runs the kubeseal command-line tool.
3. kubeseal fetches the public key from the Sealed Secrets controller (or uses a locally cached copy of the public key).
4. The CLI generates a unique, one-time symmetric session key using AES-256-GCM.
5. The actual secret values are encrypted with this symmetric session key.
6. The symmetric session key itself is then encrypted using the controller's RSA public key.
7. The encrypted session key and the encrypted secret data are packaged together into a SealedSecret Custom Resource (CR) manifest.
8. The developer safely commits the SealedSecret manifest to the Git repository.

+---------------------------------------------------------------------------------+
|                              DEVELOPER WORKSTATION                              |
|                                                                                 |
|  +------------------+      +-------------------+                                |
|  |  Kubernetes      |      | Public Key        |                                |
|  |  Secret (Plain)  |      | (Fetched/Cached)  |                                |
|  +--------+---------+      +---------+---------+                                |
|           |                          |                                          |
|           |                          |                                          |
|           v                          v                                          |
|     +--------------------------------------+                                    |
|     |            kubeseal CLI              |                                    |
|     |  - Generates AES symmetric key       |                                    |
|     |  - Encrypts Secret with AES key      |                                    |
|     |  - Encrypts AES key with Public Key  |                                    |
|     +------------------+-------------------+                                    |
|                        |                                                        |
|                        v                                                        |
|            +-----------------------+                                            |
|            | SealedSecret Manifest |                                            |
|            |     (Safe for Git)    |                                            |
|            +-----------+-----------+                                            |
+------------------------|--------------------------------------------------------+
                         |
                         v  [Push to Git & Synced via ArgoCD]
+------------------------|--------------------------------------------------------+
|                        v                                                        |
|                  +-----------+                                                  |
|                  | Git Repo  |                                                  |
|                  +-----+-----+                                                  |
|                        |                                                        |
|                        v  [ArgoCD Applies Manifest]                             |
|             +---------------------+                                             |
|             | Kubernetes Cluster  |                                             |
|             +----------+----------+                                             |
|                        |                                                        |
+------------------------|--------------------------------------------------------+
                         |
                         v
+---------------------------------------------------------------------------------+
|                        KUBERNETES CLUSTER BOUNDARY                              |
|                                                                                 |
|                        v                                                        |
|             +---------------------+                                             |
|             | SealedSecret CRD    |                                             |
|             +----------+----------+                                             |
|                        |                                                        |
|                        v                                                        |
|         +------------------------------+                                        |
|         |   Sealed Secrets Controller  | <---+ [Reads Private Key]              |
|         |  - Decrypts AES key via RSA  |     |                                  |
|         |  - Decrypts Secret via AES   |  +--+---------------+                  |
|         +--------------+---------------+  | Private Key      |                  |
|                        |                  | (Secure Storage) |                  |
|                        v                  +------------------+                  |
|             +---------------------+                                             |
|             | Native K8s Secret   |                                             |
|             | (In-Memory Decrypt) |                                             |
|             +---------------------+                                             |
+---------------------------------------------------------------------------------+
    

The Decryption Workflow (Kubernetes Cluster)

Once the SealedSecret custom resource is applied to the cluster (typically via ArgoCD), the cluster-side controller takes over:

1. The Sealed Secrets Controller constantly watches the cluster API server for the creation or modification of SealedSecret resources.
2. Upon detecting a new SealedSecret, the controller inspects the metadata to identify which private key is required to decrypt it (based on the key's active generation).
3. The controller uses its secure in-cluster RSA private key to decrypt the symmetric session key.
4. Using the decrypted symmetric session key, the controller decrypts the actual secret payload.
5. The controller dynamically generates a standard, native Kubernetes Secret resource in the exact same namespace.
6. The controller configures an ownerReference on the generated Secret pointing back to the parent SealedSecret. This ensures that if the SealedSecret is deleted, the native decrypted Secret is automatically garbage-collected.

7. Enterprise Installation & Controller Setup

To deploy Sealed Secrets in an enterprise environment, we will install the controller using Helm and configure it for production. It is highly recommended to pin your installations to specific, immutable version tags to ensure cluster stability.

Step 1: Install the Sealed Secrets Controller via Helm

We will deploy the controller into a dedicated namespace called kube-system (or a custom sealed-secrets namespace) using the official Helm chart.

# Add the Helm repository
helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets

# Update local repository cache
helm repo update

# Install the chart with production-grade configurations
helm upgrade --install sealed-secrets sealed-secrets/sealed-secrets \
  --namespace kube-system \
  --set fullnameOverride=sealed-secrets-controller \
  --set rbac.create=true \
  --set ingress.enabled=false

Step 2: Install the kubeseal CLI on your Workstation

The client-side CLI tool, kubeseal, must be installed on developer workstations and CI/CD runners. Choose the appropriate installation command for your operating system:

macOS (via Homebrew):

brew install kubeseal

Linux (Direct Binary Download):

# Fetch the latest release version
KUBESEAL_VERSION=$(curl -s https://api.github.com/repos/bitnami-labs/sealed-secrets/releases/latest | grep tag_name | cut -d '"' -f 4 | sed 's/v//')

# Download the tarball
curl -OL "https://github.com/bitnami-labs/sealed-secrets/releases/download/v${KUBESEAL_VERSION}/kubeseal-${KUBESEAL_VERSION}-linux-amd64.tar.gz"

# Extract the binary
tar -xvzf "kubeseal-${KUBESEAL_VERSION}-linux-amd64.tar.gz" kubeseal

# Move to a system-wide PATH location
sudo install -m 755 kubeseal /usr/local/bin/kubeseal

# Verify installation
kubeseal --version

Step 3: Retrieve and Cache the Public Key

By default, kubeseal attempts to connect to your Kubernetes cluster to fetch the public certificate on every encryption operation. In a secure enterprise environment, developers may not have direct access to the cluster API server. Therefore, you should fetch the public key once, distribute it to developers, or store it in your CI/CD pipelines.

# Retrieve the public key certificate from the running controller
kubeseal --controller-name=sealed-secrets-controller \
         --controller-namespace=kube-system \
         --fetch-cert > pub-sealed-secrets.pem

The generated pub-sealed-secrets.pem file is completely public. It can be safely checked into your Git repository alongside your code, allowing any developer or automated pipeline to encrypt secrets without needing direct access to the Kubernetes cluster.

8. Step-by-Step Implementation Guide

Let's walk through the complete lifecycle of creating, encrypting, committing, and validating a secret in a GitOps pipeline.

Step 1: Create a Local, Standard Kubernetes Secret Manifest

First, we define a standard Kubernetes Secret on our local machine containing sensitive database credentials. Never commit this file to Git!

Create a file named db-secret-raw.yaml:

<code>apiVersion: v1
kind: Secret
metadata:
  name: database-credentials
  namespace: production
type: Opaque
stringData:
  DB_USER: admin_user
  DB_PASS: SuperSecretComplexPassword2026!
</code>

Step 2: Encrypt the Secret Using kubeseal

Using the kubeseal CLI and the public certificate we fetched earlier, we will encrypt the raw secret into a SealedSecret custom resource.

# Encrypt using the local public certificate to prevent cluster calls
kubeseal --cert pub-sealed-secrets.pem \
         --format yaml \
         < db-secret-raw.yaml \
         > db-sealed-secret.yaml

Once the db-sealed-secret.yaml is generated, you must immediately delete the raw, unencrypted db-secret-raw.yaml file from your workstation:

rm db-secret-raw.yaml

Step 3: Analyze the SealedSecret Manifest Anatomy

Let's examine the contents of the newly generated db-sealed-secret.yaml file to understand its structure:

<code>apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: database-credentials
  namespace: production
spec:
  encryptedData:
    DB_PASS: AgByS3v...[TRUNCATED CRYPTOGRAPHIC DATA]...F9g==
    DB_USER: AgBvP9x...[TRUNCATED CRYPTOGRAPHIC DATA]...1a==
  template:
    metadata:
      name: database-credentials
      namespace: production
    type: Opaque
</code>

Notice the following crucial architectural properties:

• The kind is changed from Secret to SealedSecret.
• The apiVersion points to the Custom Resource Definition (CRD) owned by the controller: bitnami.com/v1alpha1.
• The sensitive fields inside the encryptedData block are completely encrypted. They are not Base64 encoded; they are ciphertexts generated using AES-GCM and RSA-OAEP. They are 100% safe to commit to public Git repositories.
• The template section defines the metadata and structure that the controller will use when generating the native Kubernetes Secret after decryption.

Step 4: Deploy the SealedSecret and Verify Decryption

Apply the SealedSecret manifest directly to the cluster to test the controller's decryption process:

kubectl apply -f db-sealed-secret.yaml

Now, verify that the controller detected the resource and successfully generated the native Secret:

# Check if the SealedSecret exists
kubectl get sealedsecret database-credentials -n production

# Check if the native Secret was automatically created
kubectl get secret database-credentials -n production -o yaml

Let's verify that the decrypted data matches our original raw values:

kubectl get secret database-credentials -n production -o jsonpath='{.data.DB_PASS}' | base64 --decode

The output will print: SuperSecretComplexPassword2026!. The decryption was successful!

9. Deep Dive: Sealed Secrets Scopes

One of the most powerful and critical security features of Bitnami Sealed Secrets is its scoping mechanism. Scopes prevent a highly dangerous attack vector: Secret Hijacking.

Without scopes, a malicious user with access to the Git repository could copy an encrypted secret payload from the production namespace, paste it into a SealedSecret manifest targeting their own development namespace (e.g., malicious-dev), deploy it, and read the decrypted value from their namespace.

To prevent this, Sealed Secrets enforces three distinct scopes. During encryption, the metadata (name and namespace) is cryptographically bound (using authenticated encryption AAD) to the ciphertext.

Example: Encrypting with an Explicit Scope

To enforce a namespace-wide scope during encryption, pass the --scope flag to the kubeseal command:

kubeseal --cert pub-sealed-secrets.pem \
         --scope namespace-wide \
         --format yaml \
         < db-secret-raw.yaml \
         > db-sealed-secret-ns-scoped.yaml

If a developer attempts to modify the metadata.namespace field in the output YAML of a strict or namespace-wide scoped SealedSecret, the controller will fail to decrypt it, throwing a cryptographic error in its logs and preventing unauthorized access.

10. Sealing Key Lifecycle & Rotation Strategies

A robust security posture requires regular cryptographic key rotation. If a private key remains active indefinitely, the blast radius of a potential compromise increases over time.

Automatic Key Rotation

By default, the Bitnami Sealed Secrets controller automatically generates a new 2048-bit RSA key pair every 30 days.

• When a new key is generated, it becomes the active encryption key. Any new execution of kubeseal (without an offline cert) that queries the cluster will retrieve this new active public key.
• All historical private keys are retained permanently in the cluster as Kubernetes Secrets in the controller's namespace.
• When the controller processes a SealedSecret, it automatically identifies which key generation was used to encrypt it, retrieves the corresponding historical private key, and performs decryption.
• This means existing Sealed Secrets committed to Git do not break when a key rotation occurs. They remain fully decryptable using the historical keys stored in the cluster.

Manual Key Rotation (Emergency Rotation)

If you suspect that a private key has been compromised (e.g., an administrator's backup of the private key was leaked), you must execute an emergency manual key rotation immediately.

To force the creation of a new key pair, trigger the controller to generate a new secret:

# Tell the controller to generate a new key by sending a USR1 signal to the process
kubectl exec -it -n kube-system deployment/sealed-secrets-controller -- kill -USR1 1

Alternatively, you can manually create a new certificate secret using your own custom RSA keys, or restart the deployment after deleting the active secret (if safe).

Re-encryption of Old Secrets (Best Practice)

After a key rotation (especially after a compromise), you should re-encrypt your old SealedSecret manifests with the new active public key. The kubeseal CLI provides a built-in command to automate this without needing the raw plain text secrets:

# Re-encrypt an existing SealedSecret using the newest public key
kubeseal --cert new-pub-sealed-secrets.pem \
         --re-encrypt \
         < old-sealed-secret.yaml \
         > new-sealed-secret.yaml

Commit the updated new-sealed-secret.yaml to Git. This ensures that old private keys can eventually be safely deleted from the cluster once no active manifests rely on them.

11. Seamless ArgoCD Integration Patterns

Because SealedSecret resources are standard Kubernetes Custom Resources, they integrate perfectly into ArgoCD without requiring any special plugins or custom tooling. ArgoCD simply treats them as declarative manifests.

Production GitOps Directory Structure

A recommended enterprise GitOps repository layout separating configuration from secrets:

my-gitops-infra/
├── apps/
│   └── database/
│       ├── kustomization.yaml
│       ├── deployment.yaml
│       ├── service.yaml
│       └── db-sealed-secret.yaml  <-- Safe to store in Git!
└── bootstrap/
    └── argocd-apps/
        └── database-app.yaml
    

The ArgoCD Application Manifest

Define your ArgoCD Application to watch the repository. ArgoCD will synchronize the SealedSecret to the cluster, where the Sealed Secrets controller will immediately decrypt it.

Create a file named database-app.yaml:

<code>apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: production-database
  namespace: argocd
spec:
  project: default
  source:
    repoURL: 'https://github.com/enterprise-org/my-gitops-infra.git'
    targetRevision: HEAD
    path: apps/database
  destination:
    server: 'https://kubernetes.default.svc'
    namespace: production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
</code>

Handling Synchronization Race Conditions

A common issue in GitOps is when an application pod starts up before the Sealed Secrets controller has finished decrypting the SealedSecret into a native Secret. The pod will fail to start, throwing a CreateContainerConfigError.

To solve this, configure ArgoCD Sync Waves. We can force ArgoCD to apply the SealedSecret first, wait for the cluster state to settle, and then deploy the application pods.

Add the sync-wave annotation to your db-sealed-secret.yaml:

<code>apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: database-credentials
  namespace: production
  annotations:
    argocd.argoproj.io/sync-wave: "-5"  # Deployed early in wave -5
spec:
  # ... encryptedData ...
</code>

Then, configure your deployment to run in wave 0 or higher:

<code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: database-app
  namespace: production
  annotations:
    argocd.argoproj.io/sync-wave: "0"   # Deployed after secrets are ready
# ... spec ...
</code>

12. Architectural Comparison: Sealed Secrets vs. Alternatives

Choosing the right secrets management tool depends heavily on your enterprise architecture, compliance constraints, and operational complexity.

13. Disaster Recovery & Key Backup Procedures

In a disaster recovery (DR) scenario where your entire Kubernetes cluster is lost, having your Sealed Secrets in Git is completely useless if you lose your private keys. If you rebuild the cluster from scratch and deploy the Sealed Secrets, the new controller will generate a new private key and will be completely unable to decrypt your existing manifests in Git.

Therefore, a secure, offline backup of the controller's private keys is mandatory.

Step 1: Locate and Backup the Master Sealing Key

The Sealed Secrets controller stores its RSA key pairs as standard Kubernetes Secrets with specific labels inside its installation namespace (e.g., kube-system).

# Find all active sealing key secrets in the cluster
kubectl get secrets -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key

The output will list secrets named like sealed-secrets-key-xxxx.

Export all of these secrets to a secure, encrypted backup file:

kubectl get secrets -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml > master-sealing-keys-backup.yaml

CRITICAL SECURITY WARNING: The master-sealing-keys-backup.yaml file contains the raw, unencrypted private keys. Anyone who gains access to this file can decrypt every single Sealed Secret in your Git repositories. Store this file in an enterprise-grade offline vault (e.g., HashiCorp Vault, AWS KMS, 1Password Business, or a physical hardware security module).

Step 2: Restoring Sealing Keys to a Fresh Cluster

If your cluster is destroyed, follow this exact sequence to restore decryption capabilities:

1. Do not install the Sealed Secrets controller yet.
2. Create the target namespace if it doesn't exist: kubectl create namespace kube-system.
3. Apply your backed-up master sealing keys:

   kubectl apply -f master-sealing-keys-backup.yaml -n kube-system

4. Now, install the Sealed Secrets controller using Helm.
5. The controller will start up, scan the namespace, detect the pre-existing master keys, and adopt them instead of generating a new key pair.
6. Deploy your GitOps repositories. ArgoCD will apply the SealedSecrets, and the controller will successfully decrypt them using the restored keys.

14. Production Monitoring, Alerting & Observability

To operate Sealed Secrets reliably at scale, you must monitor the controller's health and decryption performance. The controller exposes standard Prometheus metrics on port 8080 at the /metrics endpoint.

Key Metrics to Monitor

• sealed_secrets_key_renewals_total: Counter tracking the number of times the sealing key has been rotated.
• sealed_secrets_decryption_failures_total: Counter tracking failed decryption attempts. A sudden spike indicates misconfigured secrets or a potential security intrusion.
• sealed_secrets_decryption_duration_seconds: Histogram tracking how long decryption operations take.

Prometheus ServiceMonitor Configuration

To enable Prometheus Operator to scrape metrics, deploy the following ServiceMonitor:

<code>apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: sealed-secrets-monitor
  namespace: kube-system
  labels:
    release: prometheus-stack
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: sealed-secrets
  endpoints:
    - port: http
      path: /metrics
      interval: 30s
</code>

Prometheus Alerting Rules

Deploy these alerts to proactively catch failures before they impact your applications:

<code>apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: sealed-secrets-alerts
  namespace: kube-system
spec:
  groups:
  - name: sealed-secrets.rules
    rules:
    - alert: SealedSecretsDecryptionFailed
      expr: rate(sealed_secrets_decryption_failures_total[5m]) > 0
      for: 2m
      labels:
        severity: critical
      annotations:
        summary: "Sealed Secrets decryption failing in cluster"
        description: "The Sealed Secrets controller has failed to decrypt resources over the last 2 minutes. Check controller logs immediately."
</code>

15. Troubleshooting & Common Failure Modes

Here are the most common errors encountered when using Sealed Secrets in production, along with exact diagnostic and remediation steps.

Case 1: The "crypto/rsa: decryption error" Error

Symptom: The controller logs show:

bad decrypt: crypto/rsa: decryption error

And the native Secret is not generated.

Root Cause: This is almost always caused by one of the following:

• The SealedSecret was encrypted with a public key that does not exist in the current cluster's private key store (e.g., you deployed to a new cluster without restoring the master keys).
• The scope was violated. For example, the secret was encrypted using the strict scope for namespace production, but you deployed it to namespace staging.
• The resource name was changed after encryption.

Remediation:

1. Verify the namespace and name match exactly what was defined in the raw secret before encryption.
2. Check the active certificates in the cluster using:

   kubeseal --controller-name=sealed-secrets-controller --controller-namespace=kube-system --fetch-cert

   And compare it against the certificate used for encryption.

Case 2: SealedSecret Status is "Unrecoverable" or Pending

Symptom: Running kubectl describe sealedsecret <name> shows a failing condition.

Root Cause: The YAML structure of the SealedSecret template might be invalid, or the controller lacks RBAC permissions to create secrets in the target namespace.

Remediation:

1. Inspect the controller logs:

   kubectl logs -n kube-system -l app.kubernetes.io/name=sealed-secrets

2. Verify that the controller service account has ClusterRole bindings allowing it to create, update, and patch secrets cluster-wide.

Case 3: Decrypted Secret Does Not Update When SealedSecret Changes

Symptom: You committed an updated SealedSecret to Git, ArgoCD synchronized it successfully, but the decrypted Secret still contains the old values.

Root Cause: The controller might have failed to decrypt the new version (falling back to the old secret to prevent application downtime), or the synchronization loop is blocked.

Remediation: Check the controller logs for decryption errors at the timestamp of the synchronization. If decryption fails on an update, the controller leaves the existing decrypted secret untouched to avoid breaking running pods.

16. Technical Interview Questions & Detailed Answers

Q1: Can an attacker decrypt a SealedSecret if they steal the public key?

Answer: Absolutely not. Sealed Secrets relies on asymmetric cryptography (RSA). The public key is strictly a write-only mechanism used to encrypt data. Decryption is mathematically impossible without the corresponding private key, which is kept strictly secured inside the Kubernetes cluster boundary. This mathematical asymmetry is what makes the file completely safe to store in public Git repositories.

Q2: What happens to my running applications when a sealing key rotates? Do they break?

Answer: No, they do not break. The Sealed Secrets controller retains all historical private keys in the cluster. When it encounters a SealedSecret, it reads the metadata to see which key generation was used, retrieves the corresponding private key, and decrypts the secret. However, as an enterprise best practice, you should periodically run kubeseal --re-encrypt to update old manifests to use the newest active key, allowing you to eventually retire ancient keys.

Q3: Explain the security risks of the "cluster-wide" scope in Sealed Secrets.

Answer: The cluster-wide scope removes the cryptographic binding of the secret name and namespace from the ciphertext. This means that if an attacker has read access to your Git repository, they can copy the encrypted block, create their own SealedSecret manifest targeting a namespace they control (e.g., sandbox-dev), apply it, and then read the fully decrypted secret values from the native Secret resource. Therefore, cluster-wide should be strictly banned for sensitive