Securely bootstrap secrets, cert-manager, and ArgoCD in production Kubernetes clusters using a structured, maintainable workflow.

Workflow

1. Cluster Creation
   Use kops, cloud provider tools, or OpenShift’s oc CLI to spin up a fresh cluster. Ensure the control plane has permissions for initial setup.

2. Initial Secrets Setup
   Apply critical secrets (e.g., Bitwarden token, external API keys) immediately after cluster creation using Terraform or Tofu. Example:

   kubectl apply -f bitwarden-secret.yaml  # Pre-generated manifest  
   

   Why? Avoids exposure by keeping secrets out of version control and ensures ArgoCD doesn’t redeploy them publicly.

3. Deploy ArgoCD
   Use Helm to install ArgoCD into the argocd namespace:

   helm repo add argo https://argoproj.github.io/argo-cd  
   helm install argocd argo/argo-cd --namespace argocd --create-namespace  
   

   Validate with:

   kubectl get pods -n argocd  
   

4. Deploy Cert-Manager
   Install cert-manager via Helm:

   helm repo add jetstack https://charts.jetstack.io  
   helm install cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace --set extraArgs={--clusterissuer-name=letsencrypt-prod}  
   

   Check for valid certificates:

   kubectl get certificates  
   

5. ArgoCD Adoption
   Use Helmfile or raw manifests to let ArgoCD manage downstream components. Example Helmfile snippet:

   releases:  
     - name: cert-manager  
       chart: jetstack/cert-manager  
       version: 1.8.0  
       namespace: cert-manager  
   

   Sync and validate:

   argoctl sync application cert-manager  
   argoctl get applications  
   

6. Policy Enforcement
   Use Kyverno or OPA Gatekeeper to enforce secret and certificate policies. Example Kyverno policy to block unencrypted secrets:

   apiVersion: kyverno.io/v1  
   kind: ClusterPolicy  
   metadata:  
     name: block-unencrypted-secrets  
   spec:  
     validationFailureAction: enforce  
     rules:  
       - name: check-secret-type  
         match:  
           resources:  
             kinds: [Secret]  
         validate:  
           message: "Only encrypted secrets allowed"  
           pattern:  
             type: not in ["Opaque", "kubernetes.io/tls"]  
   

Tooling

• Helm/Helmfile: Version-controlled, templated deployments. Helmfile excels at managing multi-release workflows.
• Terraform/Tofu: Secure initial secret injection. Tofu simplifies secret management for GitOps workflows.
• Kyverno/OPA Gatekeeper: Enforce policies at scale. Kyverno is easier for admission control; Gatekeeper offers broader Open Policy Agent flexibility.

Tradeoffs

• Helmfile Complexity: Adds a layer of abstraction but requires learning its syntax and dependency management.
• Timing Risks: ArgoCD might attempt to sync before initial secrets are applied, causing temporary sync errors. Mitigate by applying secrets before ArgoCD deployment.

Troubleshooting

• Secrets Not Applied:
  • Check kubectl describe secret <name> for creation errors.
  • Ensure the namespace exists if using namespace: in manifests.
• ArgoCD Sync Failures:
  • Run argoctl get applications --detailed to inspect sync issues.
  • Verify git repository connectivity and permissions.
• Cert-Manager Certificate Stuck:
  • Check challenge status: kubectl describe order <order-name>.
  • Ensure DNS provider is configured correctly for external DNS.

This workflow balances security, auditability, and automation while avoiding fragile dependencies. Adjust based on your cluster’s lifecycle and compliance needs.

Source thread: How do you boostrap secrets,cert-manager,argo ?