Use external secret managers with ArgoCD SyncWaves and GitOps tooling to securely manage secrets at scale.

Secrets management in ArgoCD is a production-critical problem. Missteps here lead to downtime, security incidents, or compliance failures. Here’s how I’ve handled it across multiple environments, with a focus on reliability and auditability.

Actionable Workflow

1. Externalize secrets

   • Never store secrets in Git. Use HashiCorp Vault, AWS Secrets Manager, or Azure Key Vault.
   • Inject secrets into clusters via ArgoCD SyncWaves or the External Secrets Operator (ESO).

2. Sync secrets declaratively

   • Define secret references in ArgoCD applications using project-sync-policy and sync-wave annotations.
   • Example:

     annotations:  
       argoproj.io/sync-wave: "2"  
       external-secrets/managed: "true"  
     

3. Automate rotation

   • Use Vault’s dynamic secrets or cloud provider APIs to rotate credentials on a schedule.
   • Monitor rotation failures via alerts (e.g., Prometheus + Alertmanager).

4. Audit access

   • Enable ArgoCD audit logging and integrate with SIEM tools (e.g., Splunk, ELK).
   • Regularly review who synced what secrets and when.

Policy Example

Enforce secret sourcing and encryption:

# SyncWave policy snippet  
syncWave:  
  - name: vault-secrets  
    annotationMatch:  
      - key: external-secrets/managed  
        value: "true"  
    dependency:  
      - name: base-infrastructure  
    actions:  
      - get:  
          - secret  
      - create:  
          - secret  

Caveat: SyncWaves add complexity. Misconfigured dependencies can cause sync deadlocks. Test policies in staging first.

Tooling

• ArgoCD SyncWaves: Native support for staged secret synchronization.
• HashiCorp Vault: Centralized secret management with Kubernetes auth.
• External Secrets Operator (ESO): Simplifies cloud-native secret syncing.
• Sealed Secrets: For edge cases where external managers aren’t viable (but avoid if possible—keys are hard to manage at scale).

Tradeoff: External secret managers add latency and operational overhead. For low-risk environments, Sealed Secrets might be simpler.

Troubleshooting

Common failures:

1. Missing annotations:

   • Check ArgoCD app manifest for argoproj.io/sync-wave and secret source annotations.
   • Command: kubectl describe app <app-name> -n argocd.

2. SyncWave stalls:

   • Look for dependency loops or missing resources in earlier sync waves.
   • Command: kubectl get syncwaves -n argocd --watch.

3. Permission errors:

   • Ensure service account has RBAC access to secret manager (e.g., Vault token permissions).
   • Check logs: kubectl logs -l app.kubernetes.io/part-of=argocd -n argocd.

4. Expired certificates:

   • Rotate TLS certs for secret managers and update ArgoCD’s trust store.

Final Notes

There’s no one-size-fits-all. Start with the simplest solution that meets your compliance needs (e.g., ESO + cloud KMS), then layer in SyncWaves for advanced use cases. Always assume secrets will leak—design for fast rotation and least privilege.

Source thread: How do you guys manage secrets in ArgoCD?