Kustomize overlays introduce complexity through environment-specific configurations; manage them with structured workflows and policy enforcement to avoid drift and conflicts.

Problem Diagnosis

Kustomize overlays often sprawl into tangled configurations when teams:

• Duplicate base configs for minor tweaks
• Overuse patches without clear ownership
• Allow environment-specific hacks to accumulate
  This leads to merge conflicts, deployment drift, and debugging nightmares.

Actionable Workflow

1. Baseline Base Configurations

   • Define a single canonical base for each workload type (e.g., app-base, db-base)
   • Include only shared, non-environment-specific settings in bases
   • Example: app-base/kustomization.yaml with common labels, image, and resources

2. Enforce Naming Conventions

   • Prefix overlays with environment and purpose: env-prod-app, env-stg-db
   • Use kustomize config default to standardize output names

3. Implement Layered Overlays

   • Separate concerns into layers:
     • Global: Cluster-wide policies (e.g., monitoring, logging)
     • Environment: Env-specific settings (e.g., replica count, resource limits)
     • Team: Application-specific overrides
   • Example structure:

     overlays/
       global/
       env-prod/
       team-a/
     

4. Automate Validation

   • Integrate kustomize build --dry-run into CI to catch errors early
   • Use OPA/Gatekeeper to enforce policies (e.g., “no privileged containers”)

5. Monitor and Rotate

   • Audit overlays quarterly; delete unused ones
   • Rotate credentials/secrets referenced in overlays via tools like Vault

Policy Example

Patch to enforce labels in all overlays:

# overlays/env-prod/patches/label-enforcement.yaml
apiVersion: v1
kind: Patch
target:
  group: apps
  version: v1
  kind: Deployment
path: /metadata/labels
create: true
op: replace
value:
  env: prod
  team: sre

Tooling

• kustomize: Core tool for building overlays
• kubectl kustomize: Apply directly to cluster
• OPA/Gatekeeper: Enforce policies across overlays
• Argo CD: Sync overlays to clusters with validation

Tradeoffs

• Flexibility vs. Maintainability: More overlays = more flexibility but higher maintenance cost
• Merge Conflicts: Overlapping patches can cause silent failures
• Context Switching: Deeply nested overlays slow down debugging

Troubleshooting

Common Failures:

1. Missing patches:

   • Symptom: Config not applied as expected
   • Fix: Run kustomize build -n <namespace> to debug patch order

2. Incorrect patch paths:

   • Symptom: “Patch merge failed” errors
   • Fix: Validate paths with kustomize edit add patch -f <patch.yaml>

3. Environment drift:

   • Symptom: Prod and staging behave differently
   • Fix: Use kustomize diff env-prod env-stg to identify discrepancies

Validation Command:

kustomize build overlays/env-prod | kubectl validate -f -

Prevention

• Document ownership: Assign teams to specific overlays
• Limit patch scope: Avoid global patches unless necessary
• Use version control: Treat overlays like application code with PR reviews

Overlays are a double-edged sword—structure them rigorously or face the chaos.

Source thread: Kustomize Overlays - Adding Complexity?