Use environment-specific pipelines with Azure DevOps variables, resource groups, and Kubernetes namespaces to isolate deployments across dev, QA, UAT, and prod with controlled promotion paths.

Diagnosis: Common Issues Without Isolation

Without proper separation, teams face:

• Accidental overwrites in shared clusters
• Credential leakage between environments
• Uncontrolled promotion of untested changes
• Difficulty auditing deployments per environment

Workflow: Environment-Specific Pipeline Design

1. Resource Group Isolation

   • Create separate AKS clusters or namespaces per environment (e.g., aks-dev, aks-qa, aks-uat, aks-prod).
   • Use Azure Resource Manager (ARM) templates or Terraform for consistent provisioning.

2. Pipeline Structure

   • Define distinct pipelines in Azure DevOps: dev-pipeline.yml, uat-pipeline.yml, etc.
   • Use pipeline variables to control environment-specific settings:

     variables:  
       environment: 'dev'  
       k8sNamespace: 'dev-namespace'  
       k8sCluster: 'aks-dev'  
     

3. Namespace Enforcement

   • Apply Kubernetes RBAC to restrict service accounts per namespace.
   • Use kubectl config set-context in pipeline jobs to target the correct namespace.

4. Promotion Gates

   • Use Azure Pipelines stages with manual approvals for prod/uat.
   • Example:

     stages:  
       - stage: dev  
         jobs:  
           - job: build-deploy  
             steps:  
               - script: kubectl apply -f manifests/dev/  
       - stage: uat  
         dependsOn: dev  
         condition: succeeded()  
         jobs:  
           - job: deploy-uat  
             steps:  
               - script: kubectl apply -f manifests/uat/  
     

Policy Example: Promotion Gates and Access Control

Policy:

• Dev pipelines auto-deploy on PR merges to main.
• UAT requires manual approval and passes from dev.
• Prod requires:
  • Successful UAT deployment
  • Approval from security + infra teams
  • Audit log validation

Implementation:

• Use Azure Pipeline approvals for stages.
• Restrict prod pipeline triggers to specific users/groups.

Tooling: Azure DevOps Features to Leverage

• Variables Library: Store environment-specific secrets (e.g., prod-db-password) securely.
• Service Connections: Create separate Kubernetes service connections per cluster.
• Pipeline Artifact Binding: Use pipeline artifacts to share builds across stages.
• Azure Policy: Enforce tagging and resource creation rules for AKS clusters.

Tradeoffs and Caveats

• Complexity vs. Isolation: More pipelines = more maintenance. Start with dev/prod split, then expand.
• Cost: Separate clusters for prod/uat increase Azure costs but reduce blast radius.
• Latency: Namespace isolation in a single cluster is cheaper but riskier than multi-cluster.

Troubleshooting Common Failures

1. Incorrect Namespace Targeting

   • Symptom: Deployments land in wrong environment.
   • Fix: Validate kubectl config current-context in pipeline logs.

2. Permission Denied in Cluster

   • Symptom: RBAC blocks deployment.
   • Fix: Check service account permissions in the target namespace/cluster.

3. Variable Resolution Errors

   • Symptom: Pipeline uses dev variables in prod.
   • Fix: Use $(environment) in all resource names and paths.

4. Stuck Pipeline Approvals

   • Symptom: Manual approvals not triggering.
   • Fix: Verify approver email notifications and Azure DevOps permissions.

Final Validation

After setup:

1. Deploy a test app to dev.
2. Promote to uat manually.
3. Check audit logs in prod for compliance.
4. Validate RBAC denies cross-namespace access.

This approach balances control and flexibility—start small, enforce boundaries, and automate guardrails.

Source thread: How do you actually separate CI/CD pipelines for AKS across dev/qa/uat/prod in Azure DevOps?