Developers must meet basic Kubernetes readiness criteria before deploying to production clusters.

Diagnosis: Why This Matters

Kubernetes misconfigurations from unprepared teams lead to outages, security gaps, and wasted engineering time. Common failures include:

• Pods exposed directly without Services/Ingress
• No Pod Disruption Budgets (PDBs) for critical apps
• Ignoring readiness/liveness probes
• Misunderstanding pod lifecycle and self-healing

Repair Workflow: From Zero to Deployable

1. Educate, don’t enable:
   • Require developers to complete a 1-hour Kubernetes basics workshop (pods, services, configmaps, secrets).
   • Share a curated list of kubectl commands for debugging (e.g., kubectl describe pod, kubectl logs -f).
2. Policy enforcement:
   • Block deployments via CI/CD unless a readiness checklist is signed off (see example below).
   • Use admission controllers (e.g., OPA Gatekeeper) to reject invalid manifests.
3. Automate guardrails:
   • Provide a CLI tool that generates sanitized Helm charts or Kustomize bases after answering basic questions (e.g., “Expose to internet?”, “Set PDB?”).

Policy Example: Readiness Checklist

Developers must confirm:

• App handles restarts and graceful termination (SIGTERM handling)
• Configured liveness/readiness probes with appropriate paths and timeouts
• Defined resource requests/limits
• Secrets stored in Kubernetes Secrets or external vault
• PDB created for stateful or critical workloads

Tooling: Practical Guardrails

• CLI generator: A script that prompts for key decisions and outputs valid manifests. Example questions:

  $ k8s-gen  
  ? Should the app be accessible externally? Yes  
  ? Set hostname for Ingress? myapp.example.com  
  ? Enforce minimum replicas (PDB)? Yes  
  

• GitOps repo template: Pre-configured ArgoCD or Flux repositories with validated baselines.
• Manifest linter: Use kube-score or checkov in CI to flag common issues (e.g., missing resource limits).

Tradeoff: Automation vs. Learning

While CLI generators reduce errors, they risk creating dependency. Balance by:

• Requiring developers to explain their choices during code reviews.
• Rotating them into platform team on-call duties to experience operational impact.

Troubleshooting Common Failures

• App not reachable:
  • Check if a Service/Ingress exists (kubectl get svc,ingress).
  • Verify ports and selectors match the pod labels.
• Pod in CrashLoopBackOff:
  • Run kubectl describe pod to check termination reason.
  • Inspect logs (kubectl logs -f <pod>) for startup errors.
• Unscheduled pods during maintenance:
  • Confirm PDB is correctly configured (kubectl describe pdb).

Prevention: Cultural Shifts

• Tie deployment permissions to completed training and checklist sign-offs.
• Assign platform engineers as liaisons for complex workloads, not just fire-fighting.
• Celebrate teams that adopt self-service tooling responsibly.

This approach reduces chaos without stifling velocity. The goal isn’t to gatekeep Kubernetes—it’s to ensure deployments succeed and survive production.

Source thread: [rant] Does anyone have to deal with developers that want to deploy to kubernetes without knowing a single thing about it?