This guide provides field-tested steps for diagnosing, repairing, and preventing Kubernetes issues in production environments.

Diagnosis and Repair Workflow

1. Check cluster health

   • Run kubectl get nodes --output wide to verify node status and resource availability.
   • Use kubectl describe node <node-name> to inspect taints, tolerations, and pod capacity.
   • Look for NotReady nodes or resource pressure (e.g., disk pressure, memory pressure).

2. Verify workloads

   • List pods: kubectl get pods --all-namespaces.
   • Inspect failing pods: kubectl describe pod <pod-name> -n <namespace>.
   • Check events for errors: kubectl describe pod <pod-name> -n <namespace> | grep -A 10 Events.

3. Inspect configurations

   • Review deployments: kubectl describe deployment <deployment-name> -n <namespace>.
   • Validate configmaps/secrets: kubectl describe cm <configmap-name> -n <namespace>.
   • Test liveness/readiness probes by simulating requests against pod endpoints.

4. Review logs and metrics

   • Fetch pod logs: kubectl logs <pod-name> -n <namespace> --previous (for crashed containers).
   • Use kubectl top node and kubectl top pod to monitor resource usage.
   • Integrate with Prometheus/Grafana for historical metrics and alerts.

5. Rollback or fix

   • Rollback deployments: kubectl rollout undo deployment <deployment-name> -n <namespace>.
   • Apply corrected manifests: kubectl apply -f <updated-manifest.yaml>.

Prevention and Maintenance Policy

Example policy for production clusters:

• Daily:
  • Monitor node resource usage (kubectl top node).
  • Check for evicted pods (kubectl get pods --all-namespaces | grep Evicted).
• Weekly:
  • Audit RBAC roles/bindings (kubectl get roles,rolebindings,clusterroles,clusterrolebindings --all-namespaces).
  • Review ingress and service configurations (kubectl get ing,svc --all-namespaces).
• Monthly:
  • Perform disaster recovery drills (e.g., delete a node to test auto-recovery).
  • Test Velero backups for cluster state and persistent volumes.
• Quarterly:
  • Upgrade Kubernetes version (plan for at least one minor version behind latest).
  • Review security policies (e.g., PodSecurityPolicies, network policies).

Tooling

• kubectl: Core CLI for cluster interaction. Use kubectl api-resources to list available resources.
• k9s: Terminal UI for real-time monitoring and quick actions (e.g., k9s get all).
• kubeaudit: Scan for security misconfigurations: kubeaudit --namespace <namespace>.
• Prometheus/Grafana: Monitor cluster health and set alerts for critical metrics (e.g., node disk usage).
• OpenShift CLI (oc): For OpenShift environments, use oc get projects, oc describe buildconfig, etc.

Tradeoffs and Caveats

• Automated rollouts vs. manual approval: Canary deployments reduce risk but add complexity. Use Argo Rollouts for progressive delivery, but ensure manual approval gates for critical services.
• Alpha features: Avoid enabling alpha features (e.g., KubernetesAlphaAPIs) in production; they may destabilize the cluster.
• Managed services: Tools like EKS/GKE reduce operational overhead but limit customization (e.g., node auto-repair may conflict with custom node configurations).

Troubleshooting Common Failure Points

• CrashLoopBackoff:

  • Check pod logs: kubectl logs <pod-name> -n <namespace> --previous.
  • Verify application health checks (liveness/readiness probes).
  • Ensure dependencies (e.g., databases) are reachable.

• Image Pull Errors:

  • Confirm image name and tag in deployment manifest.
  • Check image pull policy (Always, IfNotPresent, Never).
  • Test registry access: docker pull <image-name> from a node.

• Node Resource Pressure:

  • Use kubectl describe node <node-name> to check Allocatable resources.
  • Scale horizontally with HPA or VPA, or increase node pool size.

Source thread: Kubernetes revision/reference guide?