A field-tested approach to diagnosing and resolving Kubernetes issues with actionable steps and prevention strategies.

Workflow: Observe → Diagnose → Repair → Prevent

1. Observe

   • Monitor alerts, logs, and metrics (Prometheus, Grafana, EFK stack).
   • Validate user reports: “App X is slow” → check latency percentiles, error rates.
   • Use kubectl get pods --all-namespaces and kubectl describe pod <troubled-pod> to surface anomalies.

2. Diagnose

   • Node-level: Check node status (kubectl get nodes), resource pressure (kubectl top node), and system logs (journalctl -u kubelet).
   • Pod-level: Inspect events (kubectl describe pod), container logs (kubectl logs <pod> --container=<container>), and readiness/liveness probes.
   • Network: Test connectivity with kubectl exec -it <pod> -- curl <service>, validate network policies (kubectl get networkpolicies).

3. Repair

   • Rollback deployments: kubectl rollout undo deployment/<deployment>.
   • Drain nodes for maintenance: kubectl drain <node> --ignore-daemonsets --delete-emptydir-data.
   • Force pod evictions if stuck: kubectl delete pod <pod> --force.

4. Prevent

   • Implement resource limits and requests to avoid OOM kills.
   • Use admission controllers (e.g., NamespaceLifecycle, ResourceQuota) to enforce policies.
   • Conduct blameless postmortems to update playbooks.

Policy Example: Maintenance Window Enforcement

Policy: All node maintenance requiring downtime must occur during scheduled windows (e.g., 02:00–04:00 UTC).

Implementation:

• Use kubectl taint to enforce node exclusions:

  kubectl taint nodes <node-name> node.maintenance/no-schedule:NoSchedule  
  

• Automate alerts via Prometheus for unscheduled drain attempts.

Tradeoff: Reduced flexibility for urgent fixes but minimizes user impact during peak hours.

Tooling

• CLI: kubectl, k9s (for quick navigation), oc (OpenShift users).
• Monitoring: Prometheus + Grafana for dashboards, Alertmanager for notifications.
• Logging: EFK (Elasticsearch, Fluentd, Kibana) or Loki + Promtail.
• Debugging: kubectl exec, kubectl port-forward, and sidecar debug containers.

Caveat: Over-instrumentation can create noise. Start with core metrics (CPU, memory, network I/O) before adding custom ones.

Troubleshooting Common Failures

1. Node Not Ready

   • Check kubelet status: systemctl status kubelet (on-node).
   • Verify cloud provider status (e.g., AWS IMDSv1 vs. v2 issues).

2. Pod CrashLoopBackOff

   • Run kubectl describe pod to identify image pull errors or misconfigured volumes.
   • Inspect container logs: kubectl logs <pod> --previous.

3. NetworkPolicy Blocking Traffic

   • Test with kubectl exec -it <pod> -- curl <target-service-ip>.
   • Validate policy rules: kubectl get networkpolicy <policy-name> -o yaml.

4. Persistent Volume Issues

   • Check PV/PVC binding status: kubectl get pvc --all-namespaces.
   • Use kubectl describe pvc <pvc-name> to spot provisioning errors.

Final Note

Troubleshooting is 20% tools and 80% process. Document playbooks for recurring issues, automate where possible, and prioritize observability over guesswork. Always validate fixes with kubectl get events --sort-by=.metadata.creationTimestamp to ensure stability.

Source thread: Curious about what people usually follow during the troubleshooting process?