Proactive node health checks and repairs prevent outages in Kubernetes clusters.

Actionable Workflow for Node Health Recovery

1. Identify Unhealthy Nodes:

   kubectl get nodes --output=jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}'  
   

   Look for False or Unknown in the Ready status column.

2. Inspect Node Conditions:

   kubectl describe node <node-name> | grep -A 10 "Conditions"  
   

   Check for MemoryPressure, DiskPressure, or NetworkUnavailable.

3. Check System Pods:

   kubectl get pods --all-namespaces -o wide | grep -E 'kube|coredns|etcd'  
   

   Restart failed system pods or check logs:

   kubectl logs <pod-name> -n <namespace> --previous  
   

4. Drain and Reboot (If Safe):

   kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data  
   systemctl reboot  
   

   Only if the node is tainted or non-critical workloads are running.

5. Verify Recovery:

   kubectl get node <node-name> -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}'  
   

Policy Example: Enforce Node Health Checks

Use Open Policy Agent (OPA) to block deployments on unhealthy nodes:

package kubernetes.admission  

deny[msg] {  
  input.request.kind.kind == "Pod"  
  node := input.request.metadata.labels.node  
  not (node == "healthy-node-1" || node == "healthy-node-2")  
  msg := "Pods must be scheduled to healthy nodes only."  
}  

Tradeoff: This blocks deployments during node failures, forcing manual intervention.

Tooling for Node Health

• k9s: Interactive CLI for real-time node/pod monitoring.

  k9s watch nodes  
  

• kube-node-problem-detector: Auto-detects hardware failures, kernel panics.
• Prometheus + Node Exporter: Track resource usage and disk pressure.

Caveats and Tradeoffs

• False Positives: Node pressure conditions may not always indicate failure (e.g., transient spikes).
• Overhead: Aggressive monitoring (e.g., high-frequency scrape intervals) can strain etcd.
• OpenShift Specifics: Use oc adm node commands for drain/reboot in OpenShift clusters.

Troubleshooting Common Failures

• Node NotReady:
  • Check kubelet status: systemctl status kubelet
  • Verify container runtime (e.g., containerd): systemctl status containerd
• Disk Pressure Evictions:
  • Free space: df -h on the node.
  • Clean unused images: crictl rmi --prune=true
• Network Issues:
  • Check CNI plugin status (e.g., Calico, Cilium).
  • Inspect kube-proxy logs: kubectl logs -n kube-system <kube-proxy-pod>

Conclusion

Node issues are inevitable, but systematic checks and automated policies reduce blast radius. Prioritize observability, test recovery workflows, and avoid over-engineering fixes that introduce new failure modes.

Source thread: Weekly: This Week I Learned (TWIL?) thread