Mastering advanced Kubernetes concepts like network policies, storage management, and security contexts is essential for reliable cluster operations.

Diagnosing and Repairing Common Production Issues

Advanced Kubernetes concepts matter because they directly impact cluster stability, security, and scalability. Here’s a field-tested workflow for handling critical scenarios:

1. Network Policy Enforcement and Debugging

Actionable Workflow:

1. Identify misconfigured policies: Run kubectl get networkpolicies --all-namespaces to list policies.
2. Test connectivity: Use kubectl exec -it <pod> -- curl http://<target-service> to validate allowed traffic.
3. Diagnose blocks: Check logs from CNI plugins (e.g., cilium logs) for dropped packets.

Policy Example:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-frontend-only
spec:
  podSelector:
    matchLabels:
      app: backend
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          name: frontend-ns

Tradeoff: Strict network policies improve security but can break legacy apps expecting flat networks. Test incrementally with default-deny namespaces.

2. Storage Provisioning and Resizing

Actionable Workflow:

1. Monitor storage usage: Use kubectl get storageclasses and kubectl describe pvc <pvc-name>.
2. Expand PVCs: Edit PVC size and restart pods (ensure workload supports in-place resizing).
3. Validate CSI driver health: Check kubectl get csinodes for attached volumes.

Caveat: Not all storage backends support online resizing. For NFS or legacy drivers, manual intervention (e.g., node reboot) may be required.

3. Security Contexts and Least Privilege

Actionable Workflow:

1. Audit existing pods: Run kubectl describe pod <pod-name> | grep -E "Security-Context|RunAsUser".
2. Apply non-root contexts:

   securityContext:
     runAsUser: 1000
     runAsGroup: 3000
     fsGroup: 2000
   

3. Test application compatibility: Some apps (e.g., init containers) may require elevated privileges.

Tradeoff: Restrictive security contexts reduce attack surface but may break legacy workloads. Use privileged: false and allowPrivilegeEscalation: false as defaults.

Tooling for Advanced Operations

• k9s: Interactive UI for real-time monitoring and debugging (e.g., k9s describe pod/<pod>).
• Cilium: For advanced network policy enforcement and troubleshooting (e.g., cilium connectivity test).
• Falco: Runtime security monitoring for anomalous behavior (e.g., shell in unexpected containers).
• Velero: Cluster-wide backup/restore for disaster recovery (critical for etcd, but also for stateful workloads).

Troubleshooting Common Failure Points

• Pod in Pending State:
  • Check kubectl describe pod <pod> for scheduling errors.
  • Common fixes: Free up resources, check node taints (kubectl get nodes -o wide), or scale down other workloads.
• Network Policy Blocking Legitimate Traffic:
  • Use kubectl get networkpolicies -n <namespace> to verify rules.
  • Test with kubectl exec and curl as above.
• Storage Provisioning Failures:
  • Check kubectl get events -n <namespace> --sort-by=.metadata.creationTimestamp for PVC errors.
  • Validate storage class configuration and quota limits.

Prevention Through Policy and Automation

• Adopt Pod Disruption Budgets (PDBs):

  apiVersion: policy/v1
  kind: PodDisruptionBudget
  metadata:
    name: min-available-pods
  spec:
    minAvailable: 2
    selector:
      matchLabels:
        app: critical-service
  

• Enforce quotas and limits:

  apiVersion: v1
  kind: ResourceQuota
  metadata:
    name: prod-quota
  spec:
    hard:
      pods: "100"
      limits.cpu: "4"
  

• Automate security scans: Integrate tools like Trivy or Clair into CI/CD pipelines to catch vulnerabilities before deployment.

In production, the goal isn’t to know every Kubernetes feature but to master the ones that keep clusters running when pressure is on. Focus on observability, least privilege, and recovery workflows—not just shiny objects.

Source thread: What are advanced Kubernetes concepts every cluster admin should know?