We use Kubernetes for workload standardization, portability across environments, and to meet strict air-gapped requirements that cloud-native PaaS solutions can’t satisfy.

Why Kubernetes?

Core Drivers

• Standardization: Kubernetes provides a unified control plane for deploying and managing containers across hybrid and air-gapped environments.
• Portability: Workloads run consistently on-prem, in public clouds, or offline, avoiding vendor lock-in.
• Air-Gapped Support: Kubernetes distributions (e.g., OpenShift, ROKS) can operate entirely offline, critical for regulated industries or disconnected sites.
• Operational Control: Fine-grained resource management, custom networking, and security policies vs. opaque PaaS abstractions.

Tradeoffs

• Complexity: Requires in-house expertise for cluster lifecycle management, networking, and security.
• Cost: Self-managed Kubernetes incurs higher operational overhead than Fargate but offers long-term flexibility.
• Update Risks: Air-gapped clusters demand rigorous offline image and binary management, increasing maintenance effort.

Actionable Workflow

1. Assess Constraints

   • Does your workload require internet connectivity?
   • Are there regulatory/compliance mandates for air-gapped operations?
   • What’s your team’s Kubernetes maturity?

2. Evaluate Alternatives

   • Compare Kubernetes with Fargate, Nomad, or VM-based solutions using a decision matrix (cost, portability, control).

3. Prototype

   • Deploy a small cluster (e.g., k3s, MicroK8s) in an air-gapped lab.
   • Test image mirroring, offline upgrades, and CI/CD integration.

4. Scale with Guardrails

   • Enforce policies for image sources, resource limits, and network segmentation.

Policy Example: Air-Gapped Image Management

apiVersion: v1  
kind: ConfigMap  
metadata:  
  name: image-policy  
data:  
  allowedImageSources: |  
    - "registry.internal.example.com"  
    - "localhost:5000"  
  imagePullPolicy: "Never"  

Enforcement: Use admission controllers (e.g., OPA Gatekeeper) to block images from unapproved registries.

Tooling

• Zarf: Simplifies deploying Kubernetes and air-gapped container registries via a single binary.
• KubeVela: Manages multi-cluster workflows and policy enforcement.
• Krew: Plugin manager for extending kubectl with air-gapped compatible tools.
• Velero: Backup/restore for stateful workloads in disconnected environments.

Example: Use Zarf to bootstrap a cluster offline:

# On a connected machine  
zarf init --airgap  

# Transfer bundles to air-gapped node via USB/physical media  
zarf cluster init --airgap-bundle /path/to/bundle.tgz  

Troubleshooting

Common Failures

• Image Pull Errors:

  • Check: kubectl describe pod <pod-name> for image source violations.
  • Fix: Mirror images to internal registry and update deployments.

• Node Communication Issues:

  • Check: kubectl get nodes -o wide and systemctl status kubelet on affected nodes.
  • Fix: Validate static IP assignments and firewall rules in air-gapped networks.

• Upgrade Failures:

  • Check: kubectl get componentstatuses and offline binary integrity.
  • Fix: Use Zarf or manual tarball upgrades with staged rollouts.

Conclusion

Kubernetes isn’t always the answer, but for teams needing standardization, portability, and air-gapped resilience, it’s the most pragmatic choice despite its operational demands. Start small, enforce policies early, and invest in tooling that reduces toil.

Source thread: why does your company use Kubernetes?