https://liveaverage.com/categories/devsecops/ Recent content in DevSecOps on Live Average Hugo -- gohugo.io en-us Fri, 10 Apr 2026 00:00:00 +0000 https://liveaverage.com/blog/kubernetes/2026-04-10-baremetal-k3s-migration-to-aws-eks/ Fri, 10 Apr 2026 00:00:00 +0000 https://liveaverage.com/blog/kubernetes/2026-04-10-baremetal-k3s-migration-to-aws-eks/ Migrating k3s from baremetal to AWS EKS requires careful planning, data transfer, and validation to ensure minimal downtime and data integrity. Workflow: Baremetal to EKS Migration Assess and Inventory List all workloads, persistent volumes, and ingress configurations: kubectl get pods,svc,ingress,pv,pvc --all-namespaces Document network dependencies (e.g., CNI plugins, MTU settings). Identify stateful components requiring data migration (e.g., databases, NFS mounts). Prepare AWS Environment Use eksctl to provision an EKS cluster matching your k3s version: eksctl create cluster --name my-cluster --region us-west-2 --nodegroup-name worker-nodes --node-type t3. https://liveaverage.com/blog/kubernetes/2026-04-09-kubernetes-on-hetzner-what-s-your-experience/ Thu, 09 Apr 2026 00:00:00 +0000 https://liveaverage.com/blog/kubernetes/2026-04-09-kubernetes-on-hetzner-what-s-your-experience/ Deploying Kubernetes on Hetzner requires careful node configuration, network setup, and storage planning to avoid common pitfalls. Why Hetzner? Hetzner offers cost-effective bare-metal instances with low latency, ideal for stateful workloads or latency-sensitive apps. Avoid the “cheap cloud” trap: Hetzner’s hardware is solid, but you manage the entire stack. Actionable Workflow Provision nodes with Terraform Use the hcloud Terraform provider to spin up nodes. Example: resource "hcloud_server" "k8s_node" { name = "k8s-worker-01" datacenter_id = 1 server_type = "cx11" ssh_keys = [123] } Validate with terraform plan and apply. https://liveaverage.com/blog/kubernetes/2026-04-08-how-are-you-handling-vulnerability-management-across-kubernetes/ Wed, 08 Apr 2026 00:00:00 +0000 https://liveaverage.com/blog/kubernetes/2026-04-08-how-are-you-handling-vulnerability-management-across-kubernetes/ Automate scanning, enforce policies, and prioritize fixes without blocking deployments to balance security and velocity. Vulnerability management in Kubernetes is a race between deployment speed and risk exposure. Teams often face pressure to ship features while ensuring clusters aren’t exposed to known exploits. The key is integrating security into existing workflows without creating bottlenecks. Actionable Workflow Automate Image Scanning in CI/CD Scan container images before they reach the cluster using tools like Trivy, Clair, or Anchore. https://liveaverage.com/blog/kubernetes/2026-04-07-headlamp-token-less-in-cluster-deployment/ Tue, 07 Apr 2026 00:00:00 +0000 https://liveaverage.com/blog/kubernetes/2026-04-07-headlamp-token-less-in-cluster-deployment/ Deploy Headlamp securely in Kubernetes using OIDC authentication and kubeconfig mounting, avoiding service account tokens. Problem Headlamp requires authentication to access the Kubernetes API. While service account tokens are common, they introduce rotation and exposure risks. Token-less alternatives like OIDC are preferable but require careful setup. Solution Workflow Set up an OIDC provider (e.g., Keycloak, Dex, Okta) integrated with your identity source. Configure the kube-apiserver with OIDC flags: --oidc-issuer-url=https://your-oidc-provider.com --oidc-client-id=headlamp --oidc-groups-claim=groups Deploy Headlamp with a mounted kubeconfig: Create a kubeconfig file with credentials for the target cluster. https://liveaverage.com/blog/kubernetes/2026-04-06-freemium-saas-on-k8s-automating-namespace-per-customer-provision/ Mon, 06 Apr 2026 00:00:00 +0000 https://liveaverage.com/blog/kubernetes/2026-04-06-freemium-saas-on-k8s-automating-namespace-per-customer-provision/ A practical guide to automating namespace-per-customer provisioning in Kubernetes using GitLab CI, including workflow steps, policy examples, and troubleshooting tips. Workflow Overview Repository Structure: Use a dedicated Git repo for customer onboarding manifests (e.g., customer-infra). Include namespace.yaml, resourcequotas.yaml, and limitranges.yaml per customer. Customer Onboarding Trigger: Integrate with your SaaS billing/authentication system (e.g., Stripe webhook or internal API). Store customer metadata (e.g., tier, region) in a database or GitLab CI variables. https://liveaverage.com/blog/kubernetes/2026-04-05-new-to-k8s-how-do-pods-actually-share-memory-on-a-node/ Sun, 05 Apr 2026 00:00:00 +0000 https://liveaverage.com/blog/kubernetes/2026-04-05-new-to-k8s-how-do-pods-actually-share-memory-on-a-node/ Pods share node memory via Linux cgroups and namespaces, enforcing resource limits to prevent contention. When multiple workloads run on a shared node, memory pressure often stems from misconfigured resource requests or noisy neighbors. Here’s how to diagnose, fix, and prevent these issues in production. How Memory Sharing Works Kubernetes uses cgroups (control groups) to partition memory and namespaces to isolate pods. Each container in a pod is assigned a cgroup with: https://liveaverage.com/blog/kubernetes/2026-04-04-is-there-a-good-kubernetes-client-for-ios-mainly-need-logs-shell/ Sat, 04 Apr 2026 00:00:00 +0000 https://liveaverage.com/blog/kubernetes/2026-04-04-is-there-a-good-kubernetes-client-for-ios-mainly-need-logs-shell/ k9s provides a functional iOS client for Kubernetes troubleshooting with logs, shell access, and pod restarts, though with some usability tradeoffs. Platform engineers need mobile tools for cluster diagnostics when away from desktops. iOS options are limited, but k9s fills critical gaps with console access, real-time logs, and exec capabilities via a web interface. Here’s how to deploy and use it effectively. Actionable Workflow Deploy k9s in web mode: https://liveaverage.com/blog/kubernetes/2026-04-03-the-kubernetes-book-2026/ Fri, 03 Apr 2026 00:00:00 +0000 https://liveaverage.com/blog/kubernetes/2026-04-03-the-kubernetes-book-2026/ Cluster drift undermines reliability; here’s how to detect, correct, and prevent configuration inconsistencies in Kubernetes environments. Diagnosis Cluster drift manifests as unexpected behavior, failed rollouts, or security gaps caused by configuration mismatches between declared state (e.g., Helm charts, GitOps repos) and live cluster state. Common symptoms: Pods failing with ImagePullBackoff despite valid manifests Nodes reporting NetworkPluginCgroupConflictingPlugin errors Unexplained changes to RBAC roles or network policies Diagnostic workflow: Audit logs: kubectl get events --sort-by=. https://liveaverage.com/blog/kubernetes/2026-04-02-new-to-kubernetes-what-is-the-benefit-of-using-helm-over-normal/ Thu, 02 Apr 2026 00:00:00 +0000 https://liveaverage.com/blog/kubernetes/2026-04-02-new-to-kubernetes-what-is-the-benefit-of-using-helm-over-normal/ Helm provides templating, versioning, and reusable charts to streamline Kubernetes deployments, reducing manual YAML management and human error. Why Helm Over Raw YAML? Managing YAML files at scale becomes unwieldy. Helm addresses this by: Templating: Parameterize reusable manifests (e.g., values.yaml). Versioning: Track and roll back chart versions. Dependency Management: Bundle charts for complex applications. Release Lifecycle: Manage deployments, upgrades, and rollbacks as atomic units. Workflow: Helm for Production Workloads Install Helm: brew install helm # or download binary Add Repositories: helm repo add bitnami https://charts. https://liveaverage.com/blog/kubernetes/2026-04-01-best-way-to-get-node-labels-onto-pods/ Wed, 01 Apr 2026 00:00:00 +0000 https://liveaverage.com/blog/kubernetes/2026-04-01-best-way-to-get-node-labels-onto-pods/ Use node selectors and pod annotations to propagate node labels to pods, ensuring consistent scheduling and metadata alignment. Why This Matters Node labels provide critical metadata for scheduling and routing. When pods need access to node-specific attributes (e.g., zone, hardware, or tenant), explicitly attaching these labels avoids ambiguity and ensures workloads run where they belong. Actionable Workflow Identify Required Node Labels List node labels to determine which attributes need propagation: https://liveaverage.com/blog/kubernetes/2026-03-31-what-are-good-projects-to-learn-kubernetes-practically/ Tue, 31 Mar 2026 00:00:00 +0000 https://liveaverage.com/blog/kubernetes/2026-03-31-what-are-good-projects-to-learn-kubernetes-practically/ Learning Kubernetes effectively requires hands-on projects that mirror real-world operational challenges. Start small, iterate, and prioritize observability. Here’s a field-tested approach: Actionable Workflow Local Cluster Setup Use minikube or kind to spin up a local cluster. Goal: Understand node topology, kubectl basics, and pod networking. Validation: Run kubectl get nodes, kubectl taint nodes --list, and deploy a simple NGINX pod. Deploy a Stateful Application https://liveaverage.com/blog/kubernetes/2026-03-30-how-do-you-connect-to-your-clusters/ Mon, 30 Mar 2026 00:00:00 +0000 https://liveaverage.com/blog/kubernetes/2026-03-30-how-do-you-connect-to-your-clusters/ Use a VPN with OIDC authentication for secure, role-based access to Kubernetes clusters without relying on jumphosts or local YAML files. Workflow: Connect via VPN + OIDC Establish VPN connection Use OpenVPN, WireGuard, or similar to tunnel traffic into the cluster’s VPC. Validate connectivity: ping <cluster-api-server-ip> or curl -v https://api-server. Configure OIDC authentication Deploy an OIDC provider (e.g., Keycloak, Auth0) integrated with your identity source (LDAP, SAML, etc. https://liveaverage.com/blog/kubernetes/2026-03-29-am-i-missing-something-why-can-t-i-just-fork-the-secure-bitnami/ Sun, 29 Mar 2026 00:00:00 +0000 https://liveaverage.com/blog/kubernetes/2026-03-29-am-i-missing-something-why-can-t-i-just-fork-the-secure-bitnami/ Forking Bitnami images shifts maintenance burden to your team, introducing operational complexity and security risks if not managed rigorously. Why Forking Isn’t a Panacea Bitnami’s official images include curated dependencies, security patches, and build pipelines. Forking trades licensing costs for: Build infrastructure: Maintaining CI/CD to rebuild images. Update cadence: Tracking upstream changes and applying them. Security assurance: Scanning for vulnerabilities and backporting fixes. Without these, forked images risk becoming outdated or insecure faster than they’re updated. https://liveaverage.com/blog/kubernetes/2026-03-28-why-is-it-so-cold-on-kubecon/ Sat, 28 Mar 2026 00:00:00 +0000 https://liveaverage.com/blog/kubernetes/2026-03-28-why-is-it-so-cold-on-kubecon/ Conference centers overcool for crowds; layer up with technical fabrics and monitor venue HVAC metrics. Why It Matters Large venues like KubeCon set HVAC systems for peak attendance, leaving early sessions frigid. Engineers accustomed to stable datacenter climates often underestimate this variable. Actionable Workflow Check event schedule: First-day attendance is typically lighter; pack extra layers. Layer strategically: Base: Moisture-wicking merino wool shirt Mid: Lightweight fleece or down vest Outer: Breathable shell (avoid cotton) Monitor HVAC: Use building apps (if available) to track zone temperatures. https://liveaverage.com/blog/kubernetes/2026-03-27-what-trends-are-you-seeing-around-self-hosted-software-at-kubeco/ Fri, 27 Mar 2026 00:00:00 +0000 https://liveaverage.com/blog/kubernetes/2026-03-27-what-trends-are-you-seeing-around-self-hosted-software-at-kubeco/ Self-hosted Kubernetes solutions are seeing increased adoption and vendor support at KubeCon EU 2026, driven by cost savings and hybrid deployment needs. Trends Observed Talos Linux Momentum: More teams are adopting Talos for its declarative infrastructure management, especially in hybrid and on-prem scenarios. Vendors are now actively optimizing their tools for Talos clusters. Edge Cases in Multi-Arch Clusters: Heterogenous environments (e.g., ARM/Raspberry Pi control planes with x86 workers) are becoming more common, but require careful networking and storage planning. https://liveaverage.com/blog/kubernetes/2026-03-26-how-are-you-guys-avoiding-the-extended-support-tax/ Thu, 26 Mar 2026 00:00:00 +0000 https://liveaverage.com/blog/kubernetes/2026-03-26-how-are-you-guys-avoiding-the-extended-support-tax/ Proactive lifecycle management and strict version control prevent extended support costs in Kubernetes environments. Extended Support (ES) taxes emerge when clusters run outdated Kubernetes or OpenShift versions no longer covered by standard support. This isn’t theoretical—vendors like Red Hat charge premiums for extended lifecycle support, and cloud providers may deprecated older versions without notice. Here’s how to avoid it. Actionable Workflow for Version Compliance Audit Current Versions https://liveaverage.com/blog/kubernetes/2026-03-25-what-is-the-best-way-for-me-to-protect-internal-company-only-app/ Wed, 25 Mar 2026 00:00:00 +0000 https://liveaverage.com/blog/kubernetes/2026-03-25-what-is-the-best-way-for-me-to-protect-internal-company-only-app/ Use IP allowlisting for a basic perimeter, combine with OIDC for identity verification, and consider zero trust overlays like OpenZiti for scalable security. Internal applications like Grafana require strict access controls that scale beyond brittle IP whitelists. Here’s a field-tested approach for Envoy Gateway users. Diagnose Current Access Patterns Start by mapping who/what needs access: Users: WFH employees, contractors, or services (e.g., CI/CD pipelines). Locations: Offices, cloud VPCs, or hybrid environments. https://liveaverage.com/blog/kubernetes/2026-03-24-how-do-you-handle-k8s-rbac-audits-for-compliance-iso27001-soc2/ Tue, 24 Mar 2026 00:00:00 +0000 https://liveaverage.com/blog/kubernetes/2026-03-24-how-do-you-handle-k8s-rbac-audits-for-compliance-iso27001-soc2/ Implement RBAC audits by enforcing policies with Kyverno, validating access controls, and maintaining audit trails to meet compliance requirements. Auditors for ISO27001/SOC2 care about three things: least privilege, change tracking, and access reviewability. Here’s how to handle RBAC audits without burning your cluster: Workflow for RBAC Audit Readiness Define policies: Restrict role bindings to groups, not users. Enforce verbs/ressources limits. Enforce automatically: Use Kyverno or OPA to block non-compliant changes. https://liveaverage.com/blog/kubernetes/2026-03-23-how-are-you-actually-using-falco-in-production/ Mon, 23 Mar 2026 00:00:00 +0000 https://liveaverage.com/blog/kubernetes/2026-03-23-how-are-you-actually-using-falco-in-production/ Falco detects runtime threats in Kubernetes but requires deliberate tuning and alerting integration to avoid drowning in noise. Workflow: Deploy, Tune, Integrate Deploy Falco via Operator or Helm chart (sidecar or DaemonSet). Start with defaults, but expect >5k alerts/day initially. Tune aggressively: Suppress known-false-positives using suppress rules (e.g., Spring Boot reading ConfigMaps triggering k8s_api_server_connection). Disable irrelevant rules (e.g., container escapes in non-privileged environments). Forward alerts to a actionable system (Prometheus Alertmanager, SIEM, Slack). https://liveaverage.com/blog/kubernetes/2026-03-22-going-to-kubecon-anyone-mastered-the-art-of-getting-pitched-at-a/ Sun, 22 Mar 2026 00:00:00 +0000 https://liveaverage.com/blog/kubernetes/2026-03-22-going-to-kubecon-anyone-mastered-the-art-of-getting-pitched-at-a/ A practical guide for platform engineers to manage vendor pitches efficiently at KubeCon. Context and Why It Matters KubeCon pitches are inevitable but manageable. Poor handling wastes time, creates decision fatigue, and distracts from technical learning. A structured approach reduces noise and surfaces actionable insights. Actionable Workflow Pre-Event Prep Define Goals: List 2-3 technical pain points (e.g., “improve CI/CD throughput”, “simplify cluster lifecycle”). Filter Targets: Use the KubeCon schedule/app to identify vendors aligned with your goals. https://liveaverage.com/blog/kubernetes/2026-03-21-question-am-i-missing-something-or-a-core-feature-of-k8s-kube-ap/ Sat, 21 Mar 2026 00:00:00 +0000 https://liveaverage.com/blog/kubernetes/2026-03-21-question-am-i-missing-something-or-a-core-feature-of-k8s-kube-ap/ The AlwaysDeny authorizer mode in Kubernetes does not deny requests as expected due to its design to return NoOpinion, requiring configuration adjustments. Problem Context The kube-apiserver’s AlwaysDeny authorizer mode returns NoOpinion instead of a direct deny decision. This allows subsequent authorizers in the chain to override the decision, making it unsuitable for production environments where explicit denials are required. Diagnosis Steps Check authorizer configuration: Verify the --authorization-mode flag on the kube-apiserver. https://liveaverage.com/blog/kubernetes/2026-03-20-what-makes-a-self-hosted-kubernetes-app-painful-to-run/ Fri, 20 Mar 2026 00:00:00 +0000 https://liveaverage.com/blog/kubernetes/2026-03-20-what-makes-a-self-hosted-kubernetes-app-painful-to-run/ Self-hosted Kubernetes apps often introduce hidden complexity through poor design assumptions, brittle tooling, and operational overhead that negate the benefits of container orchestration. Diagnosing the Pain Points Most self-hosted apps aren’t built with Kubernetes idioms in mind. Common issues include: Cloud-specific assumptions: Hardcoded storage classes, region-specific ingress configs, or dependencies on cloud provider APIs. Brittle upgrades: Breaking changes in Helm charts or CRDs that require manual intervention. Security pitfalls: Containers expecting root privileges or writable /etc/passwd. https://liveaverage.com/blog/kubernetes/2026-03-19-thinking-about-pivoting-to-devops-at-37-is-it-worth-it-in-the-ai/ Thu, 19 Mar 2026 00:00:00 +0000 https://liveaverage.com/blog/kubernetes/2026-03-19-thinking-about-pivoting-to-devops-at-37-is-it-worth-it-in-the-ai/ Transitioning to DevOps at 37 is viable with hands-on cloud/K8s experience, but success depends on adapting to AI-driven workflows and focusing on irreplaceable human skills like troubleshooting and system design. Diagnosis: Why the AI Era Changes (But Doesn’t Eliminate) DevOps AI automates repetitive tasks (e.g., log analysis, basic deployments) but amplifies demand for skills that machines can’t replicate: incident response, complex system design, and cross-team collaboration. DevOps roles are evolving, not disappearing—those who adapt will thrive. https://liveaverage.com/blog/kubernetes/2026-03-18-kro-kube-resource-orchestrator-has-anyone-used-it/ Wed, 18 Mar 2026 00:00:00 +0000 https://liveaverage.com/blog/kubernetes/2026-03-18-kro-kube-resource-orchestrator-has-anyone-used-it/ KRO simplifies managing interdependent CRDs through declarative resource graphs, offering sync capabilities beyond Helm, but requires careful design to avoid complexity. What KRO Solves KRO (Kube Resource Orchestrator) addresses lifecycle management for groups of resources tied to custom controllers. If your cluster uses CRDs that depend on each other (e.g., a ServiceMesh CR that provisions VirtualServices, Policies, and Gateways), KRO’s resource graphs define these relationships declaratively. Unlike Helm, which templates resources but doesn’t enforce sync post-install, KRO ensures downstream resources stay in sync with the root definition. https://liveaverage.com/blog/kubernetes/2026-03-17-does-anyone-use-kgateway-for-api-gateway-features-like-authentic/ Tue, 17 Mar 2026 00:00:00 +0000 https://liveaverage.com/blog/kubernetes/2026-03-17-does-anyone-use-kgateway-for-api-gateway-features-like-authentic/ KGateway can handle API authentication in production but requires careful setup and integration with existing IAM systems. Actionable Workflow Install KGateway: Deploy the operator via OperatorHub on OpenShift or apply CRDs manually. oc new-project kgateway --display-name="KGateway" oc apply -f https://raw.githubusercontent.com/k6io/k6-main/master/docs/assets/kgateway-operator.yaml Define Authentication Policy: Create a VirtualServer with JWT validation. Example policy below. Bind to Existing IAM: Integrate with Keycloak, Auth0, or OpenID Connect providers. Ensure TLS is enforced. https://liveaverage.com/blog/kubernetes/2026-03-16-how-should-i-scale-my-gitops-repositories-when-deploying-multipl/ Mon, 16 Mar 2026 00:00:00 +0000 https://liveaverage.com/blog/kubernetes/2026-03-16-how-should-i-scale-my-gitops-repositories-when-deploying-multipl/ Use per-project GitOps repositories with a shared platform repo for cross-cutting concerns to balance ownership and scalability. Scaling GitOps repositories requires balancing autonomy, maintainability, and operational overhead. Monorepos create coordination bottlenecks as teams grow, while per-app repos introduce fragmentation. A hybrid approach—per-project repositories paired with a centralized platform repo—provides the best path forward for most organizations. Diagnosis: When Repositories Become a Problem Monorepo pitfalls: Tight coupling between projects leads to merge conflicts, noisy CI pipelines, and broad blast radius for misconfigurations. https://liveaverage.com/blog/kubernetes/2026-03-15-what-happens-when-a-pod-crashes-because-a-file-parser-can-t-hand/ Sun, 15 Mar 2026 00:00:00 +0000 https://liveaverage.com/blog/kubernetes/2026-03-15-what-happens-when-a-pod-crashes-because-a-file-parser-can-t-hand/ When a pod crashes due to a malformed file, it enters a restart loop until the root cause is addressed and proper error handling is implemented. Diagnosis Workflow Check pod status: kubectl get pods -n <namespace> Look for CrashLoopBackOff status. Inspect logs: kubectl logs <pod-name> --previous -n <namespace> Identify exceptions (e.g., yauzl errors from malformed ZIP files). Describe the pod: kubectl describe pod <pod-name> -n <namespace> Check events for BackOff restart reasons. https://liveaverage.com/blog/kubernetes/2026-03-14-how-to-prepare-for-this-docker-kubernetes-live-build-interview-f/ Sat, 14 Mar 2026 00:00:00 +0000 https://liveaverage.com/blog/kubernetes/2026-03-14-how-to-prepare-for-this-docker-kubernetes-live-build-interview-f/ Focus on core container and orchestration workflows, hands-on troubleshooting, and real-world constraints to succeed in live build interviews. Core Preparation Workflow 1. Master Docker Fundamentals Build and push images: docker build -t myapp:1.0 . docker push myregistry/myapp:1.0 Networking: Test custom networks and service discovery: docker network create mynet docker run --network mynet --name db postgres docker run --network mynet --rm alpine nslookup db Volumes: Mount configs and persist data: docker volume create app-data docker run -v app-data:/app/data myapp:1. https://liveaverage.com/blog/kubernetes/2026-03-13-is-the-certified-kubernetes-admin-still-valuable-for-5-years-of/ Fri, 13 Mar 2026 00:00:00 +0000 https://liveaverage.com/blog/kubernetes/2026-03-13-is-the-certified-kubernetes-admin-still-valuable-for-5-years-of/ The CKA remains valuable for validating core Kubernetes skills but should complement hands-on experience and platform-specific knowledge. Why This Matters in Production Certifications like the CKA test foundational knowledge—cluster setup, networking, storage, security—but real-world stability depends on diagnosing failures under pressure. If you’re managing production clusters, the CKA forces you to confront gaps in areas like kubectl troubleshooting, RBAC configuration, or etcd backups. However, it’s not a substitute for platform-specific expertise (e. https://liveaverage.com/blog/kubernetes/2026-03-12-for-teams-running-chaos-experiments-on-kubernetes-how-do-you-pic/ Thu, 12 Mar 2026 00:00:00 +0000 https://liveaverage.com/blog/kubernetes/2026-03-12-for-teams-running-chaos-experiments-on-kubernetes-how-do-you-pic/ Begin chaos experiments by targeting services with known fragility or recent changes, not random workloads. Why This Matters Kubernetes makes failure injection easy, but chaos without focus wastes time and risks overloading stable systems. Targeting known weaknesses first surfaces actionable issues, builds team confidence, and avoids “chaos for chaos' sake.” Actionable Workflow Identify candidates: Review monitoring/alerting for recurring errors (e.g., kubectl get pods --show-labels | grep -i error). https://liveaverage.com/blog/kubernetes/2026-03-11-how-much-infrastructure-do-you-deploy/ Wed, 11 Mar 2026 00:00:00 +0000 https://liveaverage.com/blog/kubernetes/2026-03-11-how-much-infrastructure-do-you-deploy/ Infrastructure deployment remains critical but evolves with AI, automation, and cost control; focus on maintainable, auditable systems. Context and Diagnosis The “infrastructure wave” isn’t over—it’s shifting. Teams still deploy daily, but priorities have changed: AI assist: Tools like GitHub Copilot speed up repetitive tasks but don’t replace deep platform knowledge. Cost pressure: Cloud spend optimization is now a core requirement, not a nice-to-have. Platform maturity: Organizations prioritize stabilizing existing systems over greenfield deployments. https://liveaverage.com/blog/kubernetes/2026-03-10-writing-k8s-manifests-for-a-new-microservice-what-s-your-team-s/ Tue, 10 Mar 2026 00:00:00 +0000 https://liveaverage.com/blog/kubernetes/2026-03-10-writing-k8s-manifests-for-a-new-microservice-what-s-your-team-s/ We standardize on Helm charts with organizational policies, enforce compliance via Kyverno, and use ArgoCD for GitOps-driven deployments. Workflow: From Template to Deployment Start with a Helm chart Use an internal, versioned Helm chart as the base for all services. Chart includes defaults for: Resource limits/requests Liveness/readiness probes Ingress templates (with TLS enforcement) Common labels (team, env, app) Example: helm create my-service --starter-chart internal-base-chart Customize with Kustomize https://liveaverage.com/blog/kubernetes/2026-03-09-nixos-as-os-for-node/ Mon, 09 Mar 2026 00:00:00 +0000 https://liveaverage.com/blog/kubernetes/2026-03-09-nixos-as-os-for-node/ NixOS can work as a Kubernetes node OS for specific use cases but requires careful management of immutability, hardware diversity, and cluster orchestration. Practical Context Kubernetes nodes typically demand stability, predictable updates, and hardware-agnostic provisioning. NixOS offers immutable infrastructure via declarative configs but introduces friction in dynamic environments. Use cases like homelabs, edge deployments, or GPU-heavy workloads (e.g., ML clusters) may justify its complexity. Actionable Workflow Bootstrap Node Use nixos-anywhere or deploy-rs to provision base OS: nix run github:matthewbrowder/deploy-rs -- deploy --config . https://liveaverage.com/blog/kubernetes/2026-03-07-how-granular-should-cilium-network-policies-be-in-production/ Sat, 07 Mar 2026 00:00:00 +0000 https://liveaverage.com/blog/kubernetes/2026-03-07-how-granular-should-cilium-network-policies-be-in-production/ Start with default-deny ingress and refine policies only where necessary to enforce least privilege without overcomplicating maintenance. Default-deny ingress is non-negotiable for production security, but overly granular policies risk operational bloat and human error. Granularity should align with actual risk surfaces: tighten rules for high-value workloads (e.g., databases, auth services) and loosen for ephemeral or low-risk components. Actionable Workflow Baseline with cluster-wide default-deny apiVersion: cilium.io/v2 kind: CiliumClusterwideNetworkPolicy metadata: name: default-deny-ingress spec: description: "Deny all ingress by default" rules: - ingress: - {} Identify allowed sources Monitoring systems (e. https://liveaverage.com/blog/kubernetes/2026-03-06-external-secrets-operator-in-production-reconciliation-auth-trad/ Fri, 06 Mar 2026 00:00:00 +0000 https://liveaverage.com/blog/kubernetes/2026-03-06-external-secrets-operator-in-production-reconciliation-auth-trad/ The External Secrets Operator simplifies secret management but requires careful handling of reconciliation and authentication tradeoffs to avoid security and stability issues in production. Why This Matters Kubernetes Secrets stored in etcd are often a liability for enterprises due to lack of encryption-at-rest guarantees and broad access risks. External Secrets Operator (ESO) addresses this by syncing secrets from external providers (e.g., AWS Secrets Manager, HashiCorp Vault) while keeping them out of etcd. https://liveaverage.com/blog/kubernetes/2026-03-05-cilium-vs-istio-ambient-mesh-for-egress-control-in-2026/ Thu, 05 Mar 2026 00:00:00 +0000 https://liveaverage.com/blog/kubernetes/2026-03-05-cilium-vs-istio-ambient-mesh-for-egress-control-in-2026/ Cilium and Istio Ambient Mesh offer different tradeoffs for EKS egress control; choose based on mTLS needs, sidecar overhead, and operational maturity. Egress control matters because default-allow clusters leak data and violate compliance. Both tools solve this, but their approaches differ in complexity, performance, and operational burden. Workflow for Egress Control Implementation Define egress requirements Default deny all egress Whitelist specific domains/IPs (e.g., *.api.provider.com, 8.8.8.8) Decide if mTLS is required for compliance Evaluate tooling https://liveaverage.com/blog/kubernetes/2026-03-04-anyone-else-seeing-gpu-node-looks-healthy-but-jobs-fail-until-re/ Wed, 04 Mar 2026 00:00:00 +0000 https://liveaverage.com/blog/kubernetes/2026-03-04-anyone-else-seeing-gpu-node-looks-healthy-but-jobs-fail-until-re/ GPU nodes may appear healthy but fail under load due to hardware, driver, or resource issues; here’s how to diagnose and fix them. Diagnosis Workflow Run hardware diagnostics: Use nvidia-smi --query-gpu=timestamp,temperature.gpu,utilization.gpu --format=csv to check GPU health. Execute Nvidia’s MATS/MODS memory test for faulty hardware. Check node status: kubectl describe node <node-name> | grep -i "gpu\|nvidia" Look for errors in Conditions or Events. Monitor under load: Deploy a test pod with nvidia. https://liveaverage.com/blog/kubernetes/2026-03-03-is-kubent-dead/ Tue, 03 Mar 2026 00:00:00 +0000 https://liveaverage.com/blog/kubernetes/2026-03-03-is-kubent-dead/ Kubent is effectively unmaintained; consider alternatives like kubepug or ekrup for policy enforcement in Kubernetes 1.33+ environments. Diagnosis: Is Kubent Still Viable? Check Kubent’s maintenance status: Repo activity: Original doitintl/kube-no-trouble is read-only. Last commit: 2023-09. Rule coverage: Forks (e.g., dark0dave/kube-no-trouble) lack rules for Kubernetes 1.33+. Community signals: No recent releases, minimal issue responses post-2023. If you’re on Kubernetes 1.33+, Kubent’s stale rulesets expose you to undetected misconfigurations. Repair Steps: Migrate to Maintained Tools 1. https://liveaverage.com/blog/kubernetes/2026-03-02-who-s-attending-kubecon-europe-in-march-and-do-you-know-what-tal/ Mon, 02 Mar 2026 00:00:00 +0000 https://liveaverage.com/blog/kubernetes/2026-03-02-who-s-attending-kubecon-europe-in-march-and-do-you-know-what-tal/ A curated list of essential Kubecon Europe talks focused on security, AI/ML, and platform engineering, with actionable steps for planning your schedule. Actionable Workflow for Session Prioritization Review the schedule grid (https://events.linuxfoundation.org/kubecon-cloudnativecon-europe/schedule/) and filter by Security, AI/ML, and Platform Engineering tracks. Flag sessions with actionable outcomes: Look for talks that include demos, case studies, or tooling you can use immediately (e.g., “Securing Workloads with Kyverno” vs. “The Future of Kubernetes Security”). https://liveaverage.com/blog/kubernetes/2026-03-01-anyone-managing-k8s-clusters-with-limited-or-no-internet-access/ Sun, 01 Mar 2026 00:00:00 +0000 https://liveaverage.com/blog/kubernetes/2026-03-01-anyone-managing-k8s-clusters-with-limited-or-no-internet-access/ Here’s how to manage Kubernetes clusters in restricted environments using air-gapping, offline tooling, and policy enforcement. Workflow: From Zero to Operational Plan your offline supply chain Identify all dependencies: base OS images, operator binaries, Helm charts, and CI/CD artifacts. Use skopeo or regctl to mirror images to an internal registry. Example: skopeo copy --all oci:quay.io/cluster-api/provider-aws:v2.30.0 oci:my-registry.local/cluster-api/provider-aws:v2.30.0 For air-gapped clusters, pre-stage Zarf or Hauler bundles on USB drives or physical media. https://liveaverage.com/blog/kubernetes/2026-02-28-anyone-have-experience-with-vks-vmware-k8s-on-prem/ Sat, 28 Feb 2026 00:00:00 +0000 https://liveaverage.com/blog/kubernetes/2026-02-28-anyone-have-experience-with-vks-vmware-k8s-on-prem/ VMware VKS simplifies on-prem Kubernetes deployment but introduces vendor lock-in and integration friction at scale. Operational Workflow for VKS Deployment Prerequisites: vSphere 7.0+ with compatible NSX/AVI integration. Storage policies aligned with VMware’s recommended profiles. Network segmentation for management, data, and edge traffic. Deployment: Use VMware Cloud Foundation (VCF) for bundled lifecycle management. Deploy via TKGS (Tanzu Kubernetes Grid for SDDC) for tighter vSphere integration. https://liveaverage.com/blog/kubernetes/2026-02-27-how-would-you-set-this-lab-up/ Fri, 27 Feb 2026 00:00:00 +0000 https://liveaverage.com/blog/kubernetes/2026-02-27-how-would-you-set-this-lab-up/ A step-by-step guide to setting up a Raspberry Pi Kubernetes lab with kubeadm, CNI, and CSI, including common pitfalls and maintenance tips. Workflow: From Hardware to Cluster Hardware Setup Use Raspberry Pi 4 (4GB+ RAM) or Pi 5 for control plane and workers. Ensure all Pis are on the same network with static IPs or DNS entries. Flash Raspberry Pi OS Lite (64-bit) to microSD cards. OS and Prerequisites https://liveaverage.com/blog/kubernetes/2026-02-26-how-would-you-setup-the-resource-requests-and-limits-on-this-wor/ Thu, 26 Feb 2026 00:00:00 +0000 https://liveaverage.com/blog/kubernetes/2026-02-26-how-would-you-setup-the-resource-requests-and-limits-on-this-wor/ Set resource requests based on observed usage, apply limits cautiously, and validate with real metrics. Practical Workflow Baseline metrics: Use kubectl top or Prometheus to observe historical CPU/memory usage. Set requests: Allocate 20-30% above observed averages to buffer for spikes. Apply limits: Use limits only if you need strict resource isolation (e.g., batch jobs). For long-running apps, omit limits unless throttling is acceptable. Monitor post-deploy: Check for OOM kills, throttling, or performance degradation. https://liveaverage.com/blog/kubernetes/2026-02-25-building-a-tiny-bare-metal-k8s-cluster-for-self-learning/ Wed, 25 Feb 2026 00:00:00 +0000 https://liveaverage.com/blog/kubernetes/2026-02-25-building-a-tiny-bare-metal-k8s-cluster-for-self-learning/ A practical guide to building a low-cost, bare-metal Kubernetes cluster for hands-on learning with real-world tradeoffs and troubleshooting tips. Why This Matters Running Kubernetes on physical hardware teaches cluster lifecycle management, networking quirks, and storage pitfalls that cloud abstracts away. It’s invaluable for certs and debugging production-like failures. Hardware Selection Avoid Raspberry Pis for general learning—ARM compatibility issues (e.g., Fluentbit, some CRDs) and non-standard kernel page sizes waste time. Instead: https://liveaverage.com/blog/kubernetes/2026-02-24-it-s-2026-golden-applications-and-if-you-could-re-write-the-argo/ Tue, 24 Feb 2026 00:00:00 +0000 https://liveaverage.com/blog/kubernetes/2026-02-24-it-s-2026-golden-applications-and-if-you-could-re-write-the-argo/ Adopt a layered monorepo structure with Kyverno for policy enforcement, ServerSideApply, and clear separation of infrastructure and application workloads to maintain golden state integrity. Golden applications require operational discipline, not just tooling. A monorepo for ArgoCD must balance flexibility for diverse workloads (infra, web apps, custom services) while preventing drift through policy-as-code and declarative automation. Actionable Workflow Define Boundaries Split manifests into infra/ and apps/ directories. Use Kustomize overlays for environment-specific tweaks (e. https://liveaverage.com/blog/kubernetes/2026-02-23-what-kubernetes-feature-looked-great-on-paper-but-hurt-you-in-pr/ Mon, 23 Feb 2026 00:00:00 +0000 https://liveaverage.com/blog/kubernetes/2026-02-23-what-kubernetes-feature-looked-great-on-paper-but-hurt-you-in-pr/ Misconfigured resource requests and limits in Kubernetes can lead to wasted resources and scheduling issues, requiring proactive governance and tooling to mitigate. The Problem in Production Developers often copy VM-era specs to container resource requests/limits, leading to grossly oversized allocations. For example: A pod requesting 4 cores and 8GB memory but idling at 0.2 cores and 500MB Nodes blocked by “reserved” resources while workloads starve Scheduler only honors requests, not actual usage. https://liveaverage.com/blog/kubernetes/2026-02-22-how-is-load-balancing-really-used-in-production-with-kubernetes/ Sun, 22 Feb 2026 00:00:00 +0000 https://liveaverage.com/blog/kubernetes/2026-02-22-how-is-load-balancing-really-used-in-production-with-kubernetes/ Kubernetes uses Services and Ingress to abstract and manage load balancing, enabling scalable traffic distribution without manual infrastructure coordination. The Problem: Traffic Distribution at Scale In production, applications require traffic to be distributed across multiple pods reliably. Manual LB management doesn’t scale with dynamic pod lifecycles, network policies, and multi-team environments. Kubernetes abstracts this complexity through declarative resources. How Kubernetes Handles Load Balancing Core Components Services: Abstracts pods behind a stable IP and DNS name, using kube-proxy to manage endpoint mappings. https://liveaverage.com/blog/kubernetes/2026-02-21-ubuntu-server-or-debian-for-a-local-k8s-cluster-with-kubeadm/ Sat, 21 Feb 2026 00:00:00 +0000 https://liveaverage.com/blog/kubernetes/2026-02-21-ubuntu-server-or-debian-for-a-local-k8s-cluster-with-kubeadm/ For local k8s clusters with kubeadm, Debian offers stability and simplicity, while Ubuntu provides newer packages and better hardware support, depending on your needs. Why It Matters Local Kubernetes clusters demand minimal overhead and predictable behavior. Both Ubuntu and Debian work, but the choice hinges on hardware compatibility, maintenance preferences, and tolerance for bloat. Actionable Workflow Assess Hardware Requirements Use lshw or dmidecode to check hardware specs. If using cutting-edge NICs or GPUs, Ubuntu’s newer kernel (6. https://liveaverage.com/blog/kubernetes/2026-02-19-what-actually-goes-wrong-in-kubernetes-production/ Thu, 19 Feb 2026 00:00:00 +0000 https://liveaverage.com/blog/kubernetes/2026-02-19-what-actually-goes-wrong-in-kubernetes-production/ Kubernetes production failures often stem from resource exhaustion, misconfigurations, and network bottlenecks, requiring proactive monitoring and remediation. Diagnosis: Where Things Break Resource Starvation IP exhaustion (e.g., subnet limits, ENI allocation issues) CPU/Memory pressure causing evictions Disk full errors from unbounded logs or images Network Misconfigurations Subnet CIDR ranges too small for node/pod growth Firewall rules blocking necessary ports (e.g., 6443, 2379-2380) Incorrect DNS configurations breaking service discovery Node Instability https://liveaverage.com/blog/kubernetes/2026-02-18-pok-mon-inspired-kubernetes-game-in-the-terminal-worth-building/ Wed, 18 Feb 2026 00:00:00 +0000 https://liveaverage.com/blog/kubernetes/2026-02-18-pok-mon-inspired-kubernetes-game-in-the-terminal-worth-building/ A terminal-based Kubernetes game can aid learning but requires careful design to balance engagement with operational realism. Why This Matters in Production Learning Kubernetes concepts like deployments, services, and RBAC through gamification can reduce onboarding friction for new engineers. However, poorly designed games risk reinforcing bad habits (e.g., ignoring resource limits) or abstracting away critical operational realities. The goal should be to mirror real-world constraints while maintaining engagement. Actionable Workflow Define Learning Objectives https://liveaverage.com/blog/kubernetes/2026-02-17-with-nginx-ingress-being-archived-which-would-be-sufficient-for/ Tue, 17 Feb 2026 00:00:00 +0000 https://liveaverage.com/blog/kubernetes/2026-02-17-with-nginx-ingress-being-archived-which-would-be-sufficient-for/ Migrating from nginx-ingress requires evaluating Traefik, Envoy Gateway, or AWS Load Balancer Controller based on routing complexity, ecosystem integration, and operational overhead. Diagnosis: Current State and Requirements If you’re running complex nginx routing configurations (e.g., Lua scripts, custom rewrites), prioritize solutions with equivalent flexibility. Traefik’s Middleware CRDs or Envoy’s route configuration options are viable. For AWS-native teams, the AWS Load Balancer Controller (ALB) simplifies integration with ACM and VPC resources but has a hard limit of 100 backend targets per ingress. https://liveaverage.com/blog/kubernetes/2026-02-16-eks-al2-to-al2023-memory-usage-spikes-in-nginx-anyone-else/ Mon, 16 Feb 2026 00:00:00 +0000 https://liveaverage.com/blog/kubernetes/2026-02-16-eks-al2-to-al2023-memory-usage-spikes-in-nginx-anyone-else/ Diagnosing and Resolving Memory Spikes in Nginx on EKS AL2023 If you’ve migrated from Amazon Linux 2 (AL2) to AL2023 on EKS and noticed memory spikes in Nginx pods, you’re not alone. This post walks through a pragmatic diagnosis and repair workflow, with actionable steps to prevent recurrence. Diagnosis: What’s Changed? AL2023 introduces updates to containerd, kernel versions, and cgroup management. Memory spikes in Nginx often stem from: cgroup v1 vs v2 Reporting Differences https://liveaverage.com/blog/kubernetes/2026-02-16-is-using-yaml-over-the-cli-uncommon/ Mon, 16 Feb 2026 00:00:00 +0000 https://liveaverage.com/blog/kubernetes/2026-02-16-is-using-yaml-over-the-cli-uncommon/ YAML Over CLI: Production Reality vs. Tutorial Simplicity As a platform engineer, you’re not alone in preferring YAML over ad-hoc CLI commands. The question isn’t whether it’s uncommon—it’s whether you’re optimizing for tutorial simplicity or production safety. Here’s the breakdown: Why YAML Dominates Production Workflows CLI commands like kubectl create deployment are training wheels. They’re great for learning but dangerous in production: No audit trail: Who created that resource? https://liveaverage.com/blog/kubernetes/2026-02-16-do-you-run-everything-in-your-cluster/ Mon, 16 Feb 2026 00:00:00 +0000 https://liveaverage.com/blog/kubernetes/2026-02-16-do-you-run-everything-in-your-cluster/ Self-hosting databases and stateful services in Kubernetes reduces costs but demands operational maturity for reliability and recovery. Running everything in your cluster is tempting for cost control, but it shifts undifferentiated operational burden to your team. Decisions should hinge on three factors: criticality, expertise, and total cost of ownership. When to Run in Cluster Self-host stateful workloads like PostgreSQL or RabbitMQ only if: You’ve proven HA/DR workflows (e.g., backups, failover, point-in-time recovery). https://liveaverage.com/blog/kubernetes/2026-02-16-what-does-config-hell-actually-look-like-in-the-real-world/ Mon, 16 Feb 2026 00:00:00 +0000 https://liveaverage.com/blog/kubernetes/2026-02-16-what-does-config-hell-actually-look-like-in-the-real-world/ What “Config Hell” Actually Looks Like (And How to Escape It) If you’ve spent any time managing production Kubernetes clusters, you’ve probably heard the term “Config Hell.” It’s the chaotic state where configurations sprawl uncontrollably, drift between environments, and resist consistent management. But what does this look like in practice? And how do you fix it when it happens? Let’s cut through the theory and walk through real-world symptoms, diagnosis steps, and repairs. https://liveaverage.com/blog/kubernetes/2026-02-12-how-do-you-resolve-cves-in-containers-efficiently/ Thu, 12 Feb 2026 00:00:00 +0000 https://liveaverage.com/blog/kubernetes/2026-02-12-how-do-you-resolve-cves-in-containers-efficiently/ Efficient CVE Resolution in Containers: A Practical Approach Resolving CVEs in containers is a necessary but often soul-crushing task. You’re not alone in drowning under a deluge of alerts from tools like BlackDuck. The key is to focus on what matters, automate the rest, and stop playing whack-a-mole. Here’s how to do it. Diagnosis: Triage with Context Not all CVEs are created equal. Start by filtering noise: Prioritize by EPSS Score https://liveaverage.com/blog/kubernetes/2026-02-12-what-to-bundle-in-the-argo-cd-application-and-best-practices-to/ Thu, 12 Feb 2026 00:00:00 +0000 https://liveaverage.com/blog/kubernetes/2026-02-12-what-to-bundle-in-the-argo-cd-application-and-best-practices-to/ What to Bundle in Argo CD Applications (And How to Manage the Rest) If you’re deploying apps with Argo CD, you’re already halfway to a solid GitOps workflow. But when it comes to bundling resources like Gateways, HTTPRoutes, and StorageClasses, confusion sets in fast. Let’s cut through the noise with production-tested patterns. Core Principle: Separate Cluster-Scoped and Namespace-Scoped Resources Cluster-scoped resources (e.g., Gateways, StorageClasses, CRDs) belong in dedicated infrastructure apps. Namespace-scoped resources (e.