Self-hosted Kubernetes apps often introduce hidden complexity through poor design assumptions, brittle tooling, and operational overhead that negate the benefits of container orchestration.

Diagnosing the Pain Points

Most self-hosted apps aren’t built with Kubernetes idioms in mind. Common issues include:

• Cloud-specific assumptions: Hardcoded storage classes, region-specific ingress configs, or dependencies on cloud provider APIs.
• Brittle upgrades: Breaking changes in Helm charts or CRDs that require manual intervention.
• Security pitfalls: Containers expecting root privileges or writable /etc/passwd.
• Over-engineering: Apps requiring 10+ CRDs for basic functionality, turning deployments into platform projects.

In practice, this means spending more time adapting the app to Kubernetes than using it.

Actionable Workflow for Mitigation

1. Audit dependencies

   • Check for cloud-specific code or configs (e.g., AWS SDK calls, GCP metadata server dependencies).
   • Use docker inspect or crane ls to verify image assumptions.

2. Enforce container best practices

   • Require non-root users: Add USER 1000 in Dockerfiles.
   • Drop capabilities: Use securityContext.capabilities.drop: ["ALL"] in deployments.

3. Minimize adapters

   • Avoid custom init containers unless absolutely necessary.
   • Prefer sidecars over wrapper scripts (e.g., use stagger for leader election instead of bash loops).

4. Automate upgrades

   • Use Helm hooks or operators to test upgrades in staging.
   • Pin chart versions and test upgrades against a local registry mirror.

5. Document escape hatches

   • Predefine kubectl override args or ConfigMap patches for common fixes.

Policy Example

apiVersion: constraints.gatekeeper.sh/v1beta1  
kind: PodConstraintTemplate  
metadata:  
  name: nonroot-users  
spec:  
  match:  
    kinds:  
      - Pod  
  crd:  
    spec:  
      group: validation.gatekeeper.sh  
      names:  
        kind: NonRootUser  
        listKind: NonRootUserList  
        plural: nonrootusers  
        singular: nonrootuser  
      scope: Namespaced  
      versions:  
        - name: v1beta1  
          served: true  
          storage: true  
      validation:  
        openAPIV3Schema:  
          type: object  
          properties:  
            spec:  
              type: object  
              properties:  
                containers:  
                  type: array  
                  items:  
                    type: object  
                    properties:  
                      securityContext:  
                        type: object  
                        properties:  
                          runAsUser:  
                            type: integer  
                            minimum: 1000  
                    required: ["securityContext"]  
          required: ["spec"]  

Tooling

• Image scanning: Use Trivy or Clair to flag rootless issues pre-deploy.
• Conformance testing: kube-bench for CIS benchmarks, OPA/Gatekeeper for policy enforcement.
• Upgrade testing: helm upgrade --dry-run + kubeval to catch schema mismatches.
• Debugging: kubectl describe pod for admission errors, kubectl logs --previous for crashes.

Tradeoffs

Strict policies (e.g., non-root users) improve security but may break apps expecting root. Mitigate by:

• Testing images in a sandbox cluster first.
• Providing clear exceptions for legacy apps with documented risks.

Troubleshooting Common Failures

• Permission denied errors:

  • Check securityContext in deployment and image’s effective user.
  • Run id inside the container to verify UID/GID.

• CRD conflicts:

  • Use kubectl get crd -o wide to identify overlapping CRDs.
  • Isolate apps with conflicting CRDs into separate clusters.

• Storage class mismatches:

  • Patch storage classes with kubectl patch storageclass --type merge -p '{"reclaimPolicy":"Delete"}'.
  • Use allowedUnscheduled in node affinity to prevent volume binding issues.

Conclusion

Self-hosted apps in Kubernetes are manageable with disciplined policies, automated testing, and a focus on container-first design. The goal isn’t perfection but reducing toil through incremental constraints and observability.

Source thread: What makes a self-hosted Kubernetes app painful to run?