Use runtime isolation tools and strict security policies to mitigate risks from untrusted workloads in production.

Running untrusted pods—whether from third-party images, CI/CD pipelines, or personal projects—requires hardening your cluster to prevent privilege escalation, host compromise, or lateral movement. Talos Linux simplifies secure node setup, but pod isolation demands additional layers. Below is a field-tested approach to balance security and usability.

Actionable Workflow

1. Assess Trust Level

   • Categorize workloads: Known and scanned vs. untrusted/sketchy.
   • Example: Homelab CI jobs vs. random Docker Hub images.

2. Apply Baseline Security Policies

   • Enforce non-root execution, read-only root filesystems, and disable privilege escalation.
   • Use Pod Security Admission (PSA) in enforced mode with restricted profile.

3. Deploy Runtime Isolation

   • Use gVisor for lightweight isolation or Kata Containers for VM-level security.
   • Configure Kubernetes to route untrusted pods to isolated nodes (e.g., via taints/toleration).

4. Lock Down Network and Access

   • Apply default-deny network policies with Cilium or Calico.
   • Restrict service account tokens and use least-privilege RBAC roles.

5. Monitor and Audit

   • Deploy Falco or Audit Policy logs to detect anomalous behavior.

Concrete Policy Example

Pod Security Standard for Untrusted Workloads

apiVersion: policy/v1beta1
kind: PodSecurityStandard
metadata:
  name: untrusted-workloads
spec:
  root:
    seccomp:
      # Use a strict seccomp profile
      type: RuntimeDefault
    sysctl:
      # Disable dangerous sysctls
      forbidden:
        - "kernel.*"
  podSpec:
    shareProcessNamespace: false
    privilegeEscalation: false
    allowPrivilegeEscalation: false
    readOnlyRootFilesystem: true
    runAsNonRoot:
      enforce: true
    volumes:
      - configMap
      - secret
      - projected
      - emptyDir
      - persistentVolumeClaim
    # Restrict hostPath mounts
    hostPath:
      useAfterInit: false
      pathPrefixes: []

Runtime Class Configuration for gVisor

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: untrusted-runc
  labels:
    runtimeclass.kubernetes.io/scheduler-constant-pod: '1'
spec:
  runtimeHandler: gVisor

Tooling

• gVisor: Lightweight sandboxing via user-space kernel. Apply with runtimeClassName: untrusted-runc.
• Kata Containers: VM-based isolation for high-risk workloads ( heavier, but stronger boundary).
• Cilium: Enforce network policies and visibility for untrusted pods.
• Falco: Detect runtime anomalies (e.g., shell in untrusted container).
• Talos Linux: Pre-hardened OS with minimal attack surface.

Tradeoffs and Caveats

• gVisor vs. Kata:
  • gVisor: Lower overhead but weaker isolation (shared kernel). May fail syscall-compatible workloads.
  • Kata: Stronger isolation but higher resource usage and complexity.
• Performance: Runtime isolation adds latency (e.g., gVisor’s syscall interception).
• Compatibility: Some workloads (e.g., FUSE, certain drivers) may break with non-default runtimes.

Troubleshooting

• Container Breakout Attempts

  • Symptoms: Unexpected host processes, privilege escalation.
  • Fix: Audit Falco logs, verify runAsNonRoot and seccomp enforcement.

• Permission Denied in Containers

  • Symptoms: App crashes due to file access issues.
  • Fix: Check readOnlyRootFilesystem, non-root user permissions, and volume mounts.

• Network Policy Misconfigurations

  • Symptoms: Unreachable services or unexpected traffic.
  • Fix: Use kubectl get networkpolicy -o wide to validate rules. Test with cilium connect for packet inspection.

• Runtime Class Not Found

  • Symptoms: Pods stuck in Pending due to scheduler issues.
  • Fix: Verify RuntimeClass exists and nodes are labeled correctly (e.g., kubectl describe nodes).

Final Notes

Untrusted pods demand layered defenses: runtime isolation, strict Pod Security Admission, and network controls. Start with gVisor for most cases—it balances security and practicality. For high-risk workloads (e.g., public-facing apps), escalate to Kata. Always assume breaches happen; design for containment, not just prevention.

In my homelab, I pair gVisor with Cilium network policies and Falco alerts. It’s not perfect, but it raises the bar enough to sleep at night while running random Docker Hub images.

Source thread: How to run pods that you don’t fully trust?