Internal Kubernetes storage for stateful services is viable for small-scale or isolated workloads but introduces complexity and risk at larger scales.

Context: Why This Matters in Production

Stateful services like PostgreSQL and object storage (e.g., S3-compatible) require reliable, persistent storage. Running these inside Kubernetes can simplify homelabs or small deployments but creates operational debt as scale and SLAs increase. External storage (e.g., cloud-native or dedicated infrastructure) offloads complexity but introduces dependencies.

Actionable Workflow: Decide Internal vs. External Storage

1. Assess workload scale:
   • Internal: Dev/test, single-node clusters, <5GB data.
   • External: Production, multi-node, or >50GB data.
2. Evaluate SLA requirements:
   • Internal: Acceptable for RTO/RPO of hours/days.
   • External: Required for RTO <1hr or compliance needs.
3. Check team expertise:
   • Internal: Only if team can manage storage lifecycle (backups, upgrades).
   • External: Use if team lacks storage specialization.

Policy Example: Storage Class Governance

apiVersion: storage.k8s.io/v1  
kind: StorageClass  
metadata:  
  name: local-fast  
provisioner: kubernetes.io/no-provisioner  
volumeBindingMode: WaitForFirstConsumer  
reclaimPolicy: Delete  
# Only allow internal storage for non-critical workloads  
allowedTopologies:  
- selfLink: ()  

Policy:

• Dev workloads: Use local-fast (internal disk).
• Production: Mandate cephfs or s3 (external).

Tooling for Internal Storage

• Rook-Ceph: Production-grade distributed storage (complex setup, but scalable).
• Longhorn: Lightweight block storage for small clusters (easier than Ceph).
• Garage: For lightweight object storage (alternative to MinIO).
• CSI Drivers: Ensure your storage class integrates with cluster.

Tradeoffs and Caveats

• Internal storage pros: Simplified deployment, no external dependencies.
• Internal storage cons: Noisy neighbor issues, harder to scale, backup complexity.
• External storage pros: Scalability, dedicated SLAs, offloads operational burden.
• External storage cons: Cost, latency, vendor lock-in risk.

Troubleshooting Common Failures

• PersistentVolumeClaim (PVC) stuck in Pending:
  • Check StorageClass exists and is default.
  • Verify node affinity matches allowedTopologies.
• Performance degradation:
  • Monitor disk I/O with iostat or Prometheus.
  • Avoid overcommitting local disks.
• Backup failures:
  • Use Velero with external storage for consistent backups.
  • Test restore workflows monthly.

Conclusion

Internal storage is not inherently an anti-pattern but requires deliberate scoping. Use it for small, isolated workloads where simplicity outweighs scalability. For production or scale, external storage reduces risk and aligns with Kubernetes’ strengths as an orchestrator, not a storage platform. Plan for migration early—refactoring storage later is costly.

Source thread: Is running PostgreSQL / S3 storage inside Kubernetes an anti-pattern, or is external storage only necessary at larger scale?