Use Velero for cluster state, Kube-state-metrics for workloads, and Rook/Ceph for storage, but expect manual steps for dependencies and validation.

Workflow: Migrate with Minimal Downtime

1. Assess current state:

   • kubectl get nodes,pods,svc,pv,pvc --all-namespaces
   • Check storage class defaults and PV binding modes.

2. Backup cluster state:

   • Use Velero to snapshot resources:

     velero backup create --include-cluster-resources --snapshot-volumes  
     

   • Validate backup with velero backup describe <backup-name>.

3. Migrate workloads:

   • Cordon source nodes: kubectl cordon <node-name>
   • Drain pods: kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data
   • Copy images and configs manually or via CI/CD pipelines.

4. Handle storage:

   • For static PVs: Use Rook/Ceph or cloud-native tools (e.g., AWS DataSync) to replicate data.
   • Update storage classes in target cluster to match source.

5. Validate:

   • kubectl get all --all-namespaces
   • Monitor logs and metrics (Prometheus/Grafana) for anomalies.

Tools and Tradeoffs

• Velero:

  • Handles cluster resources and PV snapshots.
  • Caveat: No live migration; requires downtime or node cordon.

• Kube-state-metrics:

  • Tracks workload health post-migration.
  • Caveat: Doesn’t migrate data—only monitors.

• Rook/Ceph:

  • Replicates storage across clusters.
  • Caveat: Complex setup; may require reconfiguring storage classes.

• Cloud-specific tools (AWS Migration Hub, Azure Migrate):

  • Simplify cloud-to-cloud moves but lock you into vendor ecosystems.

Policy Example: Migration Checklist

1. All PVs must use `ReadWriteOnce` or `ReadWriteMany` for replication.  
2. Velero backups required 24h before migration.  
3. Post-migration smoke tests must pass within 15 minutes.  
4. Storage class names must match between source/target clusters.  

Troubleshooting Common Failures

• PVC binding fails:

  • Check storage class name matches target cluster.
  • Verify PVs are properly labeled and annotated.

• Pods in CrashLoopBackOff:

  • kubectl describe pod <pod-name> to check image pull errors or missing env vars.

• Velero backup hangs:

  • Check if volume snapshots are stuck: velero snapshot list.
  • Ensure cloud provider credentials have snapshot permissions.

• Node affinity mismatches:

  • Drain source nodes after workloads are fully scheduled on target.

Final Note

No tool handles everything. Plan for manual validation, test rollback paths, and document dependencies (e.g., init containers, third-party CRDs) that tools won’t catch. Start small—migrate non-critical workloads first.

Source thread: Which Migration Tools Can Handle the Cluster, Workloads, Storage & Dependencies?