When shifting from Kubernetes to Slurm, prioritize workload assessment, tooling integration, and phased migration to minimize disruption.

Why This Matters
Kubernetes excels at container orchestration, but Slurm’s batch processing efficiency often wins for HPC workloads. Sudden shifts happen due to cost pressures, workload misalignment, or organizational mandates. The key is to avoid operational whiplash by methodically mapping dependencies and validating alternatives.

Actionable Workflow

1. Audit Kubernetes Workloads

   • Run kubectl get pods --all-namespaces and classify workloads: stateless microservices, batch jobs, or stateful applications.
   • Use kubectl describe pod <name> to document resource requests/limits and volume mounts.
   • Flag workloads with tight SLAs or external dependencies (e.g., databases, message queues).

2. Evaluate Slurm Fit

   • Slurm shines for batch, parallel, and GPU-heavy jobs but lacks native support for Kubernetes-style service discovery or autoscaling.
   • Test-candidate workloads: Run a small batch job via sbatch --nodes=2 --time=00:10:00 script.sh and validate node communication.

3. Choose Migration Strategy

   • Parallel Run: Deploy Slurm alongside Kubernetes for non-critical workloads first. Use shared storage (e.g., NFS) for data consistency.
   • Strangler Pattern: Gradually route batch jobs to Slurm while keeping stateful services on Kubernetes.
   • Big Bang: Only if SLAs allow; ensure rollback plans (e.g., preserved Kubernetes manifests).

4. Implement Monitoring

   • Integrate Slurm with Prometheus/Grafana using slurm-exporter for metrics like job queues and node health.
   • Compare job completion times pre/post-migration to validate performance.

5. Iterate and Optimize

   • Tune Slurm’s partitions and node configs based on observed bottlenecks.
   • Document gotchas (e.g., Slurm’s lack of rolling updates) in runbooks.

Concrete Policy Example
Adopt a workload classification policy:

- **Tier 1 (Critical)**: Migrate only after proof-of-concept in staging.  
- **Tier 2 (Batch)**: Migrate first, using Slurm’s `--array` for parallelism.  
- **Tier 3 (Stateful)**: Keep on Kubernetes or use hybrid storage.  

Tooling

• Slurm: sbatch, scontrol, squeue for job management.
• Monitoring: slurm-exporter + Prometheus, sstat for job accounting.
• CI/CD: Use GitOps for Slurm job scripts (e.g., ArgoCD with a Slurm operator).

Tradeoffs and Caveats

• Pros: Slurm reduces complexity for batch workloads and often cuts cloud costs by optimizing node utilization.
• Cons: Lose Kubernetes’ rich ecosystem (e.g., service meshes, ingress controllers). Stateful workloads may require re-architecture.
• Assumption: Your team has HPC familiarity; otherwise, budget for training or external support.

Troubleshooting Common Issues

• Node Provisioning Failures: Check sinfo for node states. Verify cloud provider quotas or on-prem hypervisor health.
• Job Hanging: Use scontrol show job <jobid> to debug resource allocation. Ensure network policies allow node-to-node traffic.
• Permission Errors: Align Slurm’s User and Account configs with LDAP/AD groups. Test with su - <user> -c "sbatch script.sh"

Final Note
There’s no one-size-fits-all. In my experience, teams that invest in workload-specific validation and embrace incremental change avoid the most costly pitfalls. Measure twice, cut once.

Source thread: Unexpectedly got pulled from Kubernetes into Slurm - advice?