Migrating from x86 to Graviton on EKS requires image architecture alignment, performance testing, and careful handling of external binaries.

Why This Matters

Graviton (ARM) instances like c8g.2xlarge offer better price/performance and lower latency for cloud-native workloads compared to x86. However, the shift to ARM requires deliberate handling of image architectures and dependencies.

Actionable Workflow

1. Audit Existing Images

   • Check current image architectures:

     docker inspect <image> | grep Architecture  
     

   • Identify images relying on x86-specific binaries (e.g., third-party tools like ImageMagick).

2. Build Multi-Arch Images

   • Use docker buildx to build arm64 images:

     docker buildx build --platform linux/arm64 -t my-app:arm64 .  
     

   • For language-runtimes (Go, Java), ensure cross-compilation flags target ARM.

3. Test in Staging

   • Deploy to a Graviton-based EKS node (e.g., c8g.2xlarge):

     spec:  
       nodeSelector:  
         kubernetes.io/arch: arm64  
     

   • Benchmark performance against x86 (focus on CPU-bound tasks).

4. Deploy with Node Affinity

   • Use node affinity rules to ensure workloads run on compatible nodes:

     affinity:  
       nodeAffinity:  
         requiredDuringSchedulingIgnoredDuringExecution:  
           nodeSelectorTerms:  
           - matchExpressions:  
               - key: kubernetes.io/arch  
                 operator: In  
                 values:  
                 - arm64  
     

5. Monitor and Rollback

   • Watch for crashes or performance regressions.
   • Roll back using existing x86 images if issues arise.

Policy Example: Enforce ARM Images

Use Open Policy Agent (OPA) to block x86 images in production:

package kubernetes.admission  

deny[msg] {  
  input.request.kind.kind == "Pod"  
  not input.request.object.spec.containers[_].image.contains("arm64")  
  msg := "Image must be built for arm64 architecture"  
}  

Tooling

• docker buildx: For building multi-arch images.
• kubectl get nodes -o wide: Verify node architecture.
• uname -m: Check OS architecture on a node.
• perf: Benchmark CPU performance differences.

Tradeoffs and Caveats

• Emulation Overhead: Running x86 images on ARM via QEMU is possible but incurs ~20-30% performance penalty. Avoid in production.
• Binary Compatibility: Alpine-based images (musl libc) may fail with binaries expecting glibc. Prefer Debian/Ubuntu or Bookworm Slim for broader compatibility.
• Third-Party Dependencies: Some legacy tools (e.g., older .NET Framework apps) lack ARM support. Test thoroughly.

Troubleshooting

• Pod Not Scheduled:
  • Check node taints: kubectl describe node <node> → look for node.kubernetes.io/skip-pod-cidr or custom taints.
  • Ensure node affinity rules match available nodes.
• Image Pull Errors:
  • Verify image manifest includes arm64:

    docker manifest inspect my-app:arm64 | grep arm64  
    

• Performance Degradation:
  • Profile CPU usage with kubectl top pods.
  • Check for inefficient emulation (e.g., x86 binaries running on ARM).

Final Note

Graviton migration is low-risk if you control your image builds and test dependencies. For mixed workloads, use node selectors to isolate ARM and x86 nodes. Prioritize rebuilding over emulation.

Source thread: Moving from c5a.2xlarge (x86) to c8g.2xlarge (Graviton) on EKS, any real-world experiences?