Focus on running and observing AI inference workloads in AKS to build practical skills in GPU scheduling, model serving, and cost management.

Workflow: From Pod to Metrics

1. Deploy a vLLM Inference Pod
   Use a spot GPU node in AKS to minimize costs. Example:

   kubectl apply -f https://raw.githubusercontent.com/vllm-project/vllm/main/deploy/crds/vllm.example.yaml  
   

   • Watch for GPU resource allocation: kubectl describe node <gpu-node>
   • Check vLLM logs: kubectl logs -f <vllm-pod>

2. Instrument with Prometheus + DCGM Exporter
   Deploy Nvidia DCGM exporter to scrape GPU metrics:

   helm install dcgm-exporter nvidia/dcgm-exporter --set serviceAccount.create=true  
   

   • Verify metrics endpoint: curl http://dcgm-exporter:9105/metrics
   • Key metrics: nvidia_gpu_memory_used, nvidia_gpu_utilization, nvidia_dcgm_gflops5s

3. Break It

   • Use kubectl scale to increase replicas and observe scaling behavior.
   • Inject load with hey or custom scripts:

     hey -c 10 -z 30s http://vllm-service:8000/v1/completions  
     

   • Watch for OOMs, queueing delays, and GPU saturation.

Tools to Prioritize

• Prometheus + Grafana: For time-series metrics and dashboards.
• KServe: For model serving abstractions (builds on Knative).
• OpenTelemetry: For tracing inference requests across services.
• Nvidia DCGM Exporter: For GPU health and performance signals.

Tradeoffs and Caveats

• Spot Nodes vs. Dedicated: Spot instances save costs but can be preempted mid-experiment.
• DCGM Exporter Flakiness: I’ve seen it fail to collect metrics on AKS due to node labeling issues. Always validate with kubectl describe pod dcgm-exporter.
• Cold Starts: Model serving frameworks like KServe may introduce latency spikes during scale-up. Test with realistic traffic patterns.

Troubleshooting Common Failures

• No GPU Metrics in Prometheus:
  • Check DCGM exporter logs: kubectl logs -l app=dcgm-exporter
  • Ensure nodes are labeled correctly (e.g., kubernetes.io/role: agent for AKS).
• vLLM OOMs:
  • Check logs for oom killer events.
  • Adjust memory limits in the CRD: resources.requests.memory: "8Gi".
• GPU Not Scheduled:
  • Verify GPU plugin is enabled: kubectl get nodes -o wide --show-labels | grep nvidia.com/gpu.
  • Use nvidia-smi on the node to confirm GPU availability.

Policy Example: Scaling Based on GPU Memory

Define a Horizontal Pod Autoscaler (HPA) for vLLM based on GPU memory usage:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: vllm-hpa
spec:
  scaleTargetRef:
    apiVersion: "vllm.example.com/v1"
    kind: VLLM
    name: example-vllm
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: nvidia.com/gpu-memory
      target:
        type: Utilization
        averageUtilization: 80

• Validation: kubectl get hpa vllm-hpa -w to watch scaling behavior.

Final Note

AI workloads in AKS are just another SRE problem—once you instrument them. Start small, break things, and let the signals guide your next move. Don’t get lost in frameworks until you’ve seen how GPUs and model serving behave under load.

Source thread: Upskilling in AI for an AKS / Observability SRE – Where to start?