Adjust worker pool sizes and CPU limits to align thread concurrency with available resources in Quarkus/GraalVM-based Kubernetes operators.

Problem Context

Running a Java operator (Operator SDK + Quarkus/GraalVM native) with limits.cpu: 300m causes throttling despite low average CPU usage. Linux CFS enforces strict time-slicing: 300m = 30ms CPU time per 100ms cycle. High thread counts (e.g., Vert.x default 200-worker pool) exhaust this quota quickly, leading to throttling even if total CPU usage appears low.

Diagnosis Workflow

1. Check thread counts:

   kubectl exec <pod> -- jstack | grep "Thread" | wc -l  
   

   Look for mismatches between thread count and CPU limits (e.g., 200 threads on 0.3 CPU).

2. Verify throttling:

   kubectl top pod <pod> --containers  
   

   If CPU requests/limits are low but throttling is observed in metrics, thread-to-CPU alignment is likely the issue.

3. Inspect runtime behavior:

   kubectl exec <pod> -- /proc/1/stat | grep -o '(\w*)'  
   

   Check for frequent context switches or blocked threads.

Repair Steps

1. Reduce worker pool size:
   Quarkus/GraalVM native images often bake thread pool sizes at build time. Override Vert.x defaults explicitly:

   // In operator code or config  
   vertx.createWorkerPool("operator-worker", 8, 8); // Size and max queue  
   

   Or via system properties at runtime:

   -Dvertx.workerPoolSize=8  
   

2. Set event loop count:

   -Dvertx.eventLoopCount=4  
   

   Align with available CPU cores (e.g., 4 threads for 1 CPU).

3. Adjust CPU limits (if needed):
   Remove or increase limits to allow bursting, but only after fixing thread counts:

   # Example: Set limits to 1 CPU (1000m) or omit entirely  
   limits:  
     cpu: 1000m  
   

Policy Example

Configure operator thread pools via ConfigMap to enforce safe defaults:

apiVersion: v1  
kind: ConfigMap  
metadata:  
  name: operator-config  
data:  
  vertx.properties: |  
    vertx.workerPoolSize=8  
    vertx.eventLoopCount=4  

Reference in deployment:

env:  
- name: VERTX_CONFIG  
  value: "/etc/config/vertx.properties"  

Tooling

• Throttling detection: kubectl describe pod (check Events for OOM kills or throttling hints).
• Thread analysis: jstack, kill -3, or /proc/<pid>/stack.
• CPU profiling: perf, bpftrace, or kubectl exec with top.

Tradeoffs

• Removing CPU limits: Hides throttling but risks noisy neighbor issues.
• Aggressive pool reduction: May underutilize CPU under load; test with realistic workloads.
• GraalVM limitations: Native images may not dynamically adjust pools based on runtime CPU counts.

Troubleshooting

• No improvement after changes:
  • Verify config is applied (check container logs for Vert.x pool initialization).
  • Ensure no other libraries (e.g., Spring, Akka) define their own thread pools.
• Throttling persists:
  • Check for hidden CPU limits in Helm charts, OperatorHub manifests, or cluster autoscaler policies.
  • Use dmesg to rule out kernel-level throttling (e.g., CPU shares misconfiguration).

Final Note

Thread-to-CPU alignment is critical in constrained environments. Start with conservative pool sizes and scale up based on observed behavior, not theoretical throughput. Always validate changes under load before deploying to production.

Source thread: CPU throttling on Quarkus/GraalVM native K8s operator despite low avg CPU usage — thread count culprit?