Intermittent ECONNREFUSED errors from an AWS NLB in front of ingress-nginx during scaling are typically caused by health check mismatches, connection draining, or backend misconfigurations.

Understanding the Problem

AWS NLBs route traffic directly to TCP ports, bypassing HTTP-layer health checks. When ingress-nginx pods scale up or down, the NLB may attempt to route traffic to pods that are not yet ready or have been terminated, leading to connection refused errors.

Actionable Workflow

1. Verify Ingress-Nginx Readiness Probes
   Ensure readiness probes are correctly configured to block traffic until the pod is fully initialized. Example:

   readinessProbe:  
     httpGet:  
       path: /ready  
       port: 10254  
     initialDelaySeconds: 5  
     periodSeconds: 5  
   

2. Check AWS NLB Target Group Health
   Use AWS CLI to inspect target group health checks:

   aws elbv2 describe-target-health --target-group-arn <arn>  
   

   Look for unhealthy targets during scaling events.

3. Configure Connection Draining
   Set a connection draining timeout in the NLB settings (AWS Console > Listeners > Modify) to allow in-flight requests to complete before terminating pods.

4. Review Ingress-Nginx Autoscaling Behavior
   Ensure HPA (Horizontal Pod Autoscaler) is not over-aggressively scaling pods. Adjust metrics and scaling thresholds:

   kubectl get hpa ingress-nginx -o yaml  
   

5. Validate Pod Lifecycle Hooks
   Implement pre-stop lifecycle hooks to gracefully terminate connections:

   lifecycle:  
     preStop:  
       exec:  
         command: [/bin/sh, -c, "nginx -s quit"]  
   

Concrete Policy Example

Sample HPA configuration for ingress-nginx with conservative scaling:

apiVersion: autoscaling/v2  
kind: HorizontalPodAutoscaler  
metadata:  
  name: ingress-nginx  
spec:  
  scaleTargetRef:  
    apiVersion: apps/v1  
    kind: Deployment  
    name: ingress-nginx-controller  
  minReplicas: 2  
  maxReplicas: 10  
  metrics:  
  - type: Resource  
    resource:  
      name: cpu  
      target:  
        type: Utilization  
        averageUtilization: 70  

Tooling

• kubectl: Check pod statuses, events, and readiness probe endpoints.
• AWS CLI: Inspect NLB target group health and modify draining settings.
• CloudWatch Metrics: Monitor ConnectionRejected or HealthyHostCount metrics.
• Prometheus/Grafana: Track ingress-nginx pod availability and request error rates.

Tradeoffs and Caveats

• Connection Draining Delay: Longer draining periods reduce errors but increase termination latency.
• Readiness Probe Overhead: Aggressive probing can increase load on backend services.
• NLB Target Group Lag: AWS NLBs may take ~30 seconds to update target health, causing transient errors during rapid scaling.

Troubleshooting Common Failures

• Symptom: Pods report “Unhealthy” in target group despite being ready.
  Fix: Ensure the readiness probe path (/ready) is correctly configured and accessible.

• Symptom: Traffic routed to terminated pods.
  Fix: Verify NLB listener is configured to deregister targets on termination.

• Symptom: Sporadic ECONNREFUSED during scale-up.
  Fix: Increase initialDelaySeconds in readiness probe to allow pod initialization.

When in doubt, simulate scaling events in a staging environment and monitor NLB target health alongside ingress-nginx pod logs.

Source thread: Intermittent ECONNREFUSED from an AWS NLB in front of ingress-nginx, seems correlated with ingress-nginx pod scaling — what’s going on?