The AlwaysDeny authorizer mode in Kubernetes does not deny requests as expected due to its design to return NoOpinion, requiring configuration adjustments.

Problem Context

The kube-apiserver’s AlwaysDeny authorizer mode returns NoOpinion instead of a direct deny decision. This allows subsequent authorizers in the chain to override the decision, making it unsuitable for production environments where explicit denials are required.

Diagnosis Steps

1. Check authorizer configuration: Verify the --authorization-mode flag on the kube-apiserver. Common modes include RBAC, Node, Webhook, or AlwaysDeny.
2. Review logs: Look for authorizer.DecisionDeny or NoOpinion entries in kube-apiserver logs:

   kubectl logs -n kube-system <apiserver-pod-name> | grep -i authorizer  
   

3. Validate authorizer chain order: Ensure AlwaysDeny is positioned correctly if used alongside other authorizers.

Repair Workflow

1. Adjust authorizer chain: Place AlwaysDeny first in the chain to enforce denials:

   --authorization-mode=AlwaysDeny,RBAC,Node,Webhook  
   

   Note: Requires restarting the kube-apiserver.

2. Test with AlwaysAllow: Temporarily use AlwaysAllow to validate chain behavior before enforcing denials.
3. Update policies: Replace AlwaysDeny with explicit RBAC policies or admission controllers for production environments.

Prevention

• Policy reviews: Regularly audit authorization policies to avoid reliance on non-production-safe modes.
• Staging testing: Validate authorizer changes in non-production clusters before rolling to production.
• Monitoring: Alert on unexpected NoOpinion decisions in audit logs.

Tooling

• kubectl: Inspect API server flags and logs.
• Audit logs: Enable with --audit-policy-file to track authorization decisions.
• Policy generators: Use tools like kubectl create rolebinding --dry-run to test RBAC rules.

Tradeoffs

• Strict policies vs. flexibility: Enforcing AlwaysDeny early in the chain reduces attack surface but may break legitimate workflows if not carefully ordered.
• Operational overhead: Modifying the authorizer chain requires API server downtime, impacting availability during rollout.

Troubleshooting

• Misconfigured chain order: If AlwaysDeny is last, subsequent authorizers may override its decisions.
• Missing RBAC rules: Ensure roles and role bindings are correctly defined to avoid accidental denials.
• Log verbosity: Increase log level to --v=4 on the API server to capture detailed authorizer decisions.

AlwaysDeny is a diagnostic tool, not a production control. Replace it with granular RBAC or admission webhooks for reliable access enforcement.

Source thread: [Question] Am I missing something or a core feature of K8s kube-apiserver is not working as intended??