The Kubernetes API server is the central gateway for cluster management, handling requests, enforcing policies, and synchronizing state with etcd.

Core Responsibilities

The API server performs five key tasks in sequence:

1. Request Authentication: Validates client identity via tokens, certificates, or OIDC.
2. Authorization: Checks RBAC policies to ensure the user has permissions for the requested action.
3. Admission Control: Applies mutating or validating webhooks (e.g., auto-injecting sidecars).
4. Object Persistence: Serializes and stores objects in etcd as JSON.
5. Status Reconciliation: Coordinates with controllers to ensure cluster state matches desired state.

Actionable Workflow for Diagnosing API Server Issues

1. Check API Server Logs:

   kubectl logs -n kube-system <api-server-pod-name>  
   

   Look for panic, E0404, or admission denied entries.

2. Verify etcd Health:

   etcdctl endpoint health  
   

   Latency >100ms or errors indicate storage layer problems.

3. Test Request Flow:

   curl -v -H "Authorization: Bearer <token>" https://<api-server>:6443/api/v1/namespaces  
   

   Use kubectl proxy to safely test locally.

4. Audit Admission Controllers:

   kubectl get ValidatingWebhookConfiguration  
   

   Misconfigured webhooks often block legitimate requests.

Policy Example: Restricting Pod Creation

apiVersion: admissionconfiguration.k8s.io/v1  
kind: ValidatingAdmissionConfiguration  
webhooks:  
- name: validate-pod.example.com  
  rules:  
  - operations: ["CREATE"]  
    resources: ["pods"]  
  failurePolicy: Fail  
  clientConfig:  
    caBundle: <ca-cert>  

Tradeoff: Webhooks add latency; use sparingly and test with dryRun: true.

Tooling for API Server Debugging

• kubectl: kubectl get --raw /readyz for health checks.
• k9s: Observe API server resource usage in real time.
• Prometheus: Alert on apiserver_request_duration_seconds p99 > 1s.
• etcdctl: Inspect object versions with etcdctl get --prefix /registry/pods.

Caveats and Failure Modes

• Certificate Expiry: API server stops if TLS certs expire. Automate renewal with cert-manager.
• Rate Limiting: High request volumes can trigger throttling. Monitor apiserver_request_total{code="429"}.
• Etcd Latency: API server performance degrades with etcd round-trip times >50ms.

Troubleshooting Common Issues

• API Server CrashLoop:
  • Check disk space on node (df -h).
  • Rotate logs if --log-path is configured.
• Unauthorized Errors:
  • Verify --service-account-key-file is correctly mounted.
  • Check kube-apiserver manifest for --authorization-mode.
• Slow Responses:
  • Profile with kubectl get --raw /resource-usage.
  • Reduce admission webhook count if admission_delay_seconds spikes.

In production, prioritize monitoring request latency and error rates over raw uptime. A degraded API server with partial functionality is often worse than a complete outage.

Source thread: What happens inside the Kubernetes API server ? - LearnKube