Deploy Headlamp securely in Kubernetes using OIDC authentication and kubeconfig mounting, avoiding service account tokens.

Problem

Headlamp requires authentication to access the Kubernetes API. While service account tokens are common, they introduce rotation and exposure risks. Token-less alternatives like OIDC are preferable but require careful setup.

Solution Workflow

1. Set up an OIDC provider (e.g., Keycloak, Dex, Okta) integrated with your identity source.
2. Configure the kube-apiserver with OIDC flags:

   --oidc-issuer-url=https://your-oidc-provider.com  
   --oidc-client-id=headlamp  
   --oidc-groups-claim=groups  
   

3. Deploy Headlamp with a mounted kubeconfig:
   • Create a kubeconfig file with credentials for the target cluster.
   • Mount it to the Headlamp pod at /home/headlamp/.config/Headlamp/kubeconfigs/config.
   • Example snippet from a Deployment:

     volumes:  
       - name: kubeconfig  
         secret:  
           secretName: headlamp-kubeconfig  
     volumeMounts:  
       - name: kubeconfig  
         mountPath: /home/headlamp/.config/Headlamp/kubeconfigs  
     

4. Secure the deployment:
   • Use network policies to restrict access to the Headlamp service.
   • Store the kubeconfig in a sealed secret or encrypted volume.
   • Avoid exposing Headlamp publicly; use internal load balancers or VPNs.

Policy Example

Bind a service account with minimal required permissions (e.g., view cluster role) instead of cluster-admin:

apiVersion: rbac.authorization.k8s.io/v1  
kind: RoleBinding  
metadata:  
  name: headlamp-view  
subjects:  
- kind: ServiceAccount  
  name: headlamp-sa  
  namespace: default  
roleRef:  
  kind: ClusterRole  
  name: view  
  apiGroup: rbac.authorization.k8s.io  

Tooling

• kubectl: For validating cluster access and RBAC.
• k9s: To inspect Headlamp pod logs and status.
• OIDC debuggers: Use oidc-cli or provider-specific tools to validate tokens.
• Secret managers: Vault, Sealed Secrets, or Kubernetes Secrets for secure kubeconfig storage.

Tradeoffs

• OIDC complexity: Requires maintaining an OIDC provider and syncing groups/roles.
• kubeconfig mounting: Risk of credential leakage if the secret is exposed.
• PR #5032: Upcoming service account token support may simplify this in the future, but it’s not yet stable.

Troubleshooting

• Kubeconfig path issues: Verify the mount path matches Headlamp’s expected location (/home/headlamp/.config/Headlamp/kubeconfigs/config).
• RBAC denials: Check audit logs with kubectl logs -f headlamp-pod and validate RoleBindings.
• OIDC misconfigurations: Test the OIDC endpoint with curl or jwt-cli to ensure tokens are issued correctly.
• Exposed instances: Use kubectl get services -l app=headlamp to confirm the service isn’t publicly accessible.

Validation

• Access the Headlamp UI and confirm it can list pods/nodes without token errors.
• Run kubectl auth can-i checks as the Headlamp service account to validate permissions.
• Monitor audit logs for unauthorized access attempts.

This approach balances security and usability while avoiding the pitfalls of static tokens. Adjust RBAC scopes and network policies based on your organization’s risk tolerance.

Source thread: Headlamp: token-less in-cluster deployment?