Use GitLab Agent for secure, centralized access to multiple Kubernetes environments while isolating production and non-production clusters, with optional jumpboxes for compliance needs.

Problem Context

Managing access across dev, staging, and multiple production clusters (prod1, prod2) requires balancing security, compliance, and operational simplicity. Direct access from local workstations risks credential exposure, while jumpboxes add latency and complexity. GitLab Agent offers a middle ground by proxying access through a central identity layer.

Workflow: Access Strategy by Environment

1. Non-Prod (Dev/Staging)

   • Deploy GitLab Agent to each cluster.
   • Configure GitLab CI/CD runners to use the agent for deployments.
   • Grant developers access via GitLab user/group permissions.

2. Production (Prod1/Prod2)

   • Isolate clusters network-wise from non-prod.
   • Deploy GitLab Agent with restricted permissions (e.g., read-only for monitoring, limited namespaces for deployments).
   • Require MFA for GitLab users accessing production clusters.

3. Jumpbox Integration (Optional)

   • Use jumpboxes for environments requiring network segmentation (e.g., air-gapped prod clusters).
   • Configure GitLab Agent to route through the jumpbox’s outbound proxy.

Policy Example: Access Control

# GitLab group configuration snippet  
access_levels:  
  - { access_level: 50, description: "Dev Cluster Access", title: "Dev Team" }  
  - { access_level: 100, description: "Prod1 Deploy Access", title: "Prod Admins" }  

• Rule: Non-prod clusters allow broad team access; production requires explicit admin approval.
• Audit: Review GitLab’s access logs monthly for unauthorized attempts.

Tooling

• GitLab Agent: Centralized access proxy with RBAC tied to GitLab identities.
• Azure AD Integration: Enforce MFA and conditional access policies for production clusters.
• Teleport (Optional): For jumpbox-based access with session recording and audit trails.

Tradeoffs

• GitLab Agent Dependency: If GitLab.com or agents are down, access is blocked. Mitigate with redundant agents and monitoring.
• Jumpbox Overhead: Adds latency and requires maintenance. Use only when compliance mandates network isolation.
• Direct Access Risk: Exposes clusters to credential leaks. Avoid unless teams are small and highly trusted.

Troubleshooting Common Issues

• Agent Connectivity Failures:
  • Check cluster firewall rules allowing outbound HTTPS to Gitlab.com.
  • Verify agent token validity (gitlab-agent status).
• RBAC Denials:
  • Cross-check GitLab user/group membership with cluster RoleBindings.
  • Use kubectl auth can-i to test permissions.
• Jumpbox Routing Issues:
  • Test connectivity from agent to jumpbox with curl -v https://gitlab.com.
  • Inspect jumpbox proxy logs for blocked requests.

Final Note

There’s no universal “right” approach—balance compliance needs with operational friction. Start with GitLab Agent for most environments, add jumpboxes only where required, and enforce strict identity controls everywhere.

Source thread: multiple jumpboxes, local pc, one jumpbox for k8s access ?