Use IP allowlisting for a basic perimeter, combine with OIDC for identity verification, and consider zero trust overlays like OpenZiti for scalable security.

Internal applications like Grafana require strict access controls that scale beyond brittle IP whitelists. Here’s a field-tested approach for Envoy Gateway users.

Diagnose Current Access Patterns

Start by mapping who/what needs access:

• Users: WFH employees, contractors, or services (e.g., CI/CD pipelines).
• Locations: Offices, cloud VPCs, or hybrid environments.
• Protocols: HTTP(S), WebSockets, or gRPC.

Example: Grafana may need access from CI/CD (service account) and internal users (OIDC-authenticated).

Implement IP Allowlisting as First Layer

Use Envoy’s IP mapping for basic perimeter defense:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: HTTPRoute
metadata:
  name: grafana-route
spec:
  parentRefs:
  - name: main-ingress-gateway
  rules:
  - matches:
    - uri:
        exact: /grafana
      host: internal-tools.example.com
    backend:
      service:
        name: grafana
        port: 80
    extensions:
      envoy:
        config:
          accessControl:
            ipDenyList:
              - "192.168.1.0/24"
              - "10.0.0.0/8"

Caveat: IP ranges break with dynamic NAT (e.g., mobile users, cloud load balancers). Use sparingly.

Add OIDC Authentication for Identity Verification

Configure Envoy Gateway with an OIDC provider (Keycloak, Auth0, etc.):

1. Deploy an OIDC provider with groups/roles (e.g., platform-engineers).
2. Configure Envoy’s authentication middleware:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: Extension
metadata:
  name: oidc-auth
spec:
  gateway:
    gatewayClassName: envoy
  config:
    type: authz
    oidc:
      issuer: https://auth.example.com
      clientID: envoy-gateway
      clientSecret: "secret"
      scopes: ["openid", "groups"]

Policy Example: Restrict Grafana to users in the platform-engineers group:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: HTTPRoute
spec:
  rules:
  - matches:
      - uri:
          exact: /grafana
    backend:
      service:
        name: grafana
    extensions:
      envoy:
        config:
          authorization:
            rules:
              - groups:
                  - "platform-engineers"

Tradeoff: Adds latency (OIDC token validation) and requires maintaining an auth provider.

Consider Zero Trust Overlay (OpenZiti)

For environments where IP-based controls fail entirely:

1. Deploy OpenZiti with a Zero Trust controller.
2. Enroll user devices/services as identities.
3. Expose Grafana via OpenZiti’s internal DNS (e.g., grafana.mycompany.internal).

Why It Works:

• No public exposure: Services are only accessible via the Ziti network.
• Identity-based: Devices/users must authenticate to join the overlay.

Caveat: Adds operational complexity (new control plane, client configuration).

Tooling

• Envoy Gateway: For OIDC and IP-based routing.
• OpenZiti: Zero trust overlay (https://www.openziti.com).
• Keycloak/Auth0: OIDC identity providers.
• Prometheus/Grafana: Monitor access patterns and auth failures.

Troubleshooting Common Issues

1. OIDC Token Validation Failures:

   • Check clock sync between Envoy and OIDC provider.
   • Verify aud claim matches clientID.

2. OpenZiti DNS Resolution:

   • Ensure coredns or ziti-dns is correctly configured.
   • Test with dig or nslookup inside the overlay.

3. IP Allowlist Bypass:

   • Audit logs for requests from unexpected IPs.
   • Use network policies to restrict service exposure.

Final Workflow

1. Start with IP allowlists for immediate protection.
2. Layer OIDC for identity verification.
3. Adopt OpenZiti for full zero trust if scale/complexity demands it.
4. Monitor access logs and iterate on policies.

Avoid relying on any single control—layer defenses to mitigate failure modes.

Source thread: What is the best way for me to protect internal company only applications which I need to expose?