Automate scanning, enforce policies, and prioritize fixes without blocking deployments to balance security and velocity.

Vulnerability management in Kubernetes is a race between deployment speed and risk exposure. Teams often face pressure to ship features while ensuring clusters aren’t exposed to known exploits. The key is integrating security into existing workflows without creating bottlenecks.

Actionable Workflow

1. Automate Image Scanning in CI/CD

   • Scan container images before they reach the cluster using tools like Trivy, Clair, or Anchore.
   • Fail builds on critical CVEs (e.g., CVSS ≥ 7.0) but allow non-critical issues to proceed with warnings.
   • Example: Integrate Trivy into GitHub Actions to block pushes with high-severity vulnerabilities.

2. Enforce Admission Controls

   • Use OpenShift’s Image Disruption Controller or OPA Gatekeeper to block deployments with known vulnerabilities.
   • Whitelist trusted base images to reduce false positives.

3. Prioritize Fixes Contextually

   • Not all CVEs are equal: Focus on exploitable vulnerabilities in running workloads (e.g., unpatched kernels in prod clusters).
   • Use tools like Kube-hunter or kube-bench to identify runtime risks.

4. Rotate Credentials and Patch OS

   • Automate certificate rotation with cert-manager.
   • Use OS operators (e.g., Red Hat’s Machine Operator) for seamless node OS patching.

5. Monitor and Report

   • Centralize findings in a SIEM (e.g., Elastic, Splunk) for visibility across clusters.
   • Share dashboards with dev teams to foster ownership.

Policy Example: Block Critical Vulnerabilities

apiVersion: constraints.gatekeeper.sh/v1beta1  
kind: K8sRequiredProhibitedImages  
metadata:  
  name: block_critical_cves  
spec:  
  match:  
    kinds:  
      - resource: pod  
  parameters:  
    prohibitedImages:  
      - regex: ".*critical-cve-pattern.*"  

Note: This is a simplified example. Real policies require integration with vulnerability databases.

Tooling

• Trivy: Lightweight, multi-platform scanner for images and filesystems.
• Clair: Open-source vulnerability static analysis for containers (used in Quay.io).
• Anchore: Policy-as-code engine for contextual vulnerability enforcement.
• OpenShift Container Scanner: Native integration for Red Hat users.

Tradeoffs

• False Positives: Overly strict policies block legitimate deployments. Start with warnings, then enforce.
• Performance: Scanning all layers in CI can add minutes to build times. Cache results where possible.
• Coverage Gaps: Scanners miss runtime exploits. Pair with runtime security tools (e.g., Falco).

Troubleshooting

• Scan Failures:

  • Check network access to vulnerability databases (e.g., Trivy’s --skip-db-update for air-gapped envs).
  • Ensure scanner versions are up-to-date to avoid stale data.

• Admission Control Errors:

  • Audit webhook configurations with kubectl get validatingwebhooks.
  • Test policies in dry-run mode before enforcement.

• Permission Issues:

  • Scanners need read access to image registries and cluster APIs. Use RBAC carefully.

Final Note

There’s no perfect balance—vulnerability management is a risk-reduction game, not a checkbox. Focus on reducing mean-time-to-patch for critical issues while keeping non-critical findings visible but non-blocking. Share ownership with dev teams through clear policies and actionable alerts.

Source thread: How are you handling vulnerability management across Kubernetes clusters without slowing dev teams down?