Falco detects runtime threats in Kubernetes but requires deliberate tuning and alerting integration to avoid drowning in noise.

Workflow: Deploy, Tune, Integrate

1. Deploy Falco via Operator or Helm chart (sidecar or DaemonSet).
2. Start with defaults, but expect >5k alerts/day initially.
3. Tune aggressively:
   • Suppress known-false-positives using suppress rules (e.g., Spring Boot reading ConfigMaps triggering k8s_api_server_connection).
   • Disable irrelevant rules (e.g., container escapes in non-privileged environments).
4. Forward alerts to a actionable system (Prometheus Alertmanager, SIEM, Slack).
5. Monitor metrics (falco_total_rule_hits, falco_rule_drops) to gauge signal quality.

Policy Example: Allow Spring Boot API Server Connections

- rule: Allow Spring Boot API Server Connections  
  desc: Suppress false positives from Spring apps reading ConfigMaps  
  condition: k8s_api_server_connection and (spawned_process or container_id=SpringApp)  
  output: "Spring Boot accessing API server (allowed)"  
  priority: WARNING  
  suppress:  
    - type: k8s audit  
      k8s.pod.name: springboot-app-*  
      k8s.namespace.name: production  

Tooling

• Falco sidecar: For log aggregation (Elasticsearch, Loki).
• Metrics: Prometheus + Grafana dashboard for alert volume and drops.
• OpenShift: Leverage built-in Falco integration with OCP 4.12+ (limited to cluster-level rules).

Tradeoffs

• Noise vs. Coverage: Strict rules reduce alerts but may miss threats.
• Maintenance Overhead: Rules require updates as workloads evolve (e.g., new image versions).
• Performance Impact: High alert volumes can strain logging/monitoring pipelines.

Troubleshooting Common Failures

• Rule not firing? Check:
  • Falco version (older versions lack features).
  • Audit policy scope (e.g., missing --audit-policy flag).
  • Pod security context (e.g., missing sys_admin capability).
• Excessive drops? Verify falco_rules_drops metric; adjust --max_drops if needed.
• False positives from init containers? Add container.id != init to rule conditions.

In my case, Falco became valuable only after 3 weeks of tuning and integrating with PagerDuty. Without alert routing and regular rule reviews, it’s just a log generator. Start small, validate with known bad behavior (e.g., curl https://metasploit.com in a pod), and expand coverage incrementally.

Source thread: How are you actually using Falco in production?