To capture ephemeral workload spikes like blockchain gas price surges, adjust Prometheus scrape intervals, augment with node-level exporters, and implement targeted alerting.

Diagnosis: Why Default Intervals Fall Short

Kubernetes’ kube-metric-server defaults to 15-second scrapes, which smooths out micro-bursts (e.g., sudden transaction spikes in blockchain nodes). This delay obscures root causes like CPU steal or network latency spikes.

Workflow: Practical Steps to Increase Granularity

1. Adjust Prometheus Scrape Intervals

   • Edit the Prometheus job for kube-state-metrics or custom exporters:

     scrape_configs:  
       - job_name: 'blockchain-nodes'  
         static_configs:  
           - targets: ['blockchain-node:8080']  
         scrape_interval: 5s  
         metrics_path: /metrics  
     

   • Caveat: Lower intervals (e.g., 5s) increase resource usage. Test in staging first.

2. Augment with Node-Level Exporters

   • Deploy node-exporter with --collector.processes enabled to track per-process metrics (e.g., process_cpu_seconds_total).
   • Use cadvisor metrics for container-level CPU/throttling data at 1s intervals via Prometheus:

     - job_name: 'cadvisor'  
       scrape_interval: 1s  
       static_configs:  
         - targets: ['kube-system:10250']  
     

3. Leverage Distributed Tracing for Latency

   • Inject OpenTelemetry traces into blockchain transaction pipelines to correlate metrics with request flows.

4. Implement Alerting on Key Signals

   • Example Prometheus alert for CPU steal:

     - alert: HighCPUSteal  
       expr: rate(node_cpu_seconds{mode!="idle"}[5m]) > 0.9  
       for: 2m  
       labels:  
         severity: critical  
       annotations:  
         summary: "High CPU steal detected on {{ $labels.instance }}"  
     

5. Prevent Overhead with Rate Limiting

   • Use Prometheus’ scrape_config relabel_configs to filter irrelevant metrics:

     relabel_configs:  
       - source_labels: [__metric_name__]  
         regex: '(node_cpu|container_cpu|process_cpu)'  
         action: keep  
     

Tooling

• Prometheus: Custom scrape intervals + recording rules for derived metrics (e.g., rate(container_cpu_usage_seconds_total[1m])).
• Node Exporter: Extended collectors for host-level I/O, disk latency.
• OpenTelemetry: For tracing transaction propagation delays.
• Grafana: Dashboards with 1s-resolution panels for critical metrics.

Tradeoffs

• Resource Overhead: 1s intervals can triple Prometheus storage costs. Use thanos/s3 for long-term retention.
• Cardinality Explosion: High-granularity metrics increase label cardinality. Avoid dynamic labels (e.g., job=<pod-name>).

Troubleshooting

• Scrape Failures: Check Prometheus UI → Targets for scrape_interval mismatches.
• High CPU: Monitor prometheus_scrape_duration_seconds for slow exporters.
• Missing Metrics: Use label_replace to normalize exporter labels (e.g., instance IP vs. DNS).
• Clock Drift: Ensure NTP is synced across nodes to avoid metric misalignment.

Policy Example: Granularity Tiering

# prometheus.yml snippet  
global:  
  scrape_interval: 15s  
scrape_configs:  
  - job_name: 'high_priority'  
    scrape_interval: 5s  
    match_labels:  
      tier: blockchain  
  - job_name: 'low_priority'  
    scrape_interval: 30s  
    match_labels:  
      tier: batch  

For micro-bursts, prioritize 5s intervals on critical components while keeping background workloads at 15–30s to balance observability and stability.

Source thread: How to improve monitoring granularity beyond standard Kube-metric-server intervals?