Prometheus Metrics

Prometheus is a monitoring system commonly used with Kubernetes, which gathers metrics (e.g. CPU usage) from various sources and stores them. You can then query them with a tool like Grafana, which lets you display a graph of the CPU usage of your pods.

Prerequisites

The Prometheus Operator CRDs (monitoring.coreos.com/v1), which define the PodMonitor and ServiceMonitor kinds, must be installed in the cluster before you enable the values below. Without them, the sensor chart skips its PodMonitors and the cluster manager install fails, because Kubernetes does not recognize the ServiceMonitor kind.

Enabling Metrics

Monitoring of the sensors and cluster manager can be enabled by setting the following chart values in the upwind-operator chart:

agent:
  values:
    agent:
      metrics:
        enabled: true
    podMonitor:
      enabled: true
    scanAgent:
      metrics:
        enabled: true
clusterAgent:
  values:
    serviceMonitor:
      enabled: true

Note that the sensor pod is a host-networked pod and enabling metrics will require that the sensor be able to use ports 59090 (for the agent container metrics) and 59091 (for the scan agent container metrics). The metrics port can be configured via the agent.metrics.port and scanAgent.metrics.port agent chart parameters.

PodMonitor and ServiceMonitor

With the default release names, the Sensor chart creates the following PodMonitor:

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: upwind-agent
  namespace: upwind
  labels:
    app.kubernetes.io/name: upwind-agent
spec:
  podMetricsEndpoints:
    - honorLabels: true
      path: /metrics
      port: agent-metrics
      scheme: http
  jobLabel: upwind-agent
  namespaceSelector:
    matchNames:
      - upwind
  selector:
    matchLabels:
      app.kubernetes.io/instance: upwind-agent
      app.kubernetes.io/name: upwind-agent

When the scan agent is enabled, the chart also creates a second PodMonitor with a -scanner name suffix. It scrapes the scan-metrics port of pods labeled app.kubernetes.io/name: upwind-agent-scanner, which exist only when the scan agent runs as its own DaemonSet (scanAgent.sidecar: false).

The Cluster Manager chart creates the following ServiceMonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: upwind-cluster-agent
  namespace: upwind
spec:
  jobLabel: upwind-cluster-agent
  selector:
    matchExpressions:
      - key: app.kubernetes.io/name
        values:
          - upwind-cluster-agent
        operator: In
  namespaceSelector:
    matchNames:
      - upwind
  endpoints:
    - port: metrics

Neither resource sets a scrape interval or timeout by default, so Prometheus uses its own defaults. To set them, add interval, scrapeTimeout or relabelings under agent.values.podMonitor and clusterAgent.values.serviceMonitor in the upwind-operator chart values.