Prometheus Metrics

Prometheus is a monitoring system commonly used with Kubernetes, which gathers metrics (e.g. CPU usage) from various sources and stores them. You can then query them with a tool like Grafana, which lets you display a graph of the CPU usage of your pods.

Enabling Metrics

Monitoring of the sensors and cluster manager can be enabled by setting the following chart values in the upwind-operator chart:

agent:
  values:
    agent:
      metrics:
        enabled: true
    podMonitor:
      enabled: true
    scanAgent:
      metrics:
        enabled: true
clusterAgent:
  values:
    serviceMonitor:
      enabled: true

Note that the sensor pod is a host-networked pod and enabling metrics will require that the sensor be able to use ports 59090 (for the agent container metrics) and 59091 (for the scan agent container metrics). The metrics port can be configured via the agent.metrics.port and scanAgent.metrics.port agent chart parameters.

PodMonitor and ServiceMonitor

The Prometheus PodMonitor resource in Kubernetes is defined by the following:

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: monitoring-upwind-agents
  labels:
    app.kubernetes.io/name: upwind-agent
spec:
  podMetricsEndpoints:
    - honorLabels: true
      path: /metrics
      port: agent-metrics
      scheme: http
      scrapeTimeout: 30s
  jobLabel: upwind-agent
  namespaceSelector:
    matchNames:
      - upwind
  selector:
    matchLabels:
      app.kubernetes.io/instance: upwind-agent
      app.kubernetes.io/name: upwind-agent

The Prometheus ServiceMonitor resource in Kubernetes is defined by the following:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: upwind-cluster-agent
spec:
  jobLabel: upwind-cluster-agent
  selector:
    matchExpressions:
      - key: app.kubernetes.io/name
        values:
          - upwind-cluster-agent
        operator: In
  namespaceSelector:
    matchNames:
      - upwind
  endpoints:
    - port: metrics
      interval: 30s